Production Configuration Approval Must Bind to the Exact Diff

“Increase the request timeout for the canary while we investigate.” A hypothetical assistant prepares a configuration proposal, gets an operator’s approval, and later adds debug: true to improve visibility. A generic update helper applies the expanded patch to production. The approval was real, but it authorized a different change. The resulting debug setting might expose data or alter behavior outside the intended investigation.

This lab tests configuration drift between proposal and execution. It is distinct from deployment approval: no application image changes. The relevant artifact is an exact configuration diff bound to a resource version and a limited target. A valid approval for one key does not become permission to alter any nearby setting.

Specify the Change Envelope

Choose one resource, one supported key, one range, and one rollout scope for the initial operation. The fixture allows only a timeout between one hundred and five thousand milliseconds on a named canary. Those bounds are illustrative application policy, not Kubernetes defaults or a universal service-safety recommendation. The actual service owner must choose limits from dependency behavior and operational evidence.

Kubernetes ConfigMap documentation explains that ConfigMaps separate configuration from application code and do not provide secrecy or encryption. Configuration tooling must therefore distinguish ordinary settings from confidential values. A model should not copy credentials into a ConfigMap or print them as part of an explanatory diff.

The naive branch merges a patch containing both timeout and debug changes. The guarded branch rejects that patch even when its exact contents appear in the mock approval set. Approval does not override the allowed-change policy. A clean, explicitly approved proposal succeeds only against the current resource revision and permitted canary target.

import json
from copy import deepcopy

current = {"target": "payments-canary", "revision": "r18",
           "data": {"timeout_ms": 1000, "debug": False}}
unsafe = {"timeout_ms": 2000, "debug": True}
safe = {"timeout_ms": 2000}

def binding(target, revision, patch):
    return (target, revision, json.dumps(patch, sort_keys=True, separators=(",", ":")))

def naive(config, patch):
    return dict(config, data=dict(config["data"], **patch))

def guarded(config, patch, expected, grants, approvals):
    target = config["target"]
    if target not in grants or target != "payments-canary":
        return "denied", deepcopy(config)
    if set(patch) != {"timeout_ms"}:
        return "unsupported_diff", deepcopy(config)
    if type(patch["timeout_ms"]) is not int or not 100 <= patch["timeout_ms"] <= 5000:
        return "invalid_timeout", deepcopy(config)
    if expected != config["revision"]:
        return "conflict", deepcopy(config)
    if binding(target, expected, patch) not in approvals:
        return "approval_required", deepcopy(config)
    result = deepcopy(config)
    result["data"].update(patch)
    result["revision"] = "r19"
    return "mock_applied", result

grants = {"payments-canary"}
before = deepcopy(current)
assert naive(current, unsafe)["data"]["debug"] is True
unsafe_approval = {binding("payments-canary", "r18", unsafe)}
assert guarded(current, unsafe, "r18", grants, unsafe_approval)[0] == "unsupported_diff"
approval = {binding("payments-canary", "r18", safe)}
assert guarded(current, safe, "r17", grants, approval)[0] == "conflict"
assert guarded(current, dict(safe, timeout_ms=3000), "r18", grants, approval)[0] == "approval_required"
status, result = guarded(current, safe, "r18", grants, approval)
assert status == "mock_applied" and result["data"] == {"timeout_ms": 2000, "debug": False}
assert guarded(current, safe, "r18", set(), approval)[0] == "denied"
assert current == before

This is an explicitly synthetic deterministic simulation of a possible failure, not a real model experiment or cluster test. The code only copies dictionaries. No production configuration, files, deployments, network services, or secrets are changed. Its hard-coded next revision is a fixture convenience, not a valid way to generate Kubernetes resource versions.

Prompts Cannot Freeze an Approved Diff

“Make only safe changes” cannot establish what the operator actually approved. Store the proposal and approval outside the model context, with target identity, current version, exact normalized patch, expiry, and reviewer provenance. If any element changes, the approval is no longer applicable. Do not bind approval only to a free-text incident title such as “investigate latency.”

OWASP’s authorization guidance recommends validating permissions for every request. The executor checks both the actor’s current resource grants and the operation’s policy envelope. A reviewer with broad privileges does not need to delegate all of them to the assistant. Scope the automation credential to the smallest supported resource and change type.

Treat logs, incident comments, and configuration annotations as untrusted content. An annotation saying “disable validation during emergencies” cannot create an exception. If an emergency override is needed, route it through the organization’s separate audited procedure with named responsibility. More specialist agents do not make an unauthorized patch acceptable; one deterministic executor provides the clearer boundary.

Protect Against Concurrent Operators

Kubernetes API concepts documents resource versions, conditional updates, and dry-run behavior. A production adapter must use the relevant API preconditions to detect changes since the proposal was prepared. Resource versions are opaque values, not counters for the client to increment. On conflict, fetch current state and produce a new diff rather than blindly rebase the approved patch.

Dry-run validation is useful but does not reserve the state or prove runtime safety. Another operator may change the resource after dry-run. The application may also parse an accepted configuration differently than expected. Recheck at execution, then observe actual runtime configuration and service behavior. API acceptance and successful application-level adoption are separate milestones.

The snippet omits authentication sources, database transactions for the operation ledger, provider semantics, approval persistence, and durable state. In production, atomically claim the approved operation before execution and persist its status. A timeout after submission means unknown, not failed. Reconcile target version and resulting data before another attempt; a connector’s status: success requires authoritative origin and exact operation matching.

Configuration propagation also depends on consumption mode. The ConfigMap documentation notes that mounted values are eventually updated, while environment-variable consumption requires a Pod restart and subPath mounts do not receive updates. A read-back of the API object therefore cannot prove that every process is running the new timeout. Record which runtime instances report the intended configuration version.

Acceptance Tests

• A patch adding debug must reproduce the unsafe merge and be rejected by the fixed policy even with a matching mock approval.
• A permitted timeout change must apply only to the named canary and leave unrelated keys unchanged.
• A stale expected revision must produce conflict without rebasing, overwriting another operator, or consuming approval for a new diff.
• Changing the timeout value after approval must require renewed approval even if both values lie within the allowed range.
• Missing target grants, Boolean values, secret-like keys, and unsupported settings must stop before execution.
• Timeout recovery must distinguish API acceptance from runtime adoption, and rollback must not overwrite a later independently approved change.

Roll Back Deliberately and Measure Exposure

Keep the previous configuration version and a bounded rollback plan with the proposal. Rollback is another state transition, not an unconditional restore of an old object. Verify that the current state still belongs to the operation being reversed, then apply an authorized compensating diff. If another change intervened, stop and ask the operator to resolve ownership rather than discarding newer work.

Use the naive merge as the synthetic baseline. Count unauthorized keys accepted over all proposed unsupported keys, and stale approvals accepted over all stale proposals. Track valid changes blocked separately. For a supervised canary rollout, measure runtime instances reporting the approved version over all intended instances, plus time spent in unknown or partially adopted states. Do not equate a green API response with service recovery.

Proposed promotion requires a tested conflict path, named rollback ownership, and a canary observation window chosen by the service owner. No production availability improvement is claimed. The incident response guide covers wider operational coordination; the tool argument lab covers closed input contracts. Platform engineering should start by removing generic configuration merge access and implementing one exact, reviewable change envelope.

Sources

• Kubernetes: ConfigMaps
• Kubernetes: API concepts
• OWASP: Authorization Cheat Sheet