Treating the agent as untrusted on live AWS

A raw model generating infrastructure-as-code fails in four distinguishable ways, and they need different fixes.

It invents resource types and properties that do not exist. This is a schema problem, and it is the easy one. It is checkable without touching the account.

It generates something schema-valid that your account will reject: over a quota, in a region where the service is not enabled, referencing a resource that is not there. No amount of validating the document catches this, because the document is fine. The account is the missing input.

It drifts. Ask twice, get two different templates for the same request, which makes review meaningless. A reviewer cannot build intuition about output that is never the same shape twice.

And it produces resources nobody can later attribute. This one does not look like a failure at generation time. It looks like a failure six months on, when someone is trying to work out whether a thing is safe to delete.

The rollback cost is what turns all four from annoyances into blockers. A failed stack can take minutes to unwind, so "apply and see" is not a slow development loop, it is an unusable one.

• Deterministic output, so the same request produces the same template

• Account state as a planning input, not something discovered at apply time

• Validation before the provider is touched, because the provider's feedback loop is minutes long

• Attribution that survives, on every resource, without an engineer remembering to add it

One rule: the agent is untrusted, and every step is either machine-checkable or human-reviewable. Nothing in the system depends on the model being correct. That is not a hedge. It is what makes the product adoptable, because it means an engineer does not have to form an opinion about model quality to decide whether to run it.

The agent reaches AWS only through a set of purpose-built tools that wrap the provider APIs. Those tools are the entire action surface: reads for account state, generation into a deterministic template format, validation, rendering, and a controlled apply. There is no general shell, and no path by which a plan step becomes an arbitrary API call.

Between plan and apply sit two gates of different kinds. The validation loop is the machine gate, structural and semantic checks, run repeatedly, refining until the template is clean and provider-valid. The architecture diagram is the human gate, and it exists because the alternative review surface is raw template diffs, which engineers do not read carefully at the tenth change of the week.

Design decision: the agent reads the live account before it plans, not after it generates. Validating a template against a schema tells you it is well-formed. Only the account tells you it will apply.

Read the account#

Dedicated tools pull live limits, quotas, regions and existing resources. The planner is working against the account that exists rather than a general model of AWS, which is what removes the entire class of schema-valid-but-unapplicable output.

Plan into a structured spec#

The free-form request becomes a structured spec and an explicit chain of tool calls. Determinism matters more here than sophistication: a reviewer can only build judgement about output whose shape is stable.

Generate and validate in a loop#

The agent emits infrastructure-as-code, then runs structural and semantic checks and refines against the failures, repeating until the template is clean. Most errors die in this loop, before AWS is asked to start anything, and therefore before any rollback clock starts.

Render the change#

A human-readable architecture diagram of what will exist after the change. This is the review surface. It converts approval from reading config into looking at a picture, which is the only version of review that stays honest at volume.

Apply through a change set#

Execution goes through the provider's change-set mechanism with retries, so a partial failure does not leave orphaned resources behind. The distinction between "the apply failed" and "the apply failed and left four things behind" is most of the operational cost.

Tag everything#

Every created resource carries its creator, owner, environment, the tool that made it, and the request behind it. Attribution is written at creation because it will never be reconstructed afterwards.

Governance#

The agent never holds open AWS authority. The tool set bounds what it can call; deterministic templates bound what it can produce; provider validation bounds what can be applied; the change set bounds how it is applied; and mandatory tags bound what is left behind. The architecture preview is the one gate that is deliberately human, because "is this the change we meant" is not a question a validator can answer.

Orchestration#

Inspect, plan, generate, validate, revise, render, apply, inspect again. Retries sit in two distinct places (inside the validation loop, and around provider execution) because they are recovering from different things: a malformed template versus a transient or partial provider failure.

Observability#

One record links the request to the account facts it was planned against, the generated template, every validation failure and the revision that answered it, the architecture preview, the provider response, the retries, and the resources that now exist.

The validation failures are the part worth keeping. They are the system's own record of what the model gets wrong, which is the input to improving the loop rather than improving the prompt.

The failures are provider and state failures, not generation failures. Generation failures are caught upstream by design.

A change set fails partway. A resource the plan depended on has changed since the account was read. A quota moves between planning and applying. A dependent resource is in a state that blocks the change. Each of these has to end in a known state rather than a half-built one, which is why apply goes through change sets with retries instead of direct calls: the failure mode of a direct call is orphaned infrastructure that nobody knows to clean up.

The validation loop also has to know when to stop. A template that will not come clean after bounded revision is escalated as a request the agent cannot safely fulfil, rather than refined indefinitely against a constraint it cannot satisfy.

The loop, not the model, is what made this deployable.

• Template errors before deploy fell from ~10% to 4%, with most failures caught before AWS starts a change and its rollback clock

• Quota breaches and unsafe configurations blocked pre-apply, unencrypted storage, mis-scoped security groups, accidental public exposure

• No orphaned resources from a failed run, because apply goes through change sets with retries rather than direct calls

• Every agent-created resource attributable to owner, environment, tool and originating request

Directly demonstrable in the product: live account inspection, the validation loop, the architecture preview, controlled change sets, resource tagging, and a working system that still runs against a real AWS account.

Internal records not independently verified: the 10%-to-4% error change, the roughly four-month delivery, and full tagging coverage. The tagging figure is the one we would most want re-checked before it is quoted as 100%.

If you are building something in this shape, the question that decides it is not which model writes better templates. It is what your review surface is. If the answer is "a diff of generated config", adoption will stall regardless of how good the generation gets, because nobody reads the tenth one of the week.