Infrastructure as Code and Immutability

Brief

At Halcyon Clinics, an engineer adds a temporary public administration rule during an outage. The next deployment removes it, but nobody can explain why it existed for six hours. Meanwhile, a shared infrastructure state file contains a generated database password, and the release manager calls the latest machine image immutable even though operators still patch running machines by hand.

The team needs more than infrastructure written in text files. It needs an explicit desired configuration, protected resource mappings, a reviewable change calculation, and deployment evidence that identifies what actually ran. This room follows a three-instance application through a change and separates reproducibility, confidentiality, availability, and authorization: each needs its own control.

Versioned configuration records intent; only linked approval, execution, and verification records demonstrate an authorized change.

Declare the destination, then calculate the route

An imperative procedure says create a network, then create an instance, then attach a rule. A declarative configuration says three instances must exist, each attached to the private network and the approved rule set. If two instances already exist, an engine compares configuration with managed resource mappings and refreshed remote attributes to calculate the missing work. Repeating a well-designed convergence operation should not add three more instances. Declarative syntax alone does not guarantee idempotence: external scripts and unmanaged resources can still introduce side effects.

Desired state is the approved target, not the last successful deployment. A reusable module can encode a private application tier with typed inputs for instance count and image identifier, validation that count is at least two, and outputs for resource identifiers. Pin its revision: changing a shared module from version 4 to 5 may alter every consuming environment. Modules reduce duplication, but a weak default such as public administration access can multiply exposure across twenty deployments.

For configuration management controls, retain the reviewed source revision, pinned module revision, input values without secrets, and test results. An auditor can ask whether all twenty environments used the reviewed module version and whether exceptions to its network defaults were explicitly approved; a reusable folder alone proves neither.

State is a sensitive map, not the desired configuration

A state file commonly maps a logical address such as application.node[0] to a remote resource identifier and cached attributes. It may also retain generated passwords, connection strings, or sensitive output values. Hiding a value from terminal output does not necessarily remove it from state. Treat state and saved plans as sensitive artifacts: restrict readers and writers, encrypt storage and transport, audit access, and keep them out of ordinary source history. Removing a leaked file from the latest revision does not revoke exposed credentials or erase earlier copies.

Use a protected shared backend with locking or equivalent serialization so two writers do not both plan against state generation 18 and overwrite each other's mappings. Separate production and test state boundaries and credentials. Maintain versioned backups and rehearse restoration: an old state copy can reference deleted objects or omit newer ones, so restoration requires reconciliation before any destructive apply. A lock prevents competing state writes; it does not prevent someone changing a resource through another management channel.

Least-privilege and recovery evidence includes state access logs, backend access policy, lock behavior tests, and a dated restore exercise. An auditor should ask who can download historical versions and how a restored mapping is checked against actual resources, not merely whether the backend reports encryption enabled.

Plans, approvals, and drift form a change chain

A plan calculates proposed operations; apply executes them. For a change from three to four instances, a plan might report one create, zero updates, and zero deletes. A replacement is different: changing a boot image may require creating a new instance and destroying the old one. Review deletions, replacements, dependencies, permissions, and data persistence rather than approving on the basis of a small line count. Apply can partially fail; a plan is not a transaction guarantee or proof that access will succeed.

Bind approval to the source revision, environment, inputs, and saved plan digest. Restrict apply to a deployment identity separate from the author, and verify the applied artifact matches the approved one. If state or inputs have changed, invalidate the old approval and calculate a fresh plan for review. Scheduled refresh-and-compare runs detect drift: if the approved rule allows administration only from 10.24.8.0/24 but the live rule allows 0.0.0.0/0, raise an incident. Do not silently import the public rule into code to make the difference disappear.

Retain approval identity and timestamp, plan digest, execution logs, and post-deployment verification under one change identifier. Drift findings need an owner, response deadline, and documented resolution: revert unauthorized changes or approve a justified target change through review. An auditor sampling an emergency change should be able to trace detection through authorization or remediation, including any period of exposure.

Replace machines; preserve data deliberately

Immutable infrastructure replaces deployed units instead of repairing them in place. Build golden image release 12 from reviewed inputs, patch and scan it, record its digest and test results, and deploy that exact artifact. A golden image is an approved baseline, not a promise of permanent safety: new vulnerabilities still require rebuilding and replacing it. Keep credentials out of images and inject short-lived access at runtime; otherwise every replacement reproduces the same secret exposure.

For a three-instance service, create three release-12 replacements alongside release 11, check health, shift traffic, and retire the old group after observation. Capacity temporarily rises from three to six; quota and cost must support the overlap. Store durable records outside replaceable compute. Rolling back traffic to release 11 can restore application behavior but cannot undo a destructive database migration; use compatible schema transitions and independently tested backups rather than calling an old image a data recovery plan.

Baseline and vulnerability controls need image provenance, scan dates, digest pinning, deployment inventory, and proof that old instances were retired. Code review is change control only when required approvals cannot be bypassed unnoticed and the reviewed artifact is the deployed one. An auditor should ask whether running image digests match approved releases and whether emergency access left any long-lived, hand-patched exceptions.

Answers are graded on the server: the answer key never reaches your browser.