Terraform detects drift, emits JSON plans, and reapplies approved fixes safely. Build an LLM remediation loop with guardrails. Full breakdown.
Detect drift before it becomes an outage
Self-healing infrastructure starts with knowing when reality diverges from the desired state. Terraform already models that desired state in configuration and state files. Drift appears when someone changes a resource outside the usual apply path, when a provider silently mutates attributes, or when a failed partial apply leaves the stack inconsistent. The practical first step is a scheduled plan against live infrastructure: no apply, only a comparison that surfaces adds, changes, and destroys.
Treat the plan as a signal, not a mandate. Empty plans mean health. Non-empty plans mean a decision is required. Capture the plan output in machine-readable form so downstream automation can classify severity, scope, and whether a fix is reversible. Human-readable logs still help operators, but JSON plans are what make a remediation loop reliable.
Turn JSON plans into actionable fix candidates
A JSON plan lists resource addresses, before/after values, and action types. From that structure you can build a small decision layer: ignore noise (tags, metadata that always churn), flag high-risk destroys, and group related changes into a single remediation unit. The goal is a short, reviewable fix proposal: which resources, what change, and why the plan says it is needed.
Keep the proposal bound to the plan artifact. Store the plan file (or its hash), the configuration revision, and the workspace identity together. That binding is what lets you reapply the same fix later without inventing a second source of truth. If the live stack has moved again, a fresh plan will differ and the old proposal should be discarded rather than forced.
Build an LLM remediation loop with hard guardrails
An LLM is useful for explaining drift in plain language, drafting a minimal configuration patch, and summarizing blast radius for reviewers. It should never be the authority that decides what is safe. The loop is: detect drift → emit plan JSON → ask the model for a proposed config change and human-readable rationale → validate the proposal offline → open a review → apply only after approval.
- Scope: allowlist resources and modules the model may touch; deny networks, identity, and data stores by default unless explicitly approved.
- Actions: block destroy and replace unless a human opts in; prefer in-place updates that match the plan.
- Validation: run format, validate, and a new plan after any suggested edit; reject if the new plan is larger or different from the original drift.
- Apply: use non-interactive apply against the approved plan file only; never free-form apply from chat output.
- Audit: log prompt, model output, plan hashes, approver, and apply result for every cycle.
If validation fails or the plan grows, stop the loop and escalate. Self-healing that cannot fail closed will eventually heal the wrong thing.
Reapply approved fixes safely and measure the loop
Once a human or policy engine approves a fix, reapply using the locked plan, not a regenerated one from memory. Run the apply in a controlled identity with least privilege, in the correct workspace, and with a clear rollback path: previous configuration revision, previous state snapshot, and a known-good plan from before the change. After apply, run another plan immediately. A clean plan closes the incident; residual drift means the remediation was incomplete and should re-enter detection, not be marked success.
Operationally, keep the loop boring: scheduled drift checks, durable plan storage, approval gates, and apply only through the same pipeline that ships normal infrastructure changes. The LLM accelerates diagnosis and drafting. Terraform, policy, and approval remain the control plane that makes self-healing safe enough to run without constant firefighting.