Building an LLM agent that keeps documentation in sync at commit time

Every codebase has a set of documents that quietly lie.

Not maliciously. They were true once. Someone wrote a clear explanation of how a feature works, what its thresholds are, what it outputs. Then six months of commits happened. A default changed. A condition flipped. A new edge case got handled. Each change was reasonable, each one shipped with a code review — and not one of them touched the doc. So now the doc describes a system that no longer exists, and the worst part is that it still reads as authoritative. A new engineer trusts it. An on-call responder trusts it at 2am. The doc is a confident, well-formatted, completely wrong source of truth.

The usual fixes don't work. "Just update the docs when you change the code" is a discipline that survives exactly as long as the deadline pressure is low. "Generate the docs from the code" works for API signatures and nothing else — it can't explain why a threshold is what it is, or what a rule means for a user. And the tempting modern answer — "point an LLM at the repo and tell it to fix the stale docs" — fails in a more interesting way, which is most of what this post is about.

I built an agent that closes the gap a different way. It runs as a pre-commit hook, looks at exactly what you're about to commit, figures out which docs that change makes wrong, and rewrites them — then blocks the commit so you review the result before it lands. It's been keeping a large documentation set honest for a while now. And the thing I'm proudest of is how little of it is the LLM.

Why "ask the model to fix the docs" doesn't work

Start with the naive version, because understanding why it fails is the whole design rationale.

The naive agent is one prompt: "Here is my repository. Here are my docs. Update any docs that are out of date with the code." It's one line to write and it's a disaster in production for three separate reasons:

It can't see what changed. Without a notion of "this commit touched these files," the model re-evaluates the entire world every time. It has no signal for what to look at, so it either reviews everything (slow, expensive, noisy) or guesses (misses things).
It hallucinates the links. Ask a model "which doc does this code change affect?" and it will answer confidently, sometimes correctly, sometimes by pattern-matching a filename that looks related. Connecting a changed source file to the document it invalidates is a lookup, not a judgment call — and lookups are exactly where you don't want a probabilistic answer.
It invents the values. The most dangerous failure. A doc says a limit is 500. The code now says 1000. Ask a model to "update the doc" without handing it the authoritative new value and it will write a number — often a plausible-looking wrong one pulled from the surrounding prose. A confidently incorrect threshold is worse than a stale one.

Every one of these failures comes from giving the model a job it's bad at: searching, linking, and recalling exact facts. So the design principle for the whole agent falls right out of it:

Do everything mechanical in plain, deterministic code. Reserve the model for the two things it's genuinely good at: judging whether a change matters, and writing careful prose.

That sounds obvious written down. In practice it inverts where most people put the intelligence. The interesting engineering isn't the prompt — it's the pipeline that runs before the prompt, narrowing a whole repository down to "these two documents, this exact changed code, and these specific config values are the source of truth." By the time the model is invoked, the hard part is already done.

Here's the shape of it:

flowchart LR
    A["Staged files<br/>(git diff --cached)"] --> B["Deterministic mapping<br/>code symbol → doc"]
    B --> C{"Any candidate<br/>docs?"}
    C -->|no| Z["Exit 0<br/>commit proceeds"]
    C -->|yes| D["Attach config<br/>as source-of-truth"]
    D --> E["Planner agent<br/>which docs need updating?"]
    E --> F["Writer agent<br/>rewrite the markdown"]
    F --> G["Block commit<br/>developer reviews + stages"]

Two of those six boxes touch the LLM. The other four are the reason it works. Let me walk through them in order.

Step 1 — Find what changed (git, not the model)

The agent runs at commit time, so "what changed" is a question git already answers precisely. No model needed, no repository scan.

def staged_files(repo_root: Path) -> list[str]:
    """Repo-relative paths of files staged for commit (added/modified/renamed)."""
    out = _git(["diff", "--cached", "--name-only", "--diff-filter=ACMR"], repo_root)
    return [line.strip() for line in out.splitlines() if line.strip()]


def staged_diff(path: str, repo_root: Path) -> str:
    """The staged unified diff for a single file."""
    return _git(["diff", "--cached", "--", path], repo_root)

The --cached flag is the important bit: it looks at the staged snapshot — exactly what's about to be committed, not the working tree. The --diff-filter=ACMR drops deletions, because a deleted file has no new content to document against.

This is the first place the agent earns its reliability. It doesn't reason about what changed; it reads what changed, from the one tool that knows for certain.

Step 2 — Map code to docs, deterministically

This is the step I'm proudest of, and it's pure string-handling. Not a token of model usage.

The trick is to exploit a link that already exists in any well-structured docs. In my setup, every document is keyed by an authoritative identifier — a stable name that ties the human-readable doc to the code construct it describes. Concretely, each markdown doc carries a line like:

| **Internal Name** | `RATE_LIMIT_EXCEEDED` |

and the code references that same identifier through an enum or a registration decorator:

@register_rule(RuleName.RATE_LIMIT_EXCEEDED)
def check_rate_limit(ctx: RequestContext) -> RuleResult:
    ...

That shared identifier is the whole game. If I can extract RATE_LIMIT_EXCEEDED from a changed source file and find the doc whose authoritative name is RATE_LIMIT_EXCEEDED, I've linked code to documentation with zero ambiguity and zero model calls. So the mapping layer is a set of small extractors and an index:

_INTERNAL_NAME_RE = re.compile(r"\*\*Internal Name\*\*\s*\|\s*`([A-Z0-9_]+)`")
_RULE_NAME_RE     = re.compile(r"RuleName\.([A-Z0-9_]+)")
_FEATURE_NAME_RE  = re.compile(r"FeatureName\.([A-Z0-9_]+)")
_RENDER_SQL_RE    = re.compile(r"""render_sql\(\s*["']([^"']+\.sql)["']""")


class DocIndex:
    """Index over the docs directory, keyed by authoritative identifier."""

    @classmethod
    def build(cls, docs_dir: Path) -> "DocIndex":
        idx = cls()
        for doc in sorted(docs_dir.glob("*.md")):
            text = doc.read_text(encoding="utf-8")
            if m := _INTERNAL_NAME_RE.search(text):
                idx.name_to_doc[m.group(1)] = doc          # authoritative link
            for token in set(re.findall(r"`([A-Z][A-Z0-9_]{3,})`", text)):
                idx.token_to_docs.setdefault(token, set()).add(doc)  # body mentions
        return idx

Now, real systems aren't one-hop. A rule is implemented across layers — the rule itself, the feature code that assembles its inputs, a nightly job that pre-computes data, the SQL behind that job, the config that holds its thresholds. A change to any layer can invalidate the doc. So the resolver follows the chain: a changed SQL file is linked to the feature that renders it, which is linked to the rule it feeds, which is linked to the doc. Each hop is a deterministic lookup.

Strong evidence vs. weak evidence

The subtlety that makes this usable rather than noisy is grading the evidence. Not every mention is equal.

Strong: the change references a token that is a doc's authoritative identifier (or feeds one through the SQL chain). This doc is almost certainly affected. Always include it.
Weak: the change references a token that merely appears in a doc's body — for example a shared data source that a dozen docs happen to cite. That's a hint, not a smoking gun.

# A token cited across more than a few docs is "shared infrastructure"
# (think a common context object every rule reads). A body-only mention of
# such a token is too weak to act on — it would flag half the docs in the repo.
_SHARED_TOKEN_DOC_THRESHOLD = 3

def _weak(doc, token, reason):
    if len(index.token_to_docs.get(token, set())) > _SHARED_TOKEN_DOC_THRESHOLD:
        return  # suppress shared-infra noise
    weak.setdefault(doc, set()).add(reason)

Then the merge rule: if any strong candidates exist, the weak-only ones are dropped — the strong doc is the real target. If there are no strong candidates, the weak ones are kept, so the agent never goes completely blind on an unusual change. Strong wins, but weak is a safety net rather than a source of noise.

Bridging the names that don't quite match

The one place the deterministic link gets fuzzy is config. A YAML key is rarely exactly a doc's identifier. The rule is RATE_LIMIT_EXCEEDED; the config knob is rate_limit_rpm. A human sees instantly that they're related; a naive equality check sees nothing.

So there's a small "stem bridge": take each doc's identifier, generate its progressive prefixes (rate_limit_exceeded → rate_limit → rate), and link a config change if one of those multi-word stems is a prefix of a changed YAML key. A single-word stem like rate is too greedy, so the bridge requires at least two words — enough to be specific, loose enough to catch the real relationship.

def _snake_stems(snake: str) -> list[str]:
    """Progressive prefixes, longest first.
    'rate_limit_exceeded' -> ['rate_limit_exceeded', 'rate_limit', 'rate']."""
    parts = snake.split("_")
    return ["_".join(parts[: i + 1]) for i in range(len(parts) - 1, -1, -1)]

The output of this entire step is a small dict: {doc_path: {human-readable reasons}}. Usually one or two docs. Often zero — in which case the agent exits and the commit sails through, having spent a few milliseconds of regex and not one cent of inference. The cheapest request is the one you never send.

Step 3 — Attach the source of truth

Remember failure mode #3: the model inventing values. This step kills it.

For every candidate doc, the agent gathers the actual config that defines the values the doc describes — the threshold files, the parameter YAML, the per-environment overrides — and attaches it to the prompt as reference context. Small files go in whole; large ones are sliced down to just the block that mentions the relevant identifier, so the prompt stays lean and focused.

def gather_context_for_docs(docs, repo_root) -> dict[Path, list[tuple[str, str]]]:
    """For each doc, collect the config/override YAML that defines its values.

    Small files attach whole; large files contribute only the matching block.
    This is what makes thresholds accurate — and what lets the agent catch
    cross-file drift, e.g. an override value that no longer matches its default.
    """

This does two things. First, it gives the writer the real new number to put in the doc, so it never has to guess. Second — and this surprised me — it lets the agent catch cross-file drift. Because the doc, the code diff, and the config all land in the same context, the model can notice that an override value no longer matches the default it's supposed to override, even when the commit only touched one of them. The agent stopped being a doc-updater and quietly became a consistency checker.

The instruction to both agents is explicit about it: treat the attached config as the source of truth for any value; if a number isn't in the provided context, describe it qualitatively rather than inventing one. You're not asking the model to remember facts. You're handing it the facts and asking it to write them down well.

The two agents: decide, then write

Only now does the LLM enter, and it enters as two agents with sharply different jobs. This split matters more than it looks.

The planner: does this doc actually need to change?

The planner's only job is judgment. Given the diffs, the candidate docs, and the config context, it decides — per doc — whether the change actually alters documented behavior. Crucially, it's told to say no for the large class of changes that touch code without changing meaning:

pure refactors, renamed internal variables, type hints, reordered imports
logging, comments, formatting
test-only or tooling changes
performance tweaks with identical behavior

And to say yes only when an externally observable fact moved: a threshold, a pass/fail condition, an input, an output, an added or removed rule. When genuinely unsure, it errs toward yes and explains why — a stale doc that gets a second look is cheaper than a wrong doc that ships.

The output is structured, not prose. Each decision is a typed object the orchestrator can branch on:

class DocDecision(BaseModel):
    doc_path: str
    needs_update: bool = Field(
        description="True only if the change makes the current doc inaccurate. "
        "Refactors, renames, formatting, logging, and test-only edits do NOT count."
    )
    reason: str
    change_summary: str = Field(
        default="",
        description="If needs_update: a precise description of what must change "
        "(sections, values, conditions). Empty otherwise.",
    )

class PlanResult(BaseModel):
    decisions: list[DocDecision]

A structured needs_update: bool means the planner can reject work cheaply and the orchestrator can trust the verdict without parsing free text. Most commits that reach the planner get a row of confident "no — this is a refactor" decisions, and the agent stops there. The expensive writer never runs.

The writer: rewrite, but change only what must change

For the docs the planner flagged, a second agent does the rewrite. It gets the current markdown, the diff, the config context, and the planner's specific change_summary. Its constraints are deliberately narrow:

preserve the existing template, headings, and tables exactly
change only what the instruction requires
match the surrounding tone and level of detail
use the attached config for any value, never a remembered one

class WriteResult(BaseModel):
    updated_markdown: str = Field(
        description="The COMPLETE updated file. Preserve structure; change only "
        "what the instructions require."
    )
    notes: str = Field(default="", description="What changed, for the reviewer.")

Why split decide from write at all? Three reasons that all showed up in practice. Cost: rejecting a no-op change with the planner is far cheaper than having one big agent read a doc and regenerate it just to conclude nothing was needed. Focus: a writer told exactly what to change by an upstream judge produces tighter diffs than one asked to both decide and write in a single pass. Auditability: the planner's structured verdict is a record of why a doc was (or wasn't) touched — it shows up in the commit output, so the developer sees the reasoning, not just the result.

One more detail that paid for itself: the prompts live in a YAML file, not in the code. Tuning how conservative the planner is, or how the writer handles a missing value, is a config edit and a re-run — no code change, no redeploy. The behavior of the agent is data.

Wiring it into the commit

The delivery mechanism is as important as the pipeline, and it echoes a lesson I keep relearning: put the check at the moment of change. The best time to fix a doc is the instant the code that invalidates it is written — not in a quarterly "docs cleanup" that never happens. A pre-commit hook is exactly that moment.

The contract is built entirely out of exit codes:

# 0 -> docs in sync, commit proceeds
# 1 -> docs were updated (or need updating); commit blocked for review
# 2 -> configuration error (e.g. missing API key); commit blocked

And three design choices around it matter more than the code:

It never auto-commits. When a doc needs changing, the agent writes the new markdown to disk and blocks the commit. The developer reads the generated diff, stages it if they agree, and commits again. A human stays in the loop on every word that lands. An agent that silently rewrites your docs and commits them is an agent you stop trusting the first time it's wrong.

It fails open, not closed. A transient API error should not hard-block someone from committing. If the agent itself throws — gateway hiccup, timeout — it logs a warning and exits 0. Stale docs are a problem; a developer unable to commit because an external service blipped is a worse one. The hook degrades to "no-op," never to "blocked."

try:
    report = asyncio.run(run(settings, explicit_files=explicit, write=not dry_run))
except Exception as exc:
    # Don't let a transient agent error hard-block a commit.
    print(f"docs-agent: skipped (agent error: {exc}).", file=sys.stderr)
    sys.exit(0)

It only wakes up when it's relevant. The hook's trigger is scoped to the paths that actually have documented behavior — the rule code, the features, the SQL, the config. Edit a README or a test, and the agent never even starts. The cheapest pipeline is the one that knows when to stay asleep.

What it actually bought us

Docs stopped drifting silently. This was the whole point. The window between "behavior changed" and "doc updated" went from however long until someone notices to the same commit. The doc and the code move together because the tool won't let them separate.

The model's job got small enough to trust. Because the deterministic layers hand the model a tiny, pre-linked, fact-grounded problem, the failure modes that sink the naive approach — wrong links, invented numbers — mostly can't occur. The model isn't searching or recalling; it's judging and phrasing. Narrow the job and reliability follows.

Refactors are free. The planner rejects behavior-preserving changes, so the daily flood of renames and reformats doesn't generate a single doc edit or a single writer call. The agent is quiet exactly when it should be.

It catches drift no single diff shows. Pulling the doc, the code, and the config into one context turned a doc-writer into a cross-file consistency check — the override that no longer matches its default gets flagged even when the commit only touched one side.

Tuning is a config edit. Prompts in YAML means adjusting the agent's judgment doesn't touch a line of Python. The thing that most needs iterating is the thing that's easiest to iterate.

When this is overkill

I'd be selling something if I pretended every project needs this.

Small or young codebases. If your docs and code are both small enough to hold in your head, the discipline of updating them by hand is cheaper than building and maintaining a pipeline. This earns its keep when there are many documents, changed by many people, often enough that drift is inevitable.
Docs without a stable link to code. The entire deterministic mapping rests on an authoritative identifier shared between doc and code. If your docs are free-form prose with no structural tie to the symbols they describe, you'd have to introduce that convention first — and that's the real prerequisite, not the agent.
Prose that isn't fact-bearing. This shines for documentation full of checkable facts — thresholds, conditions, inputs, outputs. For narrative docs (tutorials, design rationale, "why we built it this way"), there's no config to anchor against and the value drops sharply.

Two honest costs even where it fits. The model is a real dependency in your commit path, so you need the fail-open behavior and you need to mean it. And a generated rewrite still demands a human read — the agent makes the first draft and finds the right doc, but skipping the review just relocates the trust problem rather than solving it.

The bottom line

The agent that keeps my docs honest is, by line count, mostly not an AI agent at all. It's git plumbing, a regex-built index, an evidence-grading resolver, and a config-slicer — and then, at the very end, two small model calls that do the two things code can't: decide whether a change means anything, and write the change down in good prose.

That ratio is the actual lesson, and it generalizes far past documentation. The temptation with a capable model is to hand it the whole problem and admire how much it can do. The reliable move is the opposite: shrink the model's job until it's only the irreducibly fuzzy part, and build boring, deterministic, testable code around it for everything else. The model gets more reliable as its responsibility gets narrower.

Put the check at the moment of change, keep a human in the loop, fail open, and let plain code do everything plain code can do. The first time you change a threshold and watch the right paragraph in the right document rewrite itself — correctly, with the real number, before the commit even lands — you stop thinking of your docs as something that rots. They became part of the build.