AI-generated documentation: Keeping it accurate

How three developers with different AI workflows tackled AI-generated documentation, documentation drift, and centralized knowledge.

Three developers, different workflows

We are a team of three developers working across several software repositories. Like many development teams today, we've incorporated AI coding assistants into practically every stage of our workflow.

Interestingly, we don't all use them in the same way.

One of us uses Spec Kit, another relies on Addy Osmani's agent skills, and the third follows a relatively standard Plan/Build workflow with Codex. The first two workflows use Claude.

Each approach has its own philosophy, conventions, and ways of structuring work. Some are more specification-driven, others rely more heavily on planning and agent instructions, while still others leave more room for the model to figure out the implementation.

We didn't initially consider this diversity a problem. Different developers have different preferences, and these tools are evolving quickly. We were interested in finding workflows that worked well for each of us.

But as we continued working, we encountered a problem that deserved more attention: keeping AI-generated documentation accurate and consistent with the code.

When AI starts documenting everything

AI coding assistants don't just generate code. They also generate comments, specifications, implementation plans, architecture descriptions, and other documentation.

This is understandable. These tools benefit from context, and producing documentation is one way to preserve that context for future sessions.

However, we started noticing that AI-generated documentation was becoming a significant part of what our agents produced and maintained.

A feature could involve changes to the implementation, inline comments, a specification file, and one or more documents under a docs directory. Sometimes the same information appeared in multiple places.

This introduced a new maintenance problem.

Documentation can become outdated independently of the implementation. A specification might describe an earlier version of a feature, while the code has evolved. An implementation plan might remain in the repository long after the work has been completed. An agent working on a new task might read those files and treat outdated information as authoritative.

In other words, AI can make documentation generation very cheap, but it doesn't make documentation maintenance free.

In fact, it can increase the amount of documentation that needs to be maintained.

The specification dilemma

This led us to reconsider the role of specification files.

Specification-driven workflows have obvious advantages. They provide a structured description of requirements, preserve decisions, and give agents a relatively clear starting point.

But they also introduce another representation of the system that needs to remain consistent with the implementation.

When a feature changes, what happens to its specification? Who updates it? Does every agent understand which files are authoritative? What happens when a different developer uses a different workflow?

We found ourselves facing a trade-off.

Keeping detailed specifications provides useful context, particularly for complex features. But maintaining them indefinitely can become expensive. Removing them reduces duplication but also means losing some of the reasoning and context behind implementation decisions.

There isn't necessarily a universal answer. Some specifications represent durable business rules or architectural decisions and are worth keeping. Others are essentially temporary implementation artifacts and have a shorter useful life.

The distinction matters.

A document describing an important business constraint is different from a document describing the sequence of steps an agent followed to implement a feature.

The former can remain useful long after implementation. The latter can become obsolete almost immediately.

We started thinking less about whether to keep or remove specification files and more about what information deserved a permanent home, who owned it, and how agents could find it.

Several repositories, three developers, and one knowledge base

Our setup added another dimension to the problem.

We work across several software repositories. Each developer needs a different subset of them, depending on the projects and tasks they're working on.

Rather than combining all the projects into a monorepo, we maintain a separate root repository dedicated to documentation and supporting scripts.

Each developer brings together the repositories they need in their local working environment. These project repositories are Git-ignored in the root repository, so everyone can have a different local setup without coupling the version control histories of the individual projects.

The root repository has a different purpose: it provides a centralized place for knowledge that spans projects and repositories.

As we explored ways to manage AI-generated documentation across repositories, we eventually adopted Obsidian as our common knowledge base.

Instead of distributing broader architectural knowledge, cross-project documentation, and decisions across multiple docs directories, we started centralizing them in this repository.

The individual repositories could still contain their own README files and relevant local documentation, but the broader knowledge about the system had a centralized location.

A single root repository holds the centralized knowledge base and its checks, while project repositories stay independent and point back to it.

There is also a detail in our setup that turned out to be particularly important.

The README in each individual repository explicitly explains our centralized documentation approach. It tells AI agents that broader documentation lives in the root repository and that changes to the implementation may require updating that centralized knowledge base.

This matters because an agent session might be started from inside any individual repository. Without that instruction, the agent might reasonably assume that the repository's own docs directory is the right place to put new documentation.

The README provides the missing context, even when the session starts far away from the root repository.

It's a relatively simple convention, but it helps bridge the gap between different repositories and different AI workflows.

Documentation can have deterministic checks too

Centralizing AI-generated documentation solves only part of the problem. We still need to make sure that the knowledge base remains consistent as it evolves.

One relatively simple example is link integrity.

When documentation is distributed across multiple files, links become part of its structure. Moving or renaming a document can leave references pointing nowhere. This is particularly easy to overlook when AI agents create and reorganize documentation.

We wrote a script that checks documentation links and detects dead references.

Importantly, this isn't just a utility that developers can run when they remember to do so. It runs as part of our CI pipeline.

This is a small but meaningful distinction.

Instead of relying on developers or AI agents to remember to verify documentation links, we have a deterministic check that enforces this condition automatically.

It doesn't guarantee that the documentation is accurate, complete, or useful. Those are harder problems that still require judgment. But it eliminates one specific category of inconsistency.

We find this approach interesting because it applies the same principle used in software quality checks to something that is often treated as secondary: documentation.

As AI-generated documentation becomes more common, having automated ways to verify at least some of its properties becomes increasingly useful.

Comments: a different role in a different development process

Another interesting aspect of AI-assisted development is how we think about inline comments.

Before AI coding assistants became commonplace, comments were often frowned upon.

The reasoning was familiar: code should speak for itself. Comments that merely repeat what the code does diminish readability. Instead of explaining an implementation, they force the reader to process two representations of the same thing.

That reasoning still holds.

But the economics of software development have changed.

When developers write most of the code themselves, they naturally spend a significant amount of time reading and understanding their own implementation. Today, a growing portion of that implementation can be generated by an AI assistant.

The time humans spend manually reading every line of code has decreased. Our attention is increasingly directed toward a different set of questions:

1. Does the implementation actually satisfy the requirements?
2. Are the architectural decisions appropriate?
3. Are design patterns being used correctly, or is the implementation unnecessarily complicated?
4. Does the agent understand the constraints and trade-offs of the current stage of the product?
5. Are there implicit assumptions that need to be made explicit?

This changes the potential value of comments.

A comment explaining why a particular decision was made can be more valuable than one explaining what a line of code does.

Comments can preserve context that isn't immediately apparent from the implementation: a business constraint, a workaround for an external system, a deliberate architectural compromise, or a decision that might otherwise look incorrect to someone reviewing the code.

Of course, AI-generated comments can also become noise. A comment that simply paraphrases the implementation is still unnecessary, regardless of who wrote it.

The point isn't that comments are suddenly always good. It's that their role deserves to be reconsidered in a development process where implementation and human review have different economics.

What changes when code becomes cheap?

Our experience has made us think about the changing role of software developers.

When AI can generate substantial portions of an implementation, the bottleneck shifts.

Writing code is still important, but more of our attention goes toward understanding requirements, evaluating architectural decisions, reviewing trade-offs, and preserving the knowledge that helps us make those decisions consistently.

AI-generated documentation becomes part of this equation. We need to decide which knowledge deserves to be preserved, where it belongs, and how agents can access it without creating multiple conflicting representations.

We also need to think about how our tools interact with that knowledge. A developer can have a preferred AI assistant and a preferred workflow, but the agent still needs to understand the conventions of the project and the location of its documentation.

We didn't end up standardizing on one AI development workflow. Instead, we established a shared documentation structure and a few conventions that allow different workflows to coexist.

Different AI workflows can coexist when they share documentation conventions and a single, verifiable source of knowledge.

Perhaps that's one of the more interesting changes brought by AI-assisted development: the process of producing software can become increasingly heterogeneous, while the knowledge needed to maintain it still benefits from being centralized, explicit, and verifiable.

AI makes producing documentation easier. Keeping it useful is still an engineering problem.