GitHub Spec Kit

Test Driving GitHub Spec Kit: Living Documentation

I was skeptical. After years of dealing with outdated documentation, the promise of "living documentation" sounded like another attempt to solve an age-old problem. Then I tried it on a production NuGet package. Here's what actually happened.

Executive Summary

Every developer knows the pattern: Design document says one thing, code does another, six months later nobody knows which is correct. Waterfall specs died when code changed. Agile seems to have thrown out specs entirely and instead focused on incremental changes with little updates to the full feature set documentation. Both failed for the same reason—humans won't maintain documentation when it's divorced from implementation.

GitHub Spec Kit closes the feedback loop: AI agents update documentation when implementation diverges, so specs become living artifacts instead of shelf-ware. The ROI isn't faster development—it's specs that still accurately describe the codebase three months later.

What You'll Learn
  • Solving documentation drift: How AI agents close the feedback loop humans never could—specs stay synchronized with implementation
  • Institutional knowledge persistence: Path resolution patterns, warning remediation strategies, architectural decisions captured in markdown that survives team turnover
  • The feedback cycle: AI generates → human fixes → human tells AI to update specs → knowledge persists forever
  • When it matters: Libraries, APIs, multi-year projects where tribal knowledge creates single points of failure
  • Real metrics: Implementation: 7 hours (same as always). Documentation sync: 20 minutes (vs. never). Result: Zero documentation debt.
Who This Is For
  • Solutions Architects: Translate business requirements into technology with precision
  • Development Teams: Escape the prompt-generate-debug cycle with structured workflows
  • Engineering Leaders: Build institutional knowledge that scales beyond individual contributors
  • .NET Developers: Practical patterns for NuGet packages, documentation, and quality enforcement
References

The Documentation Drift Problem

Every codebase has specs that lie. They said one thing at design time, developers changed it during implementation, and nobody updated the docs. Waterfall tried to solve this with upfront perfection—specs froze before coding started. Agile gave up entirely—"working software over comprehensive documentation."

Both failed for the same reason: humans won't maintain documentation when it's divorced from implementation. The feedback loop is too expensive.

I spent 8 years at EDS maintaining three-ring binders of waterfall specs. The specs were beautiful at handoff. Three months later, they were fiction. The human cost of keeping specs synchronized with code was unsustainable.

What If AI Could Close the Loop?

GitHub Spec Kit offers something different: AI agents that update documentation when implementation changes.

The cycle: AI generates code from spec → human fixes what's wrong → human tells AI "update the specs to match reality" → specs evolve instead of ossifying.

I tested this on a production NuGet package. Two features, 7 hours of work, 136 files changed. Every deviation from the original plan became a permanent improvement to the specs—not tribal knowledge that disappears when I leave. Here's what happened.

What Is Spec Kit?

GitHub Spec Kit is a framework that creates markdown artifacts during development: SPEC.md for requirements, PLAN.md for technical approach, TASKS.md for implementation steps. You use slash commands in Copilot ( /speckit.specify , /speckit.plan , etc.) to generate these files.

The pitch: When implementation inevitably deviates from the plan, you tell the AI to update the specs. Instead of specs rotting immediately, they evolve to match reality. That's the theory. I tested it on a real project to see if it actually works.

My Experiment: Two Real Specs

I picked WebSpark.HttpClientUtility, a production .NET NuGet package I maintain. Two features I'd been postponing: a documentation website and cleaning up compiler warnings. Perfect test cases—one creative, one tedious.

Spec 001: Build a documentation site. AI generated an Eleventy-based static site with 6 pages, NuGet API integration, and GitHub Pages deployment. It looked great. Then it broke in production because of path resolution issues.

Spec 002: Zero compiler warnings. Started with an unknown number of warnings. Goal: 0 warnings with TreatWarningsAsErrors enforced. AI tried to use #pragma warning disable suppressions. I rejected that and made it fix things properly with XML docs and null guards.

What I Learned Writing the Specs

The specs forced me to think more precisely than I usually do. For the warning cleanup, I had to define "done" upfront: zero warnings, TreatWarningsAsErrors enabled, 520 tests still passing, no pragma suppressions allowed.

Mistakes I Made
  • Too vague on baseline: I said "unknown number of warnings" instead of auditing first. AI wasted time figuring out what to fix.
  • Missing priority order: AI tried to fix everything simultaneously. Should have said: "Fix XML docs first, then null safety, then analyzers."
  • No time estimate: Without "Target: 4 hours" I lost focus during implementation.

Here's what surprised me: these mistakes became permanent improvements to the specs. After implementation, I spent 20 minutes having AI update SPEC.md to reflect what actually worked. Future features inherit those lessons.

Three Months Later: Zero Documentation Debt

I'm writing this three months after shipping those features. The real test: do the specs still match reality? I just re-read them. They do. Every constraint, every decision, every "why we don't use pathPrefix" rationale—all accurate.

Implementation took 7 hours (same as always). Documentation sync took an extra 20 minutes (updating specs after I fixed AI's mistakes). That 20-minute investment is what traditional approaches skip—and why their docs rot immediately.

Quantitative Outcomes: What Actually Matters
  • 20 minutes to sync specs (vs. never updating them in traditional approach = zero documentation debt)
  • Path resolution pattern documented (relativePath filter replaces pathPrefix—next developer won't repeat the mistake)
  • Warning remediation strategy captured (XML docs + ArgumentNullException.ThrowIfNull() pattern now team standard)
  • Test documentation philosophy formalized ("Tests are product documentation" principle added to constitution)
  • 136 files changed (29,141 insertions, 3,167 deletions across 2 production releases)
  • 520/520 tests passing (260 tests × 2 frameworks: net8.0 + net9.0 with zero regressions)

The Long-Term Value: What Happens After You Ship

Metric With Spec Kit Typical Ad Hoc Impact Over 12 Months
Implementation time 7 hours 7 hours No difference initially
Documentation sync 20 minutes (AI-assisted) Never happens × Zero technical debt accumulation
Spec accuracy after 6 months Matches implementation Fiction × New developers trust docs
Knowledge persistence Survives team turnover Walks out the door × No single points of failure
Next feature cost AI reads accurate patterns Developer reinvents solutions × Compounds with each feature
Onboarding time Read specs that match code Reverse-engineer from codebase × 3 hours saved per developer

The spec-driven approach doesn't make you faster initially—it prevents knowledge decay. When you return to the codebase a year later, or when a new team member joins, the specs accurately describe what was built and why. That's institutional knowledge, not tribal knowledge.

The Feedback Loop in Practice

Each time AI generated wrong code, I fixed it and had AI update the specs. Here's why that matters: these lessons are now permanent documentation that future developers (and AI agents) will read before making changes.

Three Implementation Lessons That Became Institutional Knowledge

Path Resolution: Spec Said One Thing, Reality Required Another
  • AI generated: Absolute paths using pathPrefix config (standard Eleventy approach)
  • What broke: GitHub Pages subdirectory deployment
  • I fixed it: Custom relativePath filter that calculates paths dynamically
  • Then I closed the loop: "Update SPEC.md and PLAN.md to document why pathPrefix failed and what works instead"
  • Result: SPEC.md now says "No environment-specific configuration." PLAN.md shows pathPrefix crossed out with the working alternative. Next developer won't try pathPrefix because the spec explains why it doesn't work.
Warning Suppression: Spec Was Too Vague
  • AI generated: #pragma warning disable directives (fastest solution)
  • Spec said: "No suppressions" but didn't say HOW to fix properly
  • I fixed it: 200+ XML docs, null guards with ArgumentNullException.ThrowIfNull()
  • Then I closed the loop: "Update SPEC.md with specific examples of acceptable vs. unacceptable fixes"
  • Result: SPEC.md now has a "✅ DO / ❌ DON'T" section. PLAN.md has a 5-step remediation strategy. TASKS.md breaks it into auditable chunks. Future features inherit this standard.
Test Documentation: Spec Didn't Ask, AI Didn't Deliver
  • AI generated: Documented library code, skipped test methods entirely
  • Spec said: "520 tests passing" but not "tests need documentation"
  • I fixed it: Added XML docs to 260 test methods explaining WHAT and WHY
  • Then I closed the loop: "Update SPEC.md to require test documentation. Add principle to CONSTITUTION.md: 'Tests are product documentation.'"
  • Result: Every future spec inherits "tests need docs" standard. AI reads the constitution before generating code. The team's quality bar persists beyond individual developers.

Why This Solves a 40-Year-Old Problem

In waterfall, specs froze at design and diverged immediately. In agile, we stopped writing specs because maintaining them was humanly impossible. GitHub Spec Kit closes the loop: when implementation teaches you something, you spend 20 minutes having AI update the specs. The path resolution lesson, the warning fix patterns, the test documentation standard—all permanent institutional knowledge that AI agents read before generating the next feature. That's what survives team turnover.

Frequently Asked Questions

No. The pattern is model-agnostic. Any LLM benefits from structured specs and tests.

It's complementary. Spec Kit codifies requirements and examples up front, then TDD validates them. The twist is that you're writing for humans and an AI partner simultaneously.

Break it into spec-able slices. Use research spikes to learn, then spec the actionable parts.

Tighten the spec, add failing tests for the misbehavior, and iterate. Avoid changing code and spec in opposite directions.

The Awkward Part: Updating Specs After Implementation

The Critical Step Everyone Skips

Here's the reality: after /speckit.implement completes, you'll tweak edge cases, adjust UX, and fix bugs the AI missed. This iteration is expected and normal. What's different is what you do next.

THIS is Where the Value Lives

When you finally get it right, tell the agent to update SPEC.md, PLAN.md, and TASKS.md to reflect what you actually built. This 20-minute step is what traditional approaches skip—and why documentation always becomes outdated.

Example: "I fixed the GitHub Pages path resolution by implementing a custom relativePath filter. Please update SPEC.md and PLAN.md to reflect this solution instead of the original pathPrefix approach, and explain why pathPrefix failed."

Without Feedback Loop
  • Specs describe what you planned, not what you built
  • Future developers follow outdated documentation
  • Institutional knowledge lives only in your head
  • Next feature repeats the same mistakes
With Feedback Loop
  • Specs evolve to match reality (living documentation)
  • Future developers see what actually works
  • Team learns from real-world implementation
  • Each spec becomes more accurate over time

The feedback loop keeps specs synchronized with reality. Your specs document what you built and what you learned—useful for your future self and your team.

Should You Try It?

Spec Kit won't make you ship faster initially. What it does: gives you documentation that still matches reality three months later. That's worth something if you maintain long-lived codebases or work on teams where knowledge walks out the door.

Try it if:

  • You maintain libraries or APIs where documentation debt is expensive
  • You work on teams where "ask Bob" isn't a sustainable knowledge strategy
  • You inherit codebases and wish the previous developer had explained their decisions
  • You're building something that will outlive your involvement

Skip it if:

  • You're prototyping and will throw away the code
  • You're solo and have no knowledge transfer problem
  • You're exploring and don't know what you're building yet
  • You're fixing a production fire and documentation can wait
What This Project Delivered: Persistent Knowledge
  • Path resolution pattern: Custom relativePath filter documented in specs—future developers won't repeat the pathPrefix mistake
  • Warning remediation strategy: XML docs + ArgumentNullException.ThrowIfNull() pattern captured in PLAN.md—team inherits the standard
  • Test documentation philosophy: "Tests are product documentation" principle added to CONSTITUTION.md—applies to all future features
  • Specs that match reality: Documentation updated after implementation reflects what actually works, not what was initially planned
  • Zero documentation debt: 20 minutes to sync specs vs. never updating them = institutional knowledge that survives team turnover

My verdict: Spec Kit solved a problem I've had for 20 years—documentation that matches reality months later. The trade-off is 20 minutes per feature updating specs after you fix what AI got wrong. If you maintain code beyond the initial sprint, that's a bargain.