A Vectorless RAG Architecture for Online Documentation: Using Retrieval-Optimized Markdown Build Artifacts

Introduction – Why AI Systems Need Inference-Time Retrieval Capabilities

The operational knowledge of a software product is typically captured in web-based documentation systems such as online documentation sites and wikis.

As organizations undergo AI transformations, engineering teams are increasingly tasked with enabling AI systems to leverage these knowledge sources. The goal is to automate internal workflows (such as AI-assisted code generation) and external processes (such as customer support question answering) while ensuring that AI-generated outputs are grounded in accurate and up-to-date information.

If documentation is publicly crawlable on the web, portions of it may have been encoded in the parameters (weights) of foundation models during pre-training. Documentation that resides behind authentication boundaries or within embedded product help systems is generally unavailable during pre-training and therefore cannot be assumed to exist in the model’s parametric knowledge. However, even for publicly accessible documentation, relying solely on a model’s parametric knowledge introduces several limitations:

1. There is no guarantee that a particular model has encountered, retained, or can accurately recall a given piece of information.
2. Software documentation evolves continuously. Omitting recent updates can result in outdated, inaccurate, or hallucinated responses.

These limitations have shaped how modern AI systems are designed. Rather than expecting model scale alone to encode and retain this information, they increasingly complement the foundation model with external knowledge sources. Those sources are consulted at inference time, through tool use interleaved with reasoning.

The same challenges apply to organizations that fine-tune or continually retrain models on their own documentation. Moreover, repeated adaptation of model weights can introduce catastrophic forgetting, where previously learned knowledge may be degraded as new information is incorporated.

Consequently, software vendors must establish robust retrieval mechanisms that provide AI systems with reliable access to complete, current, and authoritative product documentation.

Challenges of Online Documentation Retrieval

Before exploring a more robust architecture, it is useful to examine the retrieval patterns that teams typically implement first. These approaches are attractive because they are relatively easy to understand and deploy, making them common choices for prototypes and early production systems. However, they also reveal many of the practical challenges associated with documentation retrieval. While more sophisticated retrieval architectures can improve retrieval quality, they do not eliminate the underlying challenge that documentation must be structured and maintained for effective AI consumption. Understanding these limitations provides the motivation for the architecture proposed in section An Alternative Architecture Inspired by llms.txt.

Pattern 1: Naïve Vector-Based RAG over Raw Documentation

A common initial approach is to ingest raw web-based documentation into a vector database and expose vector search functions to AI agents via the Model Context Protocol (MCP). While straightforward, this architecture introduces several challenges:

1. Token Waste: Domain knowledge is surrounded by frontend source code (HTML, CSS, JavaScript), navigation menus, cookie banners, sidebars, and other presentation-layer artifacts. Unless carefully filtered, these elements increase token consumption, reduce retrieval quality, and waste valuable context window capacity.

2. Lack of Retrieval Optimization: Downloaded documentation is usually not written in a form optimized for retrieval. Consider the following example:

   Original (Discouraged)

   Refer to section 3.2 for authentication details.

   Retrieval-Optimized (Preferred)

   Authentication uses JWT tokens signed with RS256. The complete authentication flow is detailed in section 3.2.

   The second version directly exposes relevant information and is therefore much easier for retrieval systems to discover. Modern retrieval systems increasingly address this challenge through techniques such as contextual retrieval, where an LLM generates document-aware context for each chunk before indexing. Combined with hybrid retrieval and reranking strategies, such approaches can significantly improve retrieval performance. However, Anthropic’s formulation(opens in a new tab) passes the contextualizer a chunk and its source document: the context it adds is document-local, not a model of cross-document relationships. In the example above, the referenced section is frequently a different page the contextualizer never sees. Performing the same expansion during the documentation build resolves that cross-reference while it is still a link, and publishes the result instead of holding it in a separate index.

   In addition, documentation written for human readers often relies on diagrams, screenshots, charts, and tables. Traditional embedding pipelines often ignore or inadequately process these assets, resulting in knowledge loss.

3. Synchronization Overhead: The vector database must remain synchronized with the source documentation, creating operational complexity beyond the initial indexing process. Documentation changes may require expensive re-embedding operations, while renamed or deleted pages can leave stale vectors that continue to appear in retrieval results. Furthermore, the documentation release process and the index update process are often not transactionally coupled, creating a window where published documentation and the retrieval index silently diverge.

4. Retrieval Parameter Tuning: Vector-based RAG systems depend on parameters such as:

   • Chunk size
   • Chunk overlap
   • Top-k retrieval count
   • Similarity thresholds

   Determining a universally optimal configuration is difficult because retrieval quality depends heavily on the structure, size, and distribution of the underlying documentation corpus. Poor parameter choices can either omit relevant information or inject out-of-scope context, increasing token consumption while reducing response quality.

The challenge of extracting meaningful content from web pages is largely solved today. Open-source libraries such as Crawl4AI(opens in a new tab) can crawl websites and convert pages into Markdown suitable for AI consumption.

The issue of processing visual assets can be partially mitigated by inserting a multimodal (vision-language) model into the ETL pipeline. This introduces additional infrastructure and inference costs; however, the processing cost is typically paid once per documentation revision and amortized across all future retrieval requests. Unlike query-time crawling or document transformation, the work does not need to be repeated for every user interaction. Furthermore, vision-language models can still produce imperfect descriptions of complex diagrams, where human-authored textual alternatives often remain superior.

Because of the synchronization burden, some teams abandon the vector database entirely and instead crawl on every query. This is impractical for several reasons. It increases latency, repeats expensive processing steps on every request, and relies primarily on lexical filtering, which lacks semantic understanding.

Pattern 2: Direct Documentation Repository Access

To avoid the complexity of continuous synchronization and the difficulty of choosing appropriate retrieval parameters, some developers simply grant AI agents read access to the Git repositories that contain the documentation sources. The agent clones the repository into a local workspace and is typically guided by a dedicated agent skill (SKILL.md). These instructions define where documentation resides, which content should be ignored, and how updates should be processed.

This approach eliminates the need for crawling, embedding, and indexing pipelines while significantly simplifying synchronization.

However, this pattern is only viable when documentation is maintained under Git version control—a requirement that many documentation systems cannot satisfy. It also assumes that repository access can be restricted to read-only permissions.

A less obvious drawback is that documentation repositories do not necessarily reflect the state of the currently deployed product. Depending on the development workflow, repositories may contain information about upcoming releases, experimental features, or work-in-progress functionality. In real-world software delivery processes, the repository state and the production state often diverge, sometimes significantly. While this may be desirable for engineering-focused use cases, it can lead to inaccurate responses in customer-support scenarios, where answers should reflect only the functionality available in production.

Checking out a release tag can reduce version drift, provided the documentation repository is versioned and tagged in lockstep with product releases. In practice, however, many documentation workflows do not maintain such guarantees. Published documentation may diverge through hotfixes or CMS-side edits.

The same characteristic also introduces security and governance concerns. Granting AI agents direct access to documentation repositories may inadvertently expose unreleased product features, development branches, historical revisions, or other internal metadata that were never intended to be publicly accessible.

An Alternative Architecture Inspired by llms.txt

To address these limitations while establishing a long-term, maintainable approach, I propose a documentation strategy inspired by the llms.txt(opens in a new tab) initiative.

The llms.txt proposal, introduced by Jeremy Howard in 2024, suggests that websites expose an /llms.txt file containing:

• Background information
• Navigation guidance
• Links to LLM-friendly Markdown resources

This file offers a simplified view of documentation navigation, allowing agents to discover and fetch the relevant pages. A practical example can be found in the deployed Claude Code llms.txt(opens in a new tab).

In addition, the broader practice inspired by llms.txt is to serve clean Markdown representations alongside standard web pages at predictable URLs. For instance, requesting a URL with an appended .md extension can return a Markdown representation of the documentation page rather than rendered HTML. Current implementations primarily provide a faithful Markdown mirror of the human-facing page—a substantial improvement over HTML scraping, but still primarily a format conversion. The approach proposed in this article extends this idea by treating the Markdown representation as a retrieval-optimized knowledge artifact.

The resulting URL convention looks as follows:

Original web route

https://docs.example.com/auth/login

LLM-optimized route

https://docs.example.com/auth/login.md

Documentation index

https://docs.example.com/llms.txt

At the time of writing, llms.txt should be viewed as an emerging practice rather than an established web standard such as robots.txt or sitemap.xml. Interestingly, adoption appears asymmetric. According to an Ahrefs analysis(opens in a new tab) of 137,210 domains with measurable traffic in May 2026, approximately 28% publish an llms.txt file. Ahrefs explicitly notes that this figure should be treated as an upper bound, since its customer base is more technically sophisticated and SEO-aware than the web at large. Of the sites that published an llms.txt file, 97% received no requests at all during May 2026. This asymmetry is partly explained by the way adoption has occurred. Platforms such as Mintlify(opens in a new tab) automatically generate llms.txt and llms-full.txt files for hosted documentation sites, causing many deployments to expose these artifacts without requiring individual publishers to make an explicit architectural decision. As a result, publication volume should not be interpreted as evidence that AI systems actively discover and consume root-level llms.txt indexes. Nevertheless, the underlying idea—providing structured, LLM-oriented documentation artifacts—is sound. It motivates a broader strategy: treating AI-oriented documentation as an explicit build artifact rather than relying solely on passive discovery mechanisms.

LLM-Oriented Documentation as a Build Artifact

The core idea is simple. Instead of continuously attempting to transform human-oriented documentation into machine-oriented knowledge at runtime, organizations should treat the LLM-oriented representation as a first-class artifact of the documentation build process.

The technical writer maintains a single authoritative documentation source. During the documentation build process, this source is transformed into both the human-facing HTML representation and an LLM-optimized Markdown representation. The Markdown variant applies AI-oriented transformations such as representing tables as clean Markdown structures, enriching images with explicit descriptive text, expanding cross-references into self-contained summaries, and removing presentation-layer overhead. The objective is not the smallest possible document, but the highest information density per token: content that consumes context because it carries meaningful knowledge rather than because it contains markup, navigation elements, or redundant presentation artifacts. Figure 1 shows the resulting production plane: one authoritative source, two final artifacts, two audiences.

For public documentation, this introduces no additional exposure. However, enterprise documentation systems must ensure that generated LLM-oriented artifacts inherit the same authentication and authorization policies as their human-facing counterparts. Generating an additional AI-friendly representation should not create an alternative access path that bypasses existing documentation controls.

This architecture aligns well with modern documentation toolchains, which already operate on a single-source publishing model. Platforms such as Mintlify automatically generate AI-oriented artifacts including llms.txt, llms-full.txt, and Markdown representations(opens in a new tab) of individual documentation pages. Documentation frameworks such as Antora already provide extensible build pipelines(opens in a new tab) that can generate additional outputs from the same documentation source. Although support for fully retrieval-optimized documentation remains immature, the underlying single-source architecture is already well established.

The same approach also applies to embedded and offline documentation systems. The generated Markdown artifacts can be packaged alongside desktop applications, enabling AI agents to retrieve knowledge that was never publicly accessible during model pre-training. In such environments, retrieval occurs against the local documentation corpus rather than against cloud-hosted knowledge sources, preserving both functionality and data sovereignty.

For high-value content such as troubleshooting guides or architecture documentation, organizations may additionally provide manually curated LLM-facing overrides where retrieval quality justifies the additional effort. These overrides should remain part of the same documentation pipeline, validated against the authoritative source, and regenerated or reviewed when upstream content changes to prevent silent divergence.

Resource Discovery via Progressive Disclosure

An agent needs an entry point into the corpus. Rather than depending on a still-emerging public /llms.txt convention, the architecture points the agent at the index its own documentation build generates, alongside the Markdown corpus itself, as shown in Figure 1. Because that index ships with the release that produced it, it cannot drift from the pages it lists, and it can be version-scoped per supported release.

Discovery is deliberately not performed by the agent answering the question. A dedicated research skill, shown in Listing 1, runs in a forked subagent: it reads the index, fetches the pages it selects, and returns a grounded, cited answer. The pages themselves are never handed back, so the index, the candidate paths, and everything rejected along the way stay inside a context that is discarded when the search ends. This architecture separates navigation context from the main context.

---
name: claude-code-documentation
description: >-
  Use this skill to answer questions about Claude Code
  from the official documentation.
allowed-tools: WebFetch
context: fork
agent: Explore
background: false
---

# Claude Code documentation retrieval

1. Fetch the documentation index: https://code.claude.com/docs/llms.txt
2. Select the smallest set of pages that answers the question.
3. Fetch only those pages, then answer and cite them.

For larger corpora the same skill escalates rather than changes shape: hierarchical maps feed semantic or hybrid retrieval when direct Markdown loading is no longer tractable. That escalation stays inside the research subagent, so it never enlarges the reasoning context either. Figure 2 traces the complete flow, from skill invocation through to the grounded response.

Table 1 summarizes how the proposed architecture addresses the limitations discussed earlier.

Architectural Benefits

• Minimal Markup Overhead: Markdown documents contain only relevant content and avoid HTML, CSS, JavaScript, and other presentation-layer artifacts.
• Retrieval-Optimized by Design: The LLM-oriented representation is generated specifically for efficient retrieval and AI consumption. Documentation teams can define transformation rules that expand cross-references, enrich visual content with textual descriptions, and improve the discoverability of important information.
• Simplified Synchronization: Because both the HTML and LLM-oriented Markdown representations are generated from the same authoritative source, synchronization becomes a build-time concern rather than an operational retrieval problem.
• Human-Defined Knowledge Boundaries: The agent consumes complete, cohesive Markdown files, reducing reliance on arbitrary character-based splitting. While this does not eliminate chunking entirely, it replaces generic algorithmic chunking with human-defined knowledge boundaries that often align better with the structure of technical documentation.
• Curated Knowledge Exposure: Irrelevant pages such as legal notices, privacy policies, or imprint pages can be excluded from the resource map entirely.

Conclusion – Building Documentation for Humans and AI

Traditional RAG implementations often attempt to transform human-oriented documentation into machine-readable knowledge through automated extraction, chunking, and indexing pipelines.

An alternative approach is to treat LLM-oriented documentation as a first-class build artifact generated alongside the human-facing version from a shared authoritative source. This shifts part of the effort from retrieval engineering into the documentation build, and toward knowledge authoring where curated overrides are used. The approach is not based on a fundamentally new retrieval algorithm; instead, it establishes a high-quality, maintainable foundation for grounding enterprise AI agents in dynamic product documentation. The result can be improved retrieval quality, reduced operational complexity, better synchronization, and more efficient use of AI context.

This approach is gaining adoption across parts of the software ecosystem. For example, platforms such as Angular now provide an llms.txt file alongside their core documentation (see Angular: providing context with llms.txt(opens in a new tab)).

Rather than replacing vector databases entirely, this architecture defines a better knowledge layer for retrieval systems. At larger scales, where navigation structures exceed practical context limits, semantic or hybrid retrieval should operate over the LLM-oriented Markdown corpus rather than over raw HTML or scraped web content. This preserves the benefits of retrieval systems while ensuring that the indexed representation is already optimized for AI consumption.