llms-full.txt Guide: When a Site Needs a Larger Context Bundle

AI Summary: llms-full.txt is an aggregated context bundle designed for deep-reasoning AI models and autonomous agents. Rather than blindly concatenating an entire domain, a high-quality llms-full.txt bundle curates mission-critical architecture documents, API schemas, and SDK usage patterns into a structured, provenance-annotated Markdown stream that maximizes prompt cache hits.

#The Semantic Boundary: When to Choose llms-full.txt Over llms.txt

The distinction between llms.txt and llms-full.txt is architectural, not merely stylistic. Choosing between them depends on your target agent interaction model:

1. When to serve llms.txt: When an agent operates in an iterative, exploratory loop (such as an IDE agent writing a small feature or fixing a bug). The agent inspects the concise /llms.txt routing table, identifies the single relevant API document (e.g. /docs/auth.md), and fetches only that document. Total context consumption: ~2,000 tokens.
2. When to serve llms-full.txt: When an agent undertakes an end-to-end architectural migration, complex refactoring, or zero-shot code generation without tool-calling internet access. The model ingests the entire documentation suite upfront, relying on self-attention across the complete API surface. Total context consumption: 50,000 to 180,000 tokens.

#The Compilation Pipeline: Sanitizing the Markdown Stream

Never dump raw repository files or unparsed HTML into llms-full.txt. A robust compilation pipeline must apply four transformation passes:

[Raw Documentation] 
        ↓ 
  1. Frontmatter Strip (Strip Jekyll/Nextra/Docusaurus YAML headers)
        ↓ 
  2. AST Sanitization (Strip interactive JSX components, tabs, and embeds)
        ↓ 
  3. Provenance Injection (Prepend canonical URL, commit hash, and updated date)
        ↓ 
  4. Token Accounting (Verify total tokens remain under the target budget threshold)
        ↓ 
[public/llms-full.txt]

#Production Compilation Script (TypeScript / Node.js)

// scripts/compile-llms-full.ts
import fs from 'node:fs'
import path from 'node:path'
import matter from 'gray-matter'

interface DocSection {
  title: string
  canonicalUrl: string
  filePath: string
}

const SECTIONS: DocSection[] = [
  { title: 'Core Architecture', canonicalUrl: 'https://acme.dev/docs/architecture', filePath: 'docs/arch.md' },
  { title: 'Authentication & Tokens', canonicalUrl: 'https://acme.dev/docs/auth', filePath: 'docs/auth.md' },
  { title: 'REST API Reference', canonicalUrl: 'https://acme.dev/docs/api-spec', filePath: 'docs/api.md' },
]

export function compileBundle(): string {
  const chunks: string[] = [
    '# Acme Platform — Consolidated Technical Documentation\n\n',
    '> Machine-readable context bundle for AI models and autonomous coding agents.\n',
    `> Generated: ${new Date().toISOString()} | Spec: llms-full.txt v2.1\n\n`,
  ]

  for (const sec of SECTIONS) {
    const raw = fs.readFileSync(sec.filePath, 'utf8')
    const { content } = matter(raw) // Strips YAML frontmatter

    // Remove JSX component tags often found in MDX (e.g. <TabGroup>, <CardGrid>)
    const sanitized = content.replace(/<[A-Z][A-Za-z0-9.]*(\s[^>]*)?\/>/g, '')
                             .replace(/<[A-Z][A-Za-z0-9.]*(\s[^>]*)?>[\s\S]*?<\/[A-Z][A-Za-z0-9.]*>/g, '')

    chunks.push(`---\n## Section: ${sec.title}\n`)
    chunks.push(`- Canonical Source: ${sec.canonicalUrl}\n\n`)
    chunks.push(sanitized.trim() + '\n\n')
  }

  return chunks.join('')
}

fs.writeFileSync('public/llms-full.txt', compileBundle(), 'utf8')
console.log('llms-full.txt compiled successfully.')

#Guardrails: Context Poisoning & Token Saturation

When serving a monolithic text bundle, keep these failure modes in check:

1. Context Poisoning via Stale Examples: If a deprecated v1 API endpoint is left inside llms-full.txt alongside v2 endpoints, the model will frequently mix parameter types or recommend superseded security headers. Ruthlessly prune deprecated methods from the bundle.
2. Secrets & Internal URLs: Never concatenate markdown files containing test credentials, .env.local snippets, or internal staging URLs (http://staging.internal.vpc).
3. Budget Hard Caps: If your bundle exceeds 150,000 tokens, even frontier models experience reasoning latency spikes. If your docs exceed this threshold, split your bundle into scoped domain files (e.g. /docs/llms-full.txt and /api/llms-full.txt).

#Related guidance

To structure your root index, review our llms-full.txt core architecture and What is llms.txt? foundational guide, and ensure your repository adheres to AGENTS.md best practices.

#References

• The /llms.txt Specification Proposal: Official conventions for full-context bundles and source URL tracking.
• OpenAI Tokenizer & Context Architecture: Token counting formulas across cl100k and o200k byte-pair encodings.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.