LOADING PAGE
Preparing your page.
Your site data stays in this session while the page loads.
FORMAT INDEX
AI Format Standards & Specifications Library
Complete technical references and architecture guides for publishing machine-readable documentation, curated token bundles, and AI crawler preference files.
25 technical guides
4 core file standards
Static pre-rendered pages
All Format Guides
25 published articlesai.txt Standard: Declaring AI Usage Preferences Without OverclaimingEngineering and legal-tech deep dive into the IETF ai.txt standard draft, covering directive syntax, JSON schemas, edge deployment, and EU TDM reservation mechanics.proposedai.txt vs robots.txt: AI Training Consent vs Scraping GovernanceWhy robots.txt fails to govern generative AI training and how ai.txt provides legally actionable, machine-readable licensing and inference boundaries.proposedai.txt: a place for AI-use preferencesArchitecture and governance guide for authoring, publishing, and declaring machine-readable AI usage preferences and training boundaries via /ai.txt.experimentalAPI Documentation for LLMs: Make OpenAPI Easy to InspectArchitectural guide for structuring OpenAPI 3.1 schemas and Markdown API specifications for flawless LLM tool-calling and autonomous agent execution.broadly supportedAvoiding Context Pollution: Keep AI Bundles RelevantEngineering protocols for purging stale APIs, duplicated prose, and conflicting documentation from LLM context streams to prevent context rot.broadly supportedContext Window Limits: Design for 128k and 2M TokensArchitectural guide for scaling documentation across 128k, 500k, 1M, and 2M token context windows, mitigating attention degradation, and engineering prompt caching.broadly supportedJSON vs Markdown for AI: Choose the Consumer ContractArchitectural comparison of JSON and Markdown for LLM ingestion: token density benchmarks, JSON Schema validation, and RAG retrieval mechanics.broadly supportedllms-full.txt Guide: When a Site Needs a Larger Context BundleArchitectural blueprint for engineering, compiling, and deploying production llms-full.txt bundles without blowing context budgets or corrupting agent reasoning.proposedllms-full.txt vs RAG Vector DB: Flat Context vs Vector OverheadArchitectural evaluation of monolithic flat-file markdown bundles versus vector database RAG pipelines in the era of 2M-token frontier context windows.broadly supportedllms-full.txt: selected pages in one fileArchitectural guide for assembling, tokenizing, and deploying a unified llms-full.txt context bundle for massive context window LLMs and autonomous agents.proposedllms.txt vs llms-full.txt: Index Routing vs Monolithic ContextArchitectural comparison between lightweight index routing and monolithic full-text context bundles: token economics, attention saturation, and cache hit rates.broadly supportedllms.txt vs robots.txt: Context Map or Crawler Rule?Architecture breakdown contrasting llms.txt and robots.txt: understanding crawler permissions, crawl budgets, semantic routing, and context window limits.proposedllms.txt vs sitemap.xml: Semantic AI Graph vs Search Engine Crawl TreeWhy XML sitemaps fail autonomous agents and how llms.txt provides the semantic documentation hierarchy required for deterministic tool calling.broadly supportedllms.txt: a short map of your websiteArchitecture and specification guide for designing, serving, and automating a high-precision /llms.txt endpoint for frontier AI models and autonomous agents.proposedllms.txt Directory Structure: Make Scope and Links DeterministicArchitectural blueprint for organizing root and scoped llms.txt files across monorepos, micro-frontends, versioned docs, and multi-tenant platforms.proposedMarkdown for RAG: Structure Documents for Better RetrievalArchitectural guide for structuring technical Markdown to maximize vector embedding fidelity, semantic chunking precision, and RAG retrieval recall.broadly supportedMarkdown vs HTML for AI Ingestion: Token Bloat & Noise RatiosEmpirical benchmark comparing raw HTML DOM trees with sanitized Markdown ASTs: token expenditure, attention accuracy, and agent tool-calling reliability.broadly supportedPrompt Caching vs Semantic Chunking: Monolith vs EmbeddingFinancial and architectural comparison between monolithic prompt-cached documentation bundles and semantic vector chunking retrieval pipelines.broadly supportedRAG-Friendly Documentation: Make Each Passage Stand AloneArchitectural blueprint for authoring technical documentation optimized for vector retrieval, cross-encoder rerankers, and autonomous agent tool use.broadly supportedSemantic Chunking: Split Documents at Topic BoundariesEngineering guide to semantic chunking algorithms, embedding distance metrics, percentile thresholding, and hybrid AST-semantic retrieval pipelines.broadly supportedStatic vs Dynamic Edge llms.txt: CDN Hosting vs Context SlicingArchitecture review of pre-rendered static llms.txt files versus edge-computed dynamic endpoints: token budget tailoring, multi-tenant auth, and caching SLAs.proposedThe Future of the Machine-Readable Web: Build on Clear LayersArchitectural blueprint for the Dual-Stack Web: delivering visual HTML for humans while providing deterministic Markdown, JSON-LD, and OpenAPI for autonomous AI agents.proposedThe Role of ai.txt: State Preferences Without OverclaimingStrategic engineering and governance guide on deploying ai.txt: IETF AIPREF standards, AI training boundaries, edge WAF coordination, and enterprise copyright compliance.experimentalToken Budgeting for llms-full.txt: Keep Context UsefulProduction guide for architecting context budgets: BPE tokenization mechanics, tool calling overhead, KV prompt caching thresholds, and attention degradation.broadly supportedWhat Is llms.txt? A Practical Format for Machine-Readable ContextA comprehensive technical breakdown of the /llms.txt specification, explaining its token efficiency, edge routing mechanics, and role in agentic AI architectures.proposed
Check your domain against these standards.
Run a safe, zero-retention diagnostic on your site's public signals, robots.txt, sitemap, and AI files.