llms.txt Directory Structure: Make Scope and Links Deterministic

AI Summary: Organizing directory structures for llms.txt requires establishing deterministic discovery hierarchies across web roots and sub-paths. By combining a root-level index at https://example.com/llms.txt with isolated scoped manifests (e.g. /docs/llms.txt or /api/v2/llms.txt), platforms guide autonomous agents directly to relevant technical documentation without triggering circular link traversals or multi-tenant scope leakage.

#The Scope Resolution Hierarchy

When an autonomous AI agent enters a web domain, it resolves context through a top-down hierarchy:

https://acme.dev/llms.txt                   <-- Tier 1: Platform Overview (Root Index)
   ├── https://acme.dev/docs/llms.txt        <-- Tier 2: Developer Documentation Scope
   │      ├── /docs/quickstart.md
   │      └── /docs/architecture.md
   │
   └── https://acme.dev/api/v2/llms.txt      <-- Tier 2: API Contract Scope (Versioned)
          ├── /api/v2/endpoints.md
          └── /api/v2/openapi.json

#The Root Index Rule

The root /llms.txt must always exist. If an agent is pointed to your naked domain (https://acme.dev), it immediately probes https://acme.dev/llms.txt and https://acme.dev/.well-known/llms.txt. If neither exists, the agent falls back to scraping the raw HTML landing page.

#Scoped Subpath Directory Layouts

For complex platforms, hosting a single monolithic llms.txt creates information overload. Sub-path scoping allows teams to isolate independent documentation domains:

#1. Versioned API Layouts

When supporting multiple API generations, create version-specific manifests so models do not cross-pollinate v1 and v2 methods:

/api/v1/llms.txt    -> Points exclusively to /api/v1/*.md (Deprecated LTS)
/api/v2/llms.txt    -> Points exclusively to /api/v2/*.md (Active Production)

#2. Monorepo & Polyglot SDK Layouts

For organizations distributing multiple language SDKs from a single domain:

https://acme.dev/sdks/typescript/llms.txt
https://acme.dev/sdks/python/llms.txt
https://acme.dev/sdks/go/llms.txt

An engineer asking an AI agent: "Help me integrate Acme in my Go backend" can feed the specific URL https://acme.dev/sdks/go/llms.txt, constraining the model's exploratory retrieval to idiomatic Go channels and concurrency patterns without loading irrelevant TypeScript npm instructions.

#URL Resolution & Redirect Protocol

Autonomous HTTP clients utilized by LLMs (such as Python httpx, Node.js undici, or cURL) handle URL parsing with varying degrees of strictness:

#Production Directive: Always Emit Absolute HTTPS URLs

In all published llms.txt and llms-full.txt files, configure your build system to generate fully qualified absolute HTTPS URLs. This guarantees deterministic resolution regardless of whether the agent processes the file locally in a Docker container or via remote cloud proxies.

#Production Monorepo File Tree Example

acme-platform/
├── public/
│   ├── llms.txt              # Primary root router
│   ├── llms-full.txt         # Consolidated platform bundle
│   ├── ai.txt                # Machine governance policy
│   └── robots.txt            # Search crawler directives
├── docs/
│   ├── public/
│   │   └── llms.txt          # Scoped docs router (/docs/llms.txt)
│   └── content/
│       ├── getting-started.md
│       └── architecture.md
└── api-specs/
    └── public/
        └── llms.txt          # Scoped API router (/api/v2/llms.txt)

#Related guidance

To design your root router, read the foundational llms.txt guide, learn how to bundle documentation in llms-full.txt, and review boundary interactions in llms.txt vs robots.txt.

#References

• The /llms.txt Specification, Section: Subpath Scoping: Proposal guidelines on path hierarchy and discovery mechanics.
• RFC 3986: Uniform Resource Identifier (URI): Generic Syntax: Authoritative specification for URI path resolution and normalization.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.