ai.txt: a place for AI-use preferences

AI Summary: ai.txt is an experimental machine-readable specification placed at /ai.txt (or /.well-known/ai.txt). It articulates an organization's explicit policies regarding AI data harvesting, model pre-training, retrieval-augmented inference, and content attribution, bridging the gap between ambiguous Terms of Service and automated crawler infrastructure.

#The Governance Vacuum: Beyond Binary Robots.txt Directives

Historically, webmasters relied exclusively on robots.txt to control external crawlers. However, the rapid proliferation of Generative AI, Retrieval-Augmented Generation (RAG), and autonomous web agents exposed critical limitations in the robots.txt protocol:

1. Lack of Granularity: robots.txt provides only binary directives: Allow or Disallow. It cannot express nuances such as: "You may cite this documentation in search answers with attribution, but you may not use our proprietary code repositories for foundation model pre-training."
2. Ambiguous Legal Enforceability: In jurisdictions such as the European Union (under Article 4 of the Digital Single Market Directive 2019/790), rights holders must reserve Text and Data Mining (TDM) rights in an express, machine-readable manner. robots.txt was designed for server load management, not legal copyright reservation.
3. Agent Navigation Intent: Modern AI coding tools (like Cursor or Claude Code) visit web documentation on behalf of an authenticated human user to complete a task. Blocking these agents via robots.txt damages the developer experience of your own users, whereas blocking foundational web scrapers protects intellectual property.

The /ai.txt proposal seeks to standardize these distinctions through clear, declarative key-value policies.

#Syntax Specification & Policy Directives

An ai.txt file uses a simple, deterministic syntax consisting of directive headers, scope declarations, and permission flags. Below is an enterprise-grade production specification:

# ai.txt specification v1.2
# Policy URI: https://acme.dev/legal/ai-terms
# Last-Reviewed: 2026-09-10

User-Agent: *
Allow-Inference: yes
Allow-Training: no
Allow-Summarization: yes
Require-Attribution: yes
Contact: legal-ai@acme.dev

# Specialized permissions for technical documentation
Scope: https://acme.dev/docs/*
Allow-Inference: yes
Allow-Training: no
Attribution-URL: https://acme.dev/docs/attribution-guidelines

# Prohibit proprietary API schemas and benchmark fixtures
Scope: https://acme.dev/internal/*
Scope: https://acme.dev/benchmarks/*
Allow-Inference: no
Allow-Training: no

#Core Directives Glossary

• Allow-Inference: Controls whether an AI tool can fetch and synthesize the content in real time to answer a prompt submitted by an end user.
• Allow-Training: Explicitly permits or denies foundational model training, fine-tuning, or vector weight embedding for commercial or research model development.
• Require-Attribution: Instructs generative answer engines (Perplexity, Google AI Overviews, OpenAI Search) to display canonical backlinks and publisher attribution.
• Attribution-URL: Specifies the preferred landing page for brand attribution.

#Policy Enforcement Realities: Cryptographic Security vs Voluntary Compliance

Systems architects must understand what ai.txt can and cannot accomplish:

[!WARNING] Advisory Warning: Publishing an ai.txt file will not stop malicious or rogue scrapers who disregard standard web headers. If your site contains confidential IP, proprietary datasets, or enterprise code, protect it behind authenticated edge firewalls (such as Cloudflare Access, mTLS, or OAuth gateways). Never rely on ai.txt as an access control mechanism.

#Implementing Edge Validation & Serving

Deploy ai.txt at the root of your web property with deterministic caching and CORS headers so that developer IDEs and compliance checkers can inspect your policy:

// app/ai.txt/route.ts (Edge Handler)
import { NextResponse } from 'next/server'

export const runtime = 'edge'

export async function GET() {
  const policy = `# ai.txt policy for acme.dev
User-Agent: *
Allow-Inference: yes
Allow-Training: no
Require-Attribution: yes
Contact: security@acme.dev
`

  return new NextResponse(policy, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, stale-while-revalidate=604800',
      'Access-Control-Allow-Origin': '*',
      'X-Content-Type-Options': 'nosniff',
    },
  })
}

#Related guidance

To understand how crawler directives diverge from agent navigation maps, compare this specification with llms.txt vs robots.txt, inspect our foundational llms.txt guide, and review how Cursor IDE navigates site context.

#References

• The /llms.txt proposal specification: Foundational index architecture and metadata standards.
• W3C Machine-Readable Rights & TDM Declarations: Formal recommendations on automated copyright reservations under modern copyright directives.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.