How to Make Your Website Discoverable to AI Agents: llms.txt, agent-card.json, ai-catalog.json, and MCP Server Cards

Search engines find your site through sitemap.xml and robots.txt. AI agents use a different set of files — and as of mid-2026, there are four that matter: llms.txt for LLM-based crawlers and answer engines, agent-card.json for the A2A protocol, ai-catalog.json for ARD registries, and the MCP server card for tool clients. Most sites publish none of them, which means most sites are invisible to the agent traffic that's starting to arrive.

This is the practical checklist: what each file does, where it goes, and the order in which to ship them.

The Four Discovery Files

File	Location	Read by	Standard
llms.txt	/llms.txt	LLM crawlers, answer engines	community convention
A2A Agent Card	/.well-known/agent-card.json	Client agents (A2A protocol)	Linux Foundation A2A spec
ARD catalog	/.well-known/ai-catalog.json	Discovery registries	ARD spec (Google, Microsoft, Hugging Face, GoDaddy)
MCP server card	/.well-known/mcp/server-card	MCP clients	SEP-2127 (experimental)

They serve different consumers and don't substitute for each other. llms.txt describes your site's content to models that read; the other three describe your site's capabilities to agents that act.

Step 1: llms.txt — the Lowest-Effort Win

llms.txt is a Markdown file at your site root that gives LLMs a curated, token-efficient summary of what your site offers — name, one-paragraph description, key pages with short annotations. It's the robots.txt of the answer-engine era: no tooling required, shippable in an hour, and increasingly read by crawlers feeding retrieval systems.

Structure convention:

# Your Product

> One-sentence description naming what you do and for whom.

## Docs

- [Quickstart](https://example.com/docs/quickstart): Set up in 5 minutes
- [API Reference](https://example.com/docs/api): REST endpoints and auth

## Pricing

- Free tier: what it includes

Keep it factual and entity-dense — model-facing copy is indexed by what it names, not how it persuades.

Step 2: A2A Agent Card — If You Expose an Agent

If your product offers an agent other agents can delegate to (a booking flow, a support triage bot, a data-analysis service), publish an Agent Card at /.well-known/agent-card.json. The card declares your agent's protocolVersion, service endpoint, skills, input/output media types, and securitySchemes — everything a client agent negotiates before opening a task.

Two details that determine whether the card actually earns traffic:

• Skills are marketing copy for routing LLMs. A client-side model reads your skill descriptions to decide whether to engage you. "Searches fares and books flights on major carriers" gets selected; "Handles requests" does not.
• Never put secrets in the card. It states how to authenticate (OAuth 2.0, OpenID Connect discovery URL, API key header), never credentials.

We covered the full field-by-field format in A2A Agent Cards Explained.

Step 3: MCP — If You Expose Tools

Where A2A advertises a collaborating agent, MCP exposes callable tools — functions with JSON Schema inputs that hosts like Claude, Cursor, and Windsurf invoke directly. If you ship a remote MCP server, two discovery details matter:

1. Serve it at a predictable path. https://yourdomain.com/mcp is the emerging convention, and clients probe it.
2. Publish a server card at /.well-known/mcp/server-card (SEP-2127). The extension is experimental, but it's cheap to publish and it's the direction MCP discovery is heading — early publishers get found by early crawlers.

If your server requires auth, return 401 with a WWW-Authenticate header pointing at RFC 9728 protected-resource metadata. A correctly configured auth challenge is itself a discovery signal: it tells clients "there's an MCP server here; here's how to get in."

Step 4: ARD Catalog — the Index Over Everything

The newest layer, from the July 2026 Agentic Resource Discovery spec, ties the rest together. One JSON file at /.well-known/ai-catalog.json lists every agentic capability your domain offers — your A2A agent, your MCP server, plain APIs — each with a domain-anchored URN, a media type, tags, and representativeQueries for semantic search.

Then add one line to robots.txt:

Agentmap: https://yourdomain.com/.well-known/ai-catalog.json

That's the sitemap-directive equivalent for agent registries. ARD registries crawl catalogs the way Google crawls sitemaps and answer natural-language queries ("who can notarize a document?") with ranked entries. Catalogs are rare right now — which is exactly why publishing one is asymmetric upside: when registry crawling becomes routine, the indexed-early domains are the ones agents find.

Verify From the Outside

Every file above fails silently: a typo'd path, a JSON syntax error, or a CDN rule returning your SPA shell with a 200 for every route, and crawlers index nothing while your checklist says done. (SPA catch-alls are the sneakiest failure — /.well-known/agent-card.json returns HTML with status 200, which parses as a card exactly never.)

So test the way a crawler tests: from a cold client, no cookies, no auth. The Agent Readiness Detector does the full sweep in one scan — fetches your agent card, your ARD catalog, your robots.txt Agentmap line, and runs a live MCP initialize handshake — and reports each protocol as confirmed, indicated, or not detected, with the evidence.

The Priority Order

1. llms.txt — an hour of work, immediate answer-engine benefit, zero risk.
2. The protocol matching what you actually expose — agent card if you have an agent, MCP server + card if you have tools. Don't publish a card for capabilities that don't exist; a card whose endpoint 404s is worse than no card.
3. ai-catalog.json + Agentmap — fifteen minutes once the above exist, and it's the layer registries will crawl.
4. Re-scan quarterly. Specs are moving (A2A's card path already migrated once); a scheduled outside-in check catches drift before partners do.