Cloudflare runs a scanner at isitagentready.com that grades a domain on how ready it is for AI agents. Twenty one checks across five categories: discoverability, content accessibility, bot access control, protocol discovery, and commerce.

We pointed it at spintax.net, a documentation site that had shipped a full llms.txt surface the week before. A curated index, eighteen clean Markdown mirrors of the documentation pages, and a concatenated llms-full.txt for a single fetch. The score came back 21 out of 100, Level 1, “Basic Web Presence”. Content accessibility scored zero out of one.

That gap is the interesting part. The score is not a measure of how readable your site is to a model. It is a measure of which HTTP mechanisms you implement. Those are different things, and only one of them is what most sites think they are buying when they publish an llms.txt.

Here is what the audit actually measures, which four checks were worth implementing, and why eight of the failures were correct.

Why a complete llms.txt scores zero on content accessibility

The check is not looking for a file. It sends a request with Accept: text/markdown and reads the Content-Type that comes back. If the response is text/html, the check fails, no matter how much Markdown you publish at other URLs.

This is HTTP content negotiation, and it has been in the specification since before anyone was building agents. One URL, several representations, the client states a preference in a header. llms.txt is a convention: a file at a known path that a client has to know about in advance. Content negotiation is a mechanism: any client that already speaks HTTP gets the Markdown by asking for it, without knowing your site exists.

We had eighteen Markdown files sitting on disk and no way for a client to reach them except by reading the index first. The audit was right.

Doing content negotiation on Pages, on the free plan

Cloudflare sells the easy version of this. A zone level setting called Markdown for Agents intercepts responses when the request carries Accept: text/markdown and converts the HTML to Markdown on the fly. It is one toggle in AI Crawl Control, or one PATCH to the content_converter zone setting.

It is available on “Pro, Business and Enterprise plans, and SSL for SaaS customers at no cost”. The site is on Free.

The toggle would also have been the worse option even if it were available. Our mirrors are generated at build time from the same HTML, through a turndown configuration with custom rules for the card grids that would otherwise convert into unreadable blocks nested inside links. An automatic conversion at the edge cannot know about those. Hand tuned output beats generic output when you already build it.

So: a Pages Function. Note that _headers and _redirects cannot do this job at all, because neither can branch on a request header. Content negotiation needs code.

// functions/_middleware.ts
export const onRequest = async (context) => {
  const { request, env, next } = context;
  const url = new URL(request.url);

  if (request.method === 'GET' && wantsMarkdown(request.headers.get('Accept'))) {
    const mirror = await env.ASSETS.fetch(
      new Request(new URL(mirrorFor(url.pathname), url.origin)),
    );
    if (mirror.ok) {
      return new Response(mirror.body, {
        status: 200,
        headers: {
          'Content-Type': 'text/markdown; charset=utf-8',
          'Vary': 'Accept',
          'X-Robots-Tag': 'noindex',
          'Link': `<${url.origin}${url.pathname}>; rel="canonical"`,
        },
      });
    }
  }

  const response = await next();
  const out = new Response(response.body, response);
  out.headers.append('Vary', 'Accept');
  return out;
};

Three details in there earn their place.

env.ASSETS.fetch reads the already deployed static file. The Function does no conversion and holds no copy of the content.

Vary: Accept goes on both branches. One URL now has two representations, and a cache that does not key on the header will serve the Markdown to a browser.

The fallback is a miss on the mirror, not an error. If env.ASSETS.fetch returns 404, the code falls through to the HTML. Adding a page that has no mirror yet degrades quietly.

Detecting the preference deserves more than a substring match. Browsers never send text/markdown, so presence is enough signal, but a client can write Accept: text/markdown;q=0 to mean the opposite:

function wantsMarkdown(accept) {
  if (!accept) return false;
  for (const entry of accept.split(',')) {
    const [type, ...params] = entry.split(';');
    if (type.trim().toLowerCase() !== 'text/markdown') continue;
    const q = params.map(p => p.trim()).find(p => p.toLowerCase().startsWith('q='));
    return !q || Number.parseFloat(q.slice(2)) > 0;
  }
  return false;
}

The result is checkable from any terminal, which is the point of using a mechanism rather than a convention:

$ curl -sI -H 'Accept: text/markdown' https://spintax.net/
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Link: <https://spintax.net/>; rel="canonical"
Vary: Accept
X-Robots-Tag: noindex

Keeping a static site static

A root _middleware matches every request by default. On a site of 116 pages plus assets, that turns a fully static deployment into one where every request invokes a Worker, to serve Markdown on nineteen of them.

_routes.json in the build output fixes it:

{
  "version": 1,
  "include": ["/", "/docs/", "/docs/syntax", "/docs/variables/"],
  "exclude": []
}

We generate that file from the same list that generates the mirrors, so the routed paths and the negotiable paths cannot drift apart. Ninety seven localized pages, the 404 and every asset are served from static storage and never reach the Function.

Two Pages traps we hit on the way

Overlapping _headers rules concatenate

This one cost real time. The site already served its Markdown mirrors with a wildcard:

/*.md
  Content-Type: text/plain; charset=utf-8
  X-Robots-Tag: noindex

Adding skill files under /.well-known/agent-skills/ meant adding a more specific rule for them, with text/markdown. Both rules match /.well-known/agent-skills/spintax-syntax/SKILL.md. The result:

Content-Type: text/plain; charset=utf-8, text/markdown; charset=utf-8

Pages merged the two rules by appending the values. It did not let the specific rule win. The output is not a valid Content-Type and nothing warns you.

The fix was to delete the wildcard and generate one explicit rule per mirror from the build. Eighteen rules instead of one pattern, well under the limit of 100. If two _headers rules can match the same path and set the same header name, assume they will both apply.

wrangler pages dev parses _headers once

The local dev server reads _headers at startup and logs how many rules it parsed. It does not reparse after a rebuild. We fixed the wildcard, rebuilt, retested, and saw the same broken Content-Type because the server was still holding the rules it read two builds ago.

Restart the dev server after any change to _headers. The rule count in the startup log is the quickest confirmation that it picked up the new file.

The three cheap checks

Content Signals. A directive in robots.txt that states how the content may be used by AI systems. It goes inside the User-agent group, under a preamble from contentsignals.org that carries the legal weight:

User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes

Allow: /

Three signals, three values each: search for indexing, ai-input for retrieval and grounding, ai-train for model training. A documentation site that publishes Markdown mirrors for machines has an obvious answer to all three. Keep the preamble. It is what turns the line from a preference into a reservation of rights under Article 4 of the EU copyright directive.

Link response headers. RFC 8288, and the audit accepts only registered relation types. describedby for machine readable descriptions, alternate for a different representation of the same page, service-doc for the documentation entry point:

Link: </llms.txt>; rel="describedby"; type="text/plain",
      </docs/variables.md>; rel="alternate"; type="text/markdown",
      </docs/>; rel="service-doc"; type="text/html"

An agent that lands on any page can find the rest from the response headers, without parsing HTML and without knowing about llms.txt. This one costs nothing at runtime: it is static _headers output, no Function involved.

Agent Skills. A discovery index at /.well-known/agent-skills/index.json listing skill documents that an agent can install. This is the only one of the four with product value rather than audit value. A skill is a condensed, imperative document addressed to a machine that is about to do something, which is a different artifact from a documentation page written for a person.

The index entries carry a SHA256 digest of each artifact. Generate the index, never write it by hand. A digest written by hand is wrong the first time anyone edits a skill, and a wrong digest tells a client the file was tampered with. We read the name and description from each skill’s own frontmatter and hash the bytes as served, so there is one source for the metadata and no second list to maintain.

Eight failures that were correct

The checks still failing are DNS for AI Discovery, API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, MCP Server Card, A2A Agent Card and WebMCP. Five commerce checks sit alongside them, and the audit marks those neutral rather than failed, because it can tell the site sells nothing.

A documentation site has no API, no authentication and nothing to sell. Publishing an empty OAuth discovery document to raise a score advertises an authorization server that does not exist. Every one of those documents is a promise to a client about what it will find. A promise you cannot keep is worse than a missing file, because the client that reads it wastes a round trip and then has to guess whether you are broken or lying.

Three of them are worth more detail, because the reasoning generalizes.

The MCP Server Card points at a path the specification abandoned

The audit probes three locations for an MCP Server Card: /.well-known/mcp.json, /.well-known/mcp/server-cards.json, and /.well-known/mcp/server-card.json.

The proposal behind it, SEP-2127, is open and unmerged, status Draft, on the Extensions Track. The normative wire format does not live in the specification repository at all. It lives in a repository named experimental-ext-server-card. The discovery path has moved at least three times, and the current canonical one is /.well-known/ai-catalog.json, with media type application/ai-catalog+json, a catalog that indexes cards rather than being one.

None of the three paths the audit probes is the path the specification now uses. Passing that check today means publishing to an address the spec has already left.

DNS-AID: the check its own author does not pass

This one is worth the most, because it is the cleanest example of why you read the failures instead of clearing them.

DNS for AI Discovery asks you to publish SVCB records under an _agents namespace so agents can find your agent endpoints through DNS. It is a pure DNS record: no Worker, no runtime, no cost, and Cloudflare supports SVCB on the free plan. By the usual arithmetic it is the cheapest check on the entire list.

Three things say otherwise.

The specification is an individual Internet-Draft. Version 02, updated 27 May 2026, not adopted by any working group, no formal standing in the standards process, and it expires on 28 November 2026. That is not a reason to ignore it. It is a reason to know what you are building on.

The scanner probes labels the draft does not define. It queries _index._agents, _mcp._agents and _a2a._agents. The draft defines the entry point at _index._agents and selects protocols through the alpn service parameter, not through per-protocol underscore labels. _mcp and _a2a are the checker’s, not the specification’s — the same mismatch as the MCP Server Card above, from the same cause.

Nobody publishes it. Not a rhetorical nobody. Checked over DNS-over-HTTPS on 22 July 2026:

Domain _index._agents
isitagentready.com NXDOMAIN
cloudflare.com no records
agents.cloudflare.com no records

The audit site itself, and the company that wrote the audit.

And the conclusion that follows: point the scanner at itself. isitagentready.com comes back Level 4, Agent-Integrated, with dnsAid among its own failures — the same level this blog reached, by a tool that fails its own check.

To be fair to the draft, one objection against publishing does not hold. It is tempting to say we have no agents to index, but the draft puts the content of the index endpoint out of scope: it can be a live service or a static document, and there is a well-known service parameter pointing at RFC 8615 metadata. We could honestly point _index._agents at the Agent Skills index we already publish.

We are not going to, and the reason is not honesty. It is that a record satisfying a check that its own author does not implement, against a draft expiring in four months, in a namespace whose labels the checker invented, buys a number and nothing else.

An A2A Agent Card names a live endpoint

The A2A Agent Card requires a url for the A2A service. It describes an agent that other agents can hand work to. We do not run one. Publishing the card would advertise a service that answers nothing.

This also puts a ceiling on the score. Level 5 requires Auth.md, an MCP Server Card, an A2A Agent Card and an API Catalog. A content site can honestly reach maybe one of those. Level 4 is the top of the range for a site that only publishes documents, and that is a property of the ladder rather than a failing of the site.

The MCP server we did not build, and one number that decided it

The one item on the list with genuine value for us is an MCP server: give an agent a validate tool that returns structured diagnostics with line and column, and the loop from “model writes a template” to “model fixes the template” closes without a human in it.

Two findings pushed it out of this week.

The protocol is mid break. The 2026-07-28 revision makes Streamable HTTP stateless. The Mcp-Session-Id header “and the protocol-level session that came with it are also removed”, the initialize/initialized handshake is removed, and the transport “now requires Mcp-Method and Mcp-Name headers so load balancers, gateways, and rate-limiters can route on the operation”, with a new server/discover method for capabilities. Upstream calls it the largest revision of the protocol since launch. Building against the current shape means rewriting in a week.

The second finding is a number. Cloudflare’s documentation is clear that a stateless MCP server needs no Durable Objects and no paid plan, which sounds like the free tier is enough. Then you check what the free tier gives you: 10 ms of CPU per request. We measured the engine we would be wrapping, on Node, so treat it as an order of magnitude rather than a measurement of the target:

Work CPU
400 byte template, one render, warm 0.13 to 0.28 ms
400 byte template, twenty variants 3 to 6 ms
First call on a cold isolate about 12.5 ms
16 KB template, twenty variants about 85 ms

The cold isolate alone is over budget before any template is parsed. A 16 KB template overruns by a factor of eight. A dependable server on the free plan would need caps tight enough to be annoying, around 4 KB of template and five variants. The Workers Paid plan at five dollars a month raises the limit to 30 seconds of CPU and removes the question.

That is a design decision to make before writing code, not after.

Where the score landed, and where this site sits

Three checks passing became seven, and the level went from 1 to 4, “Agent-Integrated”. Six more are marked neutral because they do not apply, and the eight that fail are the eight that should.

The useful output was not the number. It was being told that a week of work on llms.txt had produced files no client could negotiate for, which is a specific and fixable thing that no amount of reading our own documentation would have surfaced.

For the record, this blog scored Level 1 when the article was written, three of twenty one: exactly where the other site started. Running the same four checks against it the same day produced a different answer, and the difference is worth more than the score.

All four went in, and it now reads Level 4, the same as the site this article is about. Content Signals, Link headers, an Agent Skills index, and a Markdown mirror of every article indexed in llms.txt. The interesting part is the fourth one, because the shape of the site changes what it costs.

This site is not on Pages. It is a Worker serving static assets, and until today it had no script at all, which is why “requests to static assets are free and unlimited” covered every page view. Negotiation needs code in front of those assets, and any page that runs that code becomes an ordinary Worker request against the free plan’s 100,000 per day.

So the code runs on as little as possible: run_worker_first lists the apex and single-segment paths, nothing else. Stylesheets, images, the mirrors themselves, robots.txt and the sitemap never reach it and stay free.

One detail is worth stealing. Cloudflare’s cache key does not include Vary unless a Cache Rule puts it there, so one URL with two representations can serve Markdown to a person out of cache. The fix is not a zone rule: it is Cache-Control: no-store on the Markdown branch only. Agents are a trickle, the HTML keeps its normal caching, and the failure mode disappears.

The other one is embarrassing and cheap. Handle HEAD, not just GET. RFC 9110 says a HEAD must answer with the headers a GET would send, and the first version of this did not — which curl -I reported immediately, that being exactly how anyone would check it by hand.

Same four checks, same day, two different sites, and the work was different on each. That is the actual lesson of running an audit: the checks are generic and your infrastructure is not.

Run it against your own domain. Then read the failures and decide which ones describe a site you are actually running:

curl -sS -X POST https://isitagentready.com/api/scan \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com"}'

The JSON endpoint returns every check with its status, message and the requests it made, which is faster than the browser and easier to diff after a deploy. It reports the level rather than the score; the number out of 100 is the web interface’s.