> Content index: https://quillly.com/docs/llms.txt > Canonical page: https://quillly.com/docs/ai-crawlers --- title: AI crawlers and llms.txt: how AI agents read your site description: Quillly serves llms.txt and Markdown versions of your pages to AI agents and counts AI crawlers in Analytics. Every file, and what the counts mean. keywords: AI crawlers, llms.txt, Markdown published: 2026-09-24 updated: 2026-09-24 url: https://quillly.com/docs/ai-crawlers word_count: 949 --- # AI crawlers, llms.txt and Markdown pages > Quillly serves llms.txt and Markdown versions of your pages to AI agents and counts AI crawlers in Analytics. Every file, and what the counts mean. Canonical: https://quillly.com/docs/ai-crawlers Published: 2026-09-24 Quillly serves every page it publishes in a form AI agents read easily, and counts the AI crawlers that fetch your pages. This page lists the llms.txt and Markdown files your site offers, shows where AI crawlers appear in Analytics and explains what the numbers mean. ## What AI agents can read Every page Quillly serves, whether a blog post, a docs page, a news story or the changelog, comes in three forms: - **The HTML page** that people read. - **A Markdown version.** Add `.md` to the page's address, so `/docs/quick-start` becomes `/docs/quick-start.md`. An agent that asks for `text/markdown` in its `Accept` header, without `text/html`, gets the same Markdown at the normal address. - **An llms.txt index** for the section: a plain list of its pages, each with a link and a short description. It follows the [llms.txt proposal](https://llmstxt.org/), which puts the file at the root of the site or of a section. Each page links to these at its end, in a line that reads **AI agent or LLM? Read this page as Markdown ยท llms.txt**. On the changelog, it reads **Read the changelog as Markdown**. ![A docs page served by Quillly on acme.com, with the Copy page button at the top and the AI agent or LLM line at the end highlighted](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/c1e0abb618b16919eb11ecc81dda3c79af2ebb62.webp) The Markdown version names the HTML page as its canonical, so search engines don't treat it as a copy. Docs pages also have a **Copy page** button, whose menu offers **View as Markdown**, **Open in ChatGPT** and **Open in Claude** to readers who want to ask an assistant about the page. ## Where the files are Which files exist depends on how your site is served. On your own domain, Quillly answers only under your sections' paths, through the [reverse proxy](https://quillly.com/docs/reverse-proxy), so the whole-site files exist only on a Quillly subdomain. | File | On your own domain | On a Quillly subdomain | | --- | --- | --- | | One section's index | `/blog/llms.txt`, `/docs/llms.txt`, `/changelog/llms.txt` or `/news/llms.txt`. | `/docs/llms.txt`, `/changelog/llms.txt` or `/news/llms.txt`. | | The whole site's index | None. | `/llms.txt`, which includes the blog. | | Full text of the 50 newest items | None. | `/llms-full.txt`. | | One blog post as Markdown | `/blog/your-post/llms.txt`. | `/your-post/llms.txt`. | | Any page as Markdown | The page's address with `.md` added. | The page's address with `.md` added. | The paths follow your sections. If your docs live at `/help`, their index is `/help/llms.txt`. The changelog index links the timeline and names its 15 newest entries, and the news index lists the 15 newest stories without sponsored ones. These files need no setup and update as you publish. A section that [publishes into WordPress, Ghost, Webflow or Shopify](https://quillly.com/docs/publish-to-cms) is served by that platform, so it has none of these files. ## AI crawlers in Analytics Open **Analytics** and find the **AI & crawlers** card. It counts crawler visits to your published pages, kept apart from your human visitors. ![The AI and crawlers card in Quillly Analytics for acme.com, with 412 AI answer crawls by OpenAI, Anthropic, Perplexity, Google and Microsoft, and the category tabs highlighted](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/e8ecc3bdc8e88306cd51d4f98c22dc8032551ea3.webp) The tabs sort crawlers by what they came for: | Tab | What it counts | | --- | --- | | **AI answers** | Assistants fetching a page because someone asked about it, such as ChatGPT-User and Claude-User. | | **Indexing** | Crawlers that keep search engines and AI search up to date. | | **Training** | Crawlers that collect pages to train AI models, such as GPTBot and ClaudeBot. | | **Other** | SEO tools, other bots and direct fetches of your Markdown and llms.txt files. | **By company** puts all of a company's crawlers on one row. **By page** ranks your pages instead: click one to see which crawlers read it. The **Pages** card has the same view: switch it from **Visitors** to **Crawls**. Read the numbers as a trend rather than an exact total: - Only pages Quillly serves are counted, so a site that only runs the tracking script shows no crawls. - One crawler reading one page counts at most once an hour. - Visits answered from a CDN cache never reach Quillly, so they aren't counted. - A client that fetches a `.md` or llms.txt file without being a known crawler, such as a script or a person, counts under **Other** as a direct fetch. Your AI reads the AI answers side with `get_website_analytics`, whose `ai_answers` field lists which assistants fetched which pages. [Open Analytics](https://quillly.com/dashboard/analytics) ## The crawler directory The [AI Crawler Directory](https://quillly.com/crawlers) lists the more than 60 AI crawlers and bots Quillly recognises. For each one it gives the company, what the crawler is for, the user-agent token it sends and whether it follows robots.txt. They're grouped as **AI answers**, **Search indexes**, **Training crawlers**, **SEO tools** and **Other AI bots**, and each has its own page. ## Blocking AI crawlers On your own domain, your own `robots.txt` decides which crawlers may read the site, because it sits at your domain's root, outside Quillly's paths. To keep a training crawler out, add a group for its user-agent token, which the directory lists. OpenAI, for example, documents [its crawlers](https://developers.openai.com/api/docs/bots) and says that disallowing GPTBot keeps a site out of model training: ```text User-agent: GPTBot Disallow: / ``` On a Quillly subdomain, Quillly serves the `robots.txt`. It allows every crawler, lists your sitemap and points to your `/llms.txt`, and there's no setting to change it. > [!NOTE] > Blocking a training crawler doesn't block the assistants that fetch a page to answer a question. Those use their own tokens, such as ChatGPT-User and Claude-User. ## Common questions ### Do I need to create an llms.txt file? No. Quillly builds the files from your published pages and serves them next to each section. They change as soon as you publish or unpublish. ### Why don't I see any AI crawls? AI crawlers usually find a page within days of publishing, and Quillly counts them only on the pages it serves. If your site only runs the Quillly tracking script, crawls aren't visible to Quillly. ## Related - [Content types and URLs](https://quillly.com/docs/content-types): the sections, their paths and their feeds. - [Serve on your own domain with a reverse proxy](https://quillly.com/docs/reverse-proxy): the setup behind the files on your own domain. - [Indexing: how your pages reach search engines](https://quillly.com/docs/indexing): the search crawlers' side.