> Content index: https://quillly.com/blogs/llms.txt > Canonical page: https://quillly.com/blogs/robots-txt-wp --- title: Robots Txt WP: What WordPress Actually Serves description: The robots txt WP serves is generated, not stored on disk. See what's really in it, which old rules to delete, and how to edit it without breaking search. keywords: robots txt wp, wordpress robots txt, robots txt file, block ai crawlers published: 2026-09-17 updated: 2026-09-17 url: https://quillly.com/blogs/robots-txt-wp word_count: 2793 --- # Robots Txt WP: What WordPress Actually Serves > WordPress never ships a robots.txt file — it generates a virtual one. See what's really in it, which old rules to delete, and how to edit it safely. Canonical: https://quillly.com/blogs/robots-txt-wp Published: 2026-09-17 ## Related Pages - [WordPress SEO Plugins: What Each One Actually Fixes](https://quillly.com/blogs/wordpress-seo-plugins) - [Crawled Currently Not Indexed: A 284-Page Test](https://quillly.com/blogs/crawled-currently-not-indexed) - [Should You Block AI Crawlers? A 2026 Decision Guide](https://quillly.com/blogs/should-you-block-ai-crawlers) - [Fake Googlebot: How to Verify a Crawler in 2026](https://quillly.com/blogs/fake-googlebot-verification) ![Yellow and green network cables neatly connected in a server rack](https://images.unsplash.com/photo-1744868562210-fffb7fa882d9?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w4OTM1MDJ8MHwxfHNlYXJjaHwxfHxzZXJ2ZXIlMjByb29tJTIwbmV0d29yayUyMGNhYmxlcyUyMGRhdGElMjBjZW50ZXJ8ZW58MHwwfHx8MTc4OTYwNTQ0Mnww&ixlib=rb-4.1.0&q=80&w=1080) *Photo by [Albert Stoynov](https://unsplash.com/@albertstoynov?utm_source=quillly&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=quillly&utm_medium=referral)* Search for "robots txt wp" and every guide hands you a file to edit. Open your file manager on a fresh WordPress install and go looking for that file, though, and it simply isn't there. Load `yoursite.com/robots.txt` in a browser and it appears anyway, complete with rules you never wrote. That gap — a file that exists for crawlers but not on disk — is where most WordPress crawling mistakes begin. **The short answer:** WordPress doesn't ship a robots txt file at all. It builds a virtual one in PHP on every request, so nothing exists on disk until you create it. The moment a real file lands at your web root, the virtual one stops being used completely — which is why plugin edits sometimes seem to vanish. ## What WordPress Actually Serves at /robots.txt WordPress intercepts requests for `/robots.txt` and generates the response in PHP through its `do_robots()` function. Nothing is written to disk, nothing appears over SFTP, and nothing is backed up with your files. It's assembled fresh on every hit. That design has one consequence worth tattooing somewhere: **a physical file always wins.** Your web server looks for a real file before PHP ever runs. Upload a `robots.txt` to your web root and WordPress's version — along with every rule your SEO plugin thinks it controls — is bypassed silently. No warning, no notice in the dashboard. Plugins hook into the [`robots_txt` filter](https://developer.wordpress.org/reference/hooks/robots_txt/) to add their rules to the virtual file. So Yoast, Rank Math and All in One SEO are all editing the same generated output, and only while no physical file exists. ![Flowchart showing how a request for slash robots dot txt is answered in WordPress, checking for a physical file before falling back to the generated virtual file](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/00fbc36c1478ac7a844168c5e816f377883c0728.webp) ## The Default Robots Txt WP Generates On a standard install with search engines allowed, the generated file is about as small as it gets: ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: (your full site URL)/wp-sitemap.xml ``` Four meaningful lines, and each one earns its place: - **`User-agent: *`** — the rules below apply to every crawler that doesn't have its own named block. - **`Disallow: /wp-admin/`** — keeps crawlers out of the admin area. There's nothing indexable in there and it's login-gated anyway, so this is tidiness more than protection. - **`Allow: /wp-admin/admin-ajax.php`** — the important one. Plenty of themes and plugins route front-end functionality through `admin-ajax.php`. Without this exception, a crawler rendering your page gets blocked mid-request and may see a broken layout. - **`Sitemap:`** — WordPress 5.5 and later point at the core sitemap at `/wp-sitemap.xml`. Install Yoast or Rank Math and the line is usually replaced with that plugin's own sitemap index instead. There's one switch that rewrites all of this. Under **Settings → Reading**, the "Discourage search engines from indexing this site" checkbox replaces the whole file with `Disallow: /` and adds a noindex directive to your pages. It's meant for staging sites, and it's the single most common reason a newly launched site gets no traffic — it stays ticked after go-live. ## Three Ways to Edit a WordPress Robots Txt File A WordPress robots txt file can come from three different places, and they don't survive the same events. Pick deliberately, because the failure modes are not the same. | Route | Survives a theme change | Survives a host migration | Best for | | --- | --- | --- | --- | | SEO plugin editor (Yoast, Rank Math, AIOSEO) | Yes | Only if the plugin comes with you | Most sites | | Physical file at the web root | Yes | Only if you copy the file across | Static rules you rarely touch | | `robots_txt` filter in a small site plugin | Yes | Yes | Developers who want rules in version control | The plugin editors are the right default for most people — our breakdown of [what each WordPress SEO plugin actually fixes](https://quillly.com/blogs/wordpress-seo-plugins) covers how they differ elsewhere. Just know what you're choosing: your crawl rules now live in that plugin's settings, and deactivating it takes them with it. The physical file is the most predictable and the easiest to forget. It doesn't appear in any dashboard, so six months later nobody remembers why the plugin's robots.txt tab has no effect. ![Three comparison cards describing the plugin editor, physical file and code filter routes for editing a WordPress robots.txt](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/fa3aa4d094824712754dbc675b6cca80e43dc7ad.webp) ## The 2012 Template That Still Circulates Search for a WordPress robots.txt template and you'll still be handed rules that were reasonable when Google couldn't render JavaScript and are actively harmful now. Delete these: - **`Disallow: /wp-includes/`** — blocks core JavaScript files that Google needs to render your pages. - **`Disallow: /wp-content/plugins/`** — blocks plugin CSS and JS. Google explicitly asks for access to the [resources needed to render a page](https://developers.google.com/search/docs/crawling-indexing/robots/intro); block them and it judges a layout that doesn't match what visitors see. - **`Disallow: /feed/` and `Disallow: /trackback/`** — feeds are how plenty of tools, aggregators and IndexNow-style pipelines discover new posts. - **`Disallow: /category/` or `/tag/`** — archive pages often rank perfectly well for broad terms. If you don't want them indexed, that's a noindex decision, not a crawl-blocking one. - **`Disallow: /wp-admin/` with no `admin-ajax.php` exception** — the one rule people copy correctly and then break by trimming the `Allow:` line under it. Not sure which of these you inherited? [Build a clean file with the robots.txt generator](https://quillly.com/tools/robots-txt-generator){cta=tool} and carry over only the rules you can actually justify. ![Warning card listing five outdated WordPress robots.txt rules that should be removed, with the reason each one causes harm](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/59a7f6a456b50541340c17f6d21d6e221506d5ed.webp) ## Robots.txt Doesn't Deindex Anything This is the misconception that costs the most, so it's worth stating bluntly: **`Disallow` controls crawling, not indexing.** A blocked URL can still appear in search results. If Google finds enough links pointing at a page, it can index the URL without ever fetching it — you'll see it in Search Console as "Indexed, though blocked by robots.txt", listed with no useful snippet because Google never got to read the page. Worse, blocking a page makes it *harder* to remove. To drop a URL from the index you need a [noindex directive](https://developers.google.com/search/docs/crawling-indexing/block-indexing), delivered as a meta tag or an `X-Robots-Tag` header — and a crawler has to be allowed to fetch the page to see it. Block it in robots.txt and the noindex is never read, so the page sits in the index indefinitely. The sequence that actually works: allow crawling, serve noindex, wait for the page to drop out, and only then add a `Disallow` if you also want to stop the crawling. It's also worth being realistic about how much robots.txt influences whether you get indexed at all. This site publishes 152 pages, every one served under an identical, fully permissive robots.txt — and the outcome per engine is nothing alike. ![Bar chart comparing the share of 152 published pages indexed by Google, Bing and Yandex under one identical permissive robots.txt](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/e7f3b33b486d6d79099b220a8db798ab8587f59b.webp) Same rules, same sitemap, same submissions — 89% indexed on one engine and 24% on another. Crawl permission is a gate, not an engine. Once you're through it, everything that decides indexing happens elsewhere, which is the pattern behind [pages that get crawled and still aren't indexed](https://quillly.com/blogs/crawled-currently-not-indexed). Want a clean file without hand-editing? The free [robots.txt generator](https://quillly.com/tools/robots-txt-generator){cta=tool} builds one with the AI-bot presets already filled in. ## When Disallow Actually Earns Its Keep Crawl budget is a real constraint, but mostly on large sites. Under a few thousand URLs, Google will generally crawl everything worth crawling and `Disallow` buys you very little. Where it does pay off is combinatorial URLs — the ones a site generates faster than anyone can link to them: - WooCommerce filter and sort parameters (`?filter_`, `?orderby=`, `?min_price=`), which multiply into thousands of near-identical pages - Cart, checkout and account URLs, which are useless in search and often session-specific - Internal search result pages (`/?s=`), an endless URL space fed by anyone typing into your search box - `?add-to-cart=` action URLs, which do something when fetched ![Decision flowchart for whether a WordPress URL belongs in robots.txt, needs a noindex tag, or should be left alone](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/fd402cd361488affffb0c2b9df256274a08e9b81.webp) ## Should You Block AI Crawlers? "AI crawler" covers three jobs that robots.txt treats identically and you shouldn't, because blocking the wrong tier costs you traffic you meant to keep. ![Three tiers of AI crawler compared: search indexers, answer fetchers and training collectors, with the consequence of blocking each](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/cd23b0676452d4da8dbd071828bbd83ac53dc148.webp) The tier that most people actually mean is the third. Blocking [GPTBot](https://quillly.com/crawlers/openai-gptbot) or [ClaudeBot](https://quillly.com/crawlers/anthropic-claudebot) keeps your writing out of future training runs without touching how you rank or whether you get cited today. `Google-Extended` is a robots.txt token only — it governs Gemini model training and has no effect on Googlebot or Search. So if you want to **block AI crawlers**, be specific about which: ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / ``` Each directive depends entirely on the operator honouring it, and the well-documented ones do — OpenAI [publishes its bot list](https://platform.openai.com/docs/bots), as do Anthropic and Google. The [crawler directory](https://quillly.com/crawlers) tracks which user-agents identify themselves and which document robots.txt compliance. For the strategy behind the decision rather than the syntax, [should you block AI crawlers](https://quillly.com/blogs/should-you-block-ai-crawlers) works through what each tier is worth. ## How to Test What Crawlers Really See Your browser is not a reliable witness here. Check it the way a crawler would: ``` curl -A "Googlebot" https://quillly.com/robots.txt ``` Then confirm it with Search Console's robots.txt report, and use URL Inspection's live test on a page you care about — it reports "Crawl allowed?" per URL, which catches rules you'd never spot by reading the file. ![Three verification steps for a WordPress robots file: fetching it as a crawler, checking the Search Console report, and running a live URL inspection](https://quillly.com/serve/v1/019c64a2-a62f-7793-aa68-2c78316d3309/images/c9b1564f646c88dc2f22e1830cc31215787d3622.webp) Three gotchas worth knowing before you go hunting: 1. **A CDN can cache robots.txt.** You fix the file, the origin serves the new one, and Cloudflare keeps handing crawlers the old version for hours. Purge after every change. 2. **The virtual file needs pretty permalinks.** With plain permalinks, WordPress's rewrite handling for `/robots.txt` may not fire at all. 3. **Not every "Googlebot" in your logs is Googlebot.** Blocking rules are trivially ignored by impersonators, so confirm the source before you react to it — [verifying a crawler](https://quillly.com/blogs/fake-googlebot-verification) takes one reverse DNS lookup. [Start a free Quillly trial](https://quillly.com/login){cta=signup} if you'd rather watch which crawlers actually reach your posts than guess from the raw logs. ## A Baseline Robots Txt WP Setup for 2026 For the overwhelming majority of WordPress sites, the whole correct file is this: ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: (your full site URL)/wp-sitemap.xml ``` The `Sitemap:` value has to be a complete absolute URL — scheme and domain included, never a relative path like `/wp-sitemap.xml`. That's it. Running WooCommerce adds a handful of genuinely useless URL spaces worth closing off: ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /?s= Disallow: /*?orderby= Disallow: /*?filter_ Sitemap: (your full site URL)/wp-sitemap.xml ``` Write your own domain into that last line and check the sitemap path against whatever your SEO plugin publishes — Yoast and Rank Math use `/sitemap_index.xml`, WordPress core uses `/wp-sitemap.xml`. Listing a sitemap that 404s is a wasted line, and it's the most common error in otherwise sensible files. If your blog is served from a subdirectory by a separate platform, the robots.txt still has to live at the root of your own domain — one file per host, no exceptions, per the [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html). Make sure that root file allows the blog path and lists the blog's sitemap alongside your main one, or the section you publish most often is the section crawlers are told to skip. > **Stop guessing whether your posts got crawled** > Quillly publishes to your own domain, submits every new post to 8 search engines, and reports back which ones actually indexed it — per engine, per URL. $19/month after a 14-day trial, no card. > → [Start your free trial](https://quillly.com/login) ## Frequently Asked Questions ### Where is the robots txt file in WordPress? Nowhere on disk, on a default install. WordPress generates a virtual robots.txt in PHP each time one is requested, so you won't find it over SFTP or in your file manager. It only becomes a real file if you upload one yourself or a plugin writes one to your web root. ### Does a WordPress site need a robots txt file at all? No. A site with no robots.txt file is treated as fully crawlable, which is what most sites want anyway. WordPress's generated default is already sensible, so leaving it alone is a legitimate choice rather than an oversight. ### Will blocking a page in robots.txt remove it from Google? No, and it can do the opposite. Disallow stops crawling, not indexing, so a linked URL can stay in the index with no snippet. Use a noindex tag and keep the page crawlable so Google can actually read that instruction. ### Should I block AI crawlers in my WordPress robots.txt? Only the tier you mean. Blocking training collectors like GPTBot or ClaudeBot doesn't affect your rankings or citations. Blocking search indexers or answer fetchers removes you from those results, which is rarely what anyone intends when they write the rule. ### Why did my robots.txt changes disappear? Almost always a physical file at your web root overriding the virtual one, so the plugin edits apply to output nobody serves. Check for a real robots.txt first, then for a CDN serving a cached copy of the old version. ### Does robots.txt affect my sitemap? Only for discovery. The `Sitemap:` line tells crawlers where to look, but blocking a URL in robots.txt while listing it in your sitemap sends two contradictory signals — and the block wins, so the page goes uncrawled. ## Key Takeaways - WordPress has no robots txt file until you make one. The robots txt WP serves is generated fresh on every request, and a real file at the web root silently overrides it. - The generated default — `Disallow: /wp-admin/` with the `admin-ajax.php` exception, plus a sitemap line — is correct for most sites as-is. - Delete inherited rules that block `/wp-includes/` and `/wp-content/plugins/`. They stop Google rendering your pages the way visitors see them. - Disallow is not deindexing. Use noindex on a crawlable page, wait for it to drop out, and only then block the crawl. - If you block AI crawlers, name them. Training collectors cost you nothing in search; indexers and answer fetchers cost you results and citations. - Test with `curl -A "Googlebot"` and Search Console's live test, and purge your CDN afterwards.