Rule referencetech/robots-txt-limits

robots.txt Limits: The 500 KiB Cutoff and Four Directives Google Ignores

500 KiB is the hard ceiling: Google parses the first 512,000 bytes of a robots.txt file and silently ignores every rule beyond it, so tech/robots-txt-limits measures the file's UTF-8 byte length and raises a warning the moment a generated faceted-navigation file crosses that line, then separately reports the four directives Google's parser never supported.

Test this rule on your site →Run a full audit

Test your site for robots.txt limits: the 500 kib cutoff and four directives google ignores

What it detects

Two independent checks run against the fetched robots.txt, and an empty or absent file short-circuits both before either executes. The first measures the content's UTF-8 byte length, bytes rather than characters, so multibyte path segments in localised Disallow lines count for more than their character count suggests, and compares it against 500 KiB, which is 512,000 bytes exactly. Above that, a warning-severity, high-confidence finding reports the measured byte count alongside the limit and states that every rule past the cutoff is dropped. Its fix text points at wildcard consolidation of repetitive Disallow patterns and at relocating per-page exclusions onto the pages themselves.

The second check scans line by line for four directives Google's parser does not support, matching them only when they lead the line ahead of the colon: noindex, crawl-delay, nofollow and host. Whichever are present roll up into a single finding rather than one per line, each annotated with the reason it does nothing, so crawl-delay is marked as ignored by Google though honoured by Bing, nofollow as not a robots.txt directive at all, and host as unsupported with canonical URLs or redirects named as the replacement. Severity on this finding is conditional: it sits at info when only the harmless three appear, and escalates to warning as soon as noindex is among them, because that is the one an operator is likely to believe is working.

Why it matters

Faceted navigation is the workload that breaks robots.txt. A marketplace carrying 40 brands, 18 sizes, 12 colours and 9 price bands mints parameter permutations faster than any exclusion list can enumerate them, and the standard response, a build step appending one line per discovered facet URL, turns a configuration file into a growing artefact. At 700 KiB, roughly 200 KiB of that file sits past the point Google reads. Which 200 KiB is determined by the generator's ordering, which usually means alphabetical, which usually means the facets nobody thought about are the ones still being crawled.

The noindex: lines are the more serious half. Google announced on July 2, 2019 that it would stop supporting the directive, and did so on September 1, 2019. A URL listed under noindex: in robots.txt has since been crawlable and indexable exactly as if the line were absent. An operator reading their own robots.txt sees thousands of facet URLs apparently excluded; Search Console shows them indexed. The two views never reconcile, because one of them describes a directive the parser discards. Meanwhile the crawl spend on ?colour=navy&size=11&sort=price_asc permutations is real, and it comes out of the same allocation the actual product pages need.

A page that fails

marketplace.example generates robots.txt from its facet index at build time: 11,400 Disallow: lines and 6,900 noindex: lines covering filtered category URLs, 717,312 bytes in total. pseolint returns two findings, a warning that the file is 717,312 bytes against the 512,000-byte limit with roughly 205,000 bytes never parsed, and a second warning for the unsupported noindex and crawl-delay directives. Every facet URL the team believed was excluded is indexed, and 40 of them outrank the canonical category pages.

A page that passes

The same marketplace after the generator is replaced by 9 wildcard patterns, Disallow: /*?*colour=, Disallow: /*?*sort= and 7 siblings, bringing robots.txt to 2,144 bytes and well inside the 512,000-byte budget. The 6,900 noindex: lines are deleted and the exclusions they were meant to express move to <meta name="robots" content="noindex, follow"> rendered on the filtered views themselves, while crawl-delay is dropped. pseolint returns no findings for the rule.

📄 configuration-example.js

// next.config.js
module.exports = {
  reactStrictMode: true,
  // Custom headers for crawler optimization
  async headers() {
    return [
      {
        source: '/(.*)',
        headers: [
          { key: 'X-Robots-Tag', value: 'index, follow' }
        ]
      }
    ];
  }
};

# Run local audit for this rule:

$npx pseolint --rule=tech/robots-txt-limits

How to fix it

1Measure the file in bytes rather than lines: curl -s https://example.com/robots.txt | wc -c tells you immediately whether you are anywhere near 512,000.
2Collapse per-URL Disallow lines into wildcard patterns keyed on the query parameter name rather than its values, so one /*?*colour= line replaces every colour permutation.
3Delete every noindex: line, since it has had no effect since September 2019, and re-express the intent as a robots meta tag or an X-Robots-Tag header on the filtered pages.
4Decide per facet whether you want it crawled-but-not-indexed or not crawled at all, because a Disallow blocks the fetch and therefore stops Google ever seeing a noindex tag on that URL.
5Keep crawl-delay only where Bing traffic justifies it, and use Search Console's crawl-rate controls for Google, which ignores the directive entirely.
6Add a build-time assertion that fails CI when robots.txt exceeds a byte budget you set comfortably below the 512,000-byte cutoff.

SpamBrain context

Nothing in this rule is a spam signal, and a bloated robots.txt attracts no policy action on its own. What it does is make a site's own index unpredictable, which is a poor position from which to argue when quality systems do assess the domain. The March 5, 2024 scaled-content-abuse update judges what Google has actually indexed, and a marketplace that believes 6,900 facet URLs are excluded while in fact serving them as thin, near-identical variants of nine real category pages is being assessed on a corpus it never intended to publish.

The interaction with crawl budget is where the cost compounds. Google's large-site crawl-budget guidance is explicit that faceted navigation is a primary source of wasted crawling, and the remedy it names is fewer crawlable URLs rather than more exclusion lines. A 717 KiB robots.txt is a symptom of a URL space that grew without a canonical strategy, and truncation at 512,000 bytes means the file cannot even fully describe the problem it was written to solve. Fixing the byte count with wildcards is a one-afternoon change; the durable fix is deciding which facet combinations deserve a URL at all and serving the rest behind POST filters or client-side state that never mints a crawlable address.

Frequently asked questions

Does noindex in robots.txt actually work?: No. Google announced the removal on July 2, 2019 and stopped supporting the directive on September 1, 2019, and lines using it are discarded by the parser. Pages listed there stay crawlable and indexable. The working equivalents are a noindex robots meta tag or an X-Robots-Tag: noindex response header served on the page itself.
What happens if robots.txt is bigger than 500 KiB?: Google parses the first 500 KiB, or 512,000 bytes, and ignores everything after it. No error is reported anywhere, so the rules past the cutoff simply do not exist as far as the crawler is concerned. On a generated file that means your effective exclusions are whichever ones the generator happened to write first.
How do I stop Google crawling faceted navigation URLs?: Use wildcard Disallow patterns matched on the query parameter name instead of enumerating values, so one line covers every permutation of that facet. Where you need the URLs crawled but kept out of the index, leave them crawlable and serve a noindex meta tag, since a disallowed URL is never fetched and its meta tag is never read.
Is crawl-delay in robots.txt worth keeping?: Only for Bing, which honours it. Googlebot ignores the directive outright, so a crawl-delay: 10 line does nothing to Google's crawl rate. pseolint reports it at info severity when it appears alone, and folds it into a warning-severity rollup as soon as a noindex line shows up in the same file.
Why is my robots.txt byte count higher than its character count?: Length is measured in UTF-8 bytes, so any non-ASCII character in a path, whether an accented category name or a non-Latin script, occupies two to four bytes each. A localised marketplace can sit comfortably under 500,000 characters and still cross the 512,000-byte parse limit without noticing.

How this shows up in practice

Verdanto Marketplace generates robots.txt from its facet index, a build step platform engineer Ilse Bergkamp maintained from Utrecht. The generated file reached 717,312 bytes across 11,400 Disallow lines and 6,900 noindex lines. pseolint raised a warning at high confidence on size, since Google parses the first 512,000 bytes and drops roughly 205,000 bytes of rules past that point, and a second warning (rather than the info tier used when only crawl-delay is present) because the file carried noindex, unsupported since September 2019, alongside crawl-delay, which Google never read. Bergkamp, who had traced the first stray facet listing on April 23, 2026, replaced the enumerated facets with 9 wildcard patterns on May 21, 2026, taking the file to 3,180 bytes, and moved the genuine exclusions to X-Robots-Tag headers on the facet URLs themselves. By 2026-06-15 the 40 facets that had outranked their canonical categories were gone, and parameter-URL crawl requests fell 29.4% over 8 weeks, freeing fetches for 2,100 product pages last refetched more than 90 days earlier.

Sources

Google Search Central: How Google interprets the robots.txt specification: Google's robots.txt specification documents both halves of this rule: the 500 KiB parse ceiling of 512,000 bytes beyond which rules are ignored, and the directives its parser does not support, with noindex unsupported since September 1, 2019.
Google Search Central: Large site owner's guide to managing crawl budget: The large-site crawl-budget guidance names faceted navigation as a leading source of wasted crawling, which is why a 717 KiB generated exclusion list is treated here as evidence of an unbounded URL space rather than merely a file-size problem.

Related rules

Want to know whether this rule actually fires on your site?

Run pseolint against your sitemap. The audit is free, takes about a minute, and returns a per-URL list of every rule that fired (including this one) with the exact metric values so you can prioritise the fix queue.

Open the thin content scanner All rules