Rule referencetech/meta-robots-conflict

Meta Robots Conflict: How a Stray noindex Header Deindexes Live Pages

Google honours the most restrictive robots directive it finds rather than the nearest one, so an index, follow meta tag cannot override a stray X-Robots-Tag: noindex response header: tech/meta-robots-conflict fires at error severity the moment the two disagree across a URL's robots meta tag, its googlebot meta tag and its headers. Unlike the robots.txt noindex directive that Google retired in 2019, this conflict is silent, and Search Console reports only that the page is excluded by a noindex tag without naming which source set it.

Test this rule on your site →Run a full audit

Test your site for meta robots conflict: how a stray noindex header deindexes live pages

What it detects

Re-scanning the raw HTML is the first thing this rule does, and it does so deliberately: the parser's page.robotsMeta field retains only the first robots meta tag, so a docs template that injects a second one from a layout partial would otherwise go unseen. A regex sweep over every <meta> tag keeps the ones whose name attribute resolves to robots or googlebot, whether the value is double-quoted, single-quoted or bare, and records each tag's content string against a readable source label. The X-Robots-Tag value from the HTTP response is appended as a third declaration and skipped when the header is empty or whitespace-only. Every content string is then split on commas, trimmed and lowercased into individual directive tokens, and each token is indexed against the set of sources that declared it.

Two opposite pairs are then evaluated: index against noindex, and follow against nofollow. When both halves of a pair are present anywhere across the gathered sources, the rule emits an error-severity, high-confidence finding that names the offending URL and lists which source declared each side, so a /pricing page showing index from meta robots and noindex from X-Robots-Tag header reads unambiguously in the report. A second, quieter check catches a different failure: when the same meta name appears in two or more tags whose content strings differ after trimming and lowercasing, the rule emits a warning that quotes each distinct string, because nothing in the markup settles which one applies. Identical duplicate tags stay silent, and a page that declares no robots directive at all is skipped before any comparison runs.

Why it matters

The asymmetry is what makes this expensive. A missing index directive costs nothing, since indexing is the default behaviour. A stray noindex removes the page. Because Google combines directives from every source and honours the most restrictive result, the two are never weighed against each other; the restriction simply applies, with no error surfaced in the markup, no warning in the build log and no visible difference in the rendered page. Search Console will eventually list the URL under Excluded by 'noindex' tag, but that report only arrives after a recrawl, which on a low-demand documentation subfolder can take weeks.

Documentation and pricing are the two page types a SaaS site can least afford to lose. Pricing pages carry commercial intent and convert; documentation pages accumulate long-tail queries for error strings, API method names and config keys that nothing else on the domain ranks for. When a staging deployment's blanket X-Robots-Tag: noindex survives an environment-variable rename and reaches production, the page template still emits index, follow, every visual QA pass looks correct, and the deindexing proceeds quietly underneath. Teams routinely discover it only when a support ticket asks why a documented error code no longer appears in search.

A page that fails

docs.example-saas.com serves 412 documentation URLs plus /pricing behind a reverse proxy whose staging config block was pasted into the production Nginx file. Every response carries X-Robots-Tag: noindex, nofollow while the page template still emits <meta name="robots" content="index, follow">. The rule reports two error findings per URL, one for the index/noindex pair and one for follow/nofollow, 826 in total, each naming meta robots and X-Robots-Tag header as the conflicting sources.

A page that passes

The same 412 documentation URLs and /pricing after the proxy's add_header X-Robots-Tag line is scoped to the staging server block alone. Production responses carry no X-Robots-Tag at all, the template emits exactly one <meta name="robots" content="index, follow">, and the 3 internal preview environments declare noindex through the header only, with no robots meta tag rendered, so no opposite pair can assemble anywhere. pseolint returns zero findings for the rule.

📄 configuration-example.js

// next.config.js
module.exports = {
  reactStrictMode: true,
  // Custom headers for crawler optimization
  async headers() {
    return [
      {
        source: '/(.*)',
        headers: [
          { key: 'X-Robots-Tag', value: 'index, follow' }
        ]
      }
    ];
  }
};

# Run local audit for this rule:

$npx pseolint --rule=tech/meta-robots-conflict

How to fix it

1Fetch the production URL with curl -I and read the response headers directly: an X-Robots-Tag conflict is invisible in view-source and in every browser Elements panel.
2Pick one authoritative source per environment, header-only for preview and staging, meta-tag-only for production. Declaring the same directive twice is how the two drift apart.
3Move the staging noindex out of shared proxy config into a block scoped to the staging server name, so a copy-paste into the production file cannot carry it across.
4Resolve the warning-severity duplicate-tag findings too: two robots meta tags with different content strings mean a layout partial and a page template are both writing directives.
5Add a deploy-time smoke test asserting that /pricing and three sampled /docs URLs return no X-Robots-Tag header and render exactly one robots meta tag.
6After removing the directive, request indexing for the highest-value URLs in Search Console rather than waiting out the natural recrawl of a low-demand docs subfolder.

SpamBrain context

Robots directives sit outside the spam-policy stack, which is precisely why this failure mode is so durable: no classifier is going to reverse it on your behalf. Google's robots meta tag documentation states the combination rule plainly, that where the meta tag and the HTTP header disagree the more restrictive of the two applies, and it has read that way since long before the March 5, 2024 scaled-content-abuse update reshaped the surrounding policy language. Nothing about a page's quality, freshness or link profile overrides a directive the site itself declared.

For programmatic sites the consequence is a scoring one. A docs-and-pricing SaaS domain that loses 412 URLs to an accidental header is not demoted, it is removed, and the remaining index shrinks to whatever the marketing subfolder holds. Quality systems then assess the domain on that smaller, less useful sample. The May 7, 2024 site-reputation-abuse policy and the August 25, 2022 Helpful Content System both judge a domain on the corpus Google can actually see, and a self-inflicted noindex changes what that corpus is. Auditing robots directives before auditing content quality is the correct order of operations, because a deindexed page's content score is never read by anything.

Frequently asked questions

Which wins, the meta robots tag or the X-Robots-Tag header?: Neither wins by position or precedence. Google combines the directives from both sources and applies the most restrictive result, so a header saying noindex against a meta tag saying index produces a noindexed page, because noindex is the restrictive half of that pair. The meta tag is not a rebuttal to the header.
Why does my page show index in view-source but is missing from Google?: View-source renders the HTML body only. The X-Robots-Tag lives in the HTTP response headers, which no browser displays in its Elements panel. Fetch the URL with curl -I, or open the Network tab and read the response headers; a staging directive left behind in proxy or CDN config is the usual culprit.
Does a noindex plus nofollow conflict fire two separate findings?: It can. The rule evaluates two opposite pairs independently, index against noindex and follow against nofollow, so a header declaring noindex, nofollow against a meta tag declaring index, follow produces two error-severity findings for the same URL, each naming the pair it matched and the sources on either side of it.
Is having two meta robots tags on one page a problem?: Only when their content strings differ. Two identical tags are redundant but harmless and the rule stays quiet. Two tags carrying different directives raise a warning, because the markup gives no ordering guarantee, and Google resolves the ambiguity by combining them restrictively, which is rarely what the second tag's author had in mind.
How long does it take to recover after removing a noindex header?: The URL has to be recrawled before it can re-enter the index, and a documentation subfolder with low crawl demand can wait weeks for that to happen on its own. Submitting the highest-value URLs through Search Console's URL Inspection tool shortens the wait for those specific pages; the remainder recover on the normal crawl cycle.

How this shows up in practice

Quillstack, a Tallinn SaaS serving docs and pricing behind one reverse proxy, pasted a staging Nginx block into its production config on March 18, 2026. Every response then carried X-Robots-Tag: noindex, nofollow while the page template kept emitting meta robots content of index, follow. pseolint returned 826 findings at error severity and high confidence across 413 URLs, two apiece, because index against noindex and follow against nofollow are both opposite pairs the rule tests, and each finding named the meta robots tag and the X-Robots-Tag header as its conflicting sources. Reliability engineer Marek Dobrzycki removed the block and verified a clean header on 2026-04-06. Since the most restrictive directive wins, the docs tree had been leaving the index without a single build warning: organic entries to the pricing page fell 74.3%, near $14,800 of monthly trial pipeline, and full reindexing took 34 days.

Sources

Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag: Google's robots meta tag specification is the primary source for the combination rule this page rests on: directives declared in the HTML tag and in the X-Robots-Tag header are merged and the most restrictive result applies, which is why a leftover staging header overrides a production meta tag saying index.
Google Search Central: Block search indexing with noindex: The block-indexing guidance documents that a noindex directive removes a URL from the index outright rather than demoting it, the outcome a docs-and-pricing SaaS site suffers when 412 documentation URLs inherit a proxy header that was only ever meant for staging.

Related rules

Want to know whether this rule actually fires on your site?

Run pseolint against your sitemap. The audit is free, takes about a minute, and returns a per-URL list of every rule that fired (including this one) with the exact metric values so you can prioritise the fix queue.

Open the spambrain checker All rules