Rule referencetech/html-size

HTML Size: The 2 MB Per-File Cutoff That Truncates Your Markup

Googlebot reads only the first 2 MB of any single fetched file, uncompressed, so tech/html-size warns the moment a document's UTF-8 byte length reaches 1,572,864 bytes (1.5 MB) and escalates to an error at 2,097,152 bytes (2 MB), the point where markup, links, and JSON-LD past the cutoff stop existing as far as Google is concerned.

Test this rule on your site →Run a full audit

Test your site for html size: the 2 mb per-file cutoff that truncates your markup

What it detects

The rule measures one number: the UTF-8 byte length of the served HTML document, computed with Buffer.byteLength over the page markup. Anything below 1,572,864 bytes (1.5 MB) is skipped before a finding is ever constructed, so most sites never see this rule fire at all. From 1.5 MB up to the cutoff the finding is a warning; at 2,097,152 bytes (2 MB) and above it becomes an error. Both tiers carry high confidence, and both name the page URL alongside its size rounded to one decimal place. There is no site-wide aggregate, no percentage, and no averaging: each document is judged on its own bytes, which is exactly how Googlebot fetches it.

What the rule deliberately does not measure matters as much. Google's Googlebot documentation caps a crawl at the first 2 MB of each fetched file, uncompressed, with PDFs given 64 MB; the February 2026 revision of that page replaced the 15 MB figure most audit tools still quote. The budget is per resource, so a stylesheet, a map bundle, and every JSON file a page loads each carry their own 2 MB allowance. A dataset page whose HTML is 118 kB while it streams 21 MB of vector tiles has a Core Web Vitals conversation ahead of it and no truncation problem whatsoever, and tech/html-size stays quiet on it. Compression is beside the point too: the limit applies to the uncompressed document, so a 340 kB gzip response can be a 2.6 MB file as far as the cutoff is concerned.

On a municipal open-data portal the trigger is almost always a hydration blob. Cascade County's portal renders each of its 1,840 dataset pages by inlining the entire record set into a script tag so a client-side grid can boot without a network round trip; on /datasets/street-tree-inventory that payload is 2.4 MB and the finished document weighs 2.6 MB. The rule reports 2.6 MB at error severity, and the remedy is structural rather than editorial. Nothing about the prose is wrong. The document is simply carrying data that belongs behind an endpoint.

Why it matters

Truncation is silent. There is no HTTP error, no Search Console message, and no rendering failure to notice, because bytes past the cutoff are never handed to the parser and the crawled copy of the page just ends mid-document. On the street-tree page everything the county actually wanted indexed sits after the 2.4 MB payload: the license notice, the weekly refresh cadence, the 40 links to sibling datasets, and the schema.org Dataset JSON-LD. All of it lands past byte 2,097,152. Google receives an H1, a breadcrumb, and the opening of a script tag.

The link graph is the part that compounds. Each of those 1,840 dataset pages carries 40 outbound links to related records, every one of them on the far side of the cutoff, so a portal that looks densely interlinked in a browser hands Google 1,840 dead ends instead. Crawl economics degrade in parallel: a full pass moves roughly 4.7 GB while each document weighs 2.6 MB, against about 173 MB once the payload is served separately, and Google's crawl-budget guidance for large sites is direct about low-information fetches being deprioritised. A county that publishes 1,840 datasets is spending its entire crawl allowance transmitting a table Google will never read.

For an open-data portal the JSON-LD is not decoration. Google's Dataset structured-data support is what lets a municipal record set surface as a dataset rather than as an ordinary web page, and structured data has to be inside the fetched bytes to register. Cascade County can maintain flawless schema.org Dataset markup on all 1,840 pages and have precisely none of it counted, because in every single case the markup is written after the payload that pushed it past the cutoff. The failure is invisible in a browser, invisible in any check that pastes markup into a validator instead of fetching the URL, and invisible to anything reading the DOM after JavaScript has run.

A page that fails

data.cascadecounty.gov/datasets/street-tree-inventory ships 2.6 MB of HTML. A script tag of type application/json holds the full street-tree table inline, 86,400 rows of species, planting date, and trunk diameter, weighing 2.4 MB on its own so a client-side grid can render without a fetch. The license notice, the 40 related-dataset links, and the schema.org Dataset JSON-LD all sit after that block, past byte 2,097,152. tech/html-size fires an error, and Google indexes a page that appears to consist of a heading and nothing else.

A page that passes

The same URL after the payload moves to /api/datasets/street-tree-inventory.json, fetched on demand and paged 500 rows at a time: the document drops from 2.6 MB to 96 kB, comfortably under the 1.5 MB warning tier, and the rule stays silent. The server now renders a 12-row preview, the row count (86,400), the Monday 06:00 refresh cadence, the license notice, the schema.org Dataset JSON-LD, and all 40 related-dataset links inside the first 100 kB of markup, which puts every one of them not merely inside the 2 MB budget but near the top of it.

📄 configuration-example.js

// next.config.js
module.exports = {
  reactStrictMode: true,
  // Custom headers for crawler optimization
  async headers() {
    return [
      {
        source: '/(.*)',
        headers: [
          { key: 'X-Robots-Tag', value: 'index, follow' }
        ]
      }
    ];
  }
};

# Run local audit for this rule:

$npx pseolint --rule=tech/html-size

How to fix it

1Move hydration blobs out of the document: serve the 2.4 MB record set from a JSON endpoint the grid fetches after paint and keep the HTML a shell around it.
2Server-render a bounded preview instead of the whole table. Twelve rows plus a row count of 86,400 communicate the dataset as well as 86,400 inlined rows do, and cost four orders of magnitude less.
3Order the document so load-bearing markup comes first: JSON-LD, canonical, license text, and outbound dataset links belong above any large payload, so they survive even if a page creeps back over the line.
4Measure the byte length of the served HTML rather than the transfer size. The 2 MB budget is uncompressed, so a 340 kB gzip response can still be a 2.6 MB document.
5Treat the 1.5 MB warning tier as the real deadline. A dataset page sitting at 1.6 MB is one weekly refresh away from silent truncation, and nothing will tell you when it crosses.
6Split by resource rather than trimming blindly: the map bundle and the stylesheet each get their own 2 MB fetch allowance, so relocating bytes into a second file is a genuine fix, not a loophole.

SpamBrain context

Nothing about a large HTML file is a spam signal and Google publishes no penalty for one. The reason a size check belongs in an audit that mostly hunts scaled-content abuse is second-order: truncation manufactures thin content out of a page that is genuinely substantial. The crawled version of the street-tree page is a heading, a breadcrumb, and a dangling script tag, which is exactly the shape that the scaled-content-abuse clause Google added to its spam policies on March 5, 2024, plus the helpful-content guidance beside it, are tuned to distrust. The page is not low-effort. It is only being read as though it were, and 1,840 pages read that way describe a domain rather than an accident.

The 2 MB number also attracts folklore, and pseolint takes a position on it. The claim that search engines cannot process more than 2 MB of total website size is a real figure bolted to the wrong object: the limit is per fetched file, uncompressed, and each resource carries its own. The 15 MB figure is not so much wrong as expired, having been the documented limit for years until the February 2026 revision of Google's Googlebot page cut it, and PDFs sit at 64 MB rather than either number. pseolint's folklore list keeps both entries on purpose, so the rule fires on the number in the living document rather than on a remembered one.

That is also why the rule refuses to grade total page weight. A 21 MB dataset page whose HTML is 118 kB is a Core Web Vitals matter handled by a different check, and merging the two produces the worst species of audit finding: technically alarming and actionably wrong. tech/html-size answers exactly one question, whether Google will read to the end of this document, and a county portal with 1,840 dataset pages is precisely the scale at which that answer stops being academic.

Frequently asked questions

Is there a 2 MB limit on total page size for SEO?: No. The documented 2 MB applies to each file Googlebot fetches, uncompressed, and every stylesheet, script, and JSON file gets its own budget; PDFs are allowed 64 MB. A dataset page with 118 kB of HTML pulling 21 MB of map tiles is never truncated. It may well have a Core Web Vitals problem, which is a separate check with separate remedies.
Does Googlebot still index the first 15 MB of HTML?: That figure was accurate for years and most audit tools still repeat it, which is why pseolint documents it as expired rather than as a myth. The February 2026 revision of Google's Googlebot page cut the documented per-file crawl limit to 2 MB, and tech/html-size flags against the current number: warning from 1.5 MB, error from 2 MB.
Does gzip compression count toward Googlebot's crawl limit?: No, and this trips up teams who check transfer size in DevTools. The cutoff applies to the uncompressed document, so a dataset page arriving as a 340 kB gzip response can still be a 2.6 MB file that Googlebot stops reading partway through. Measure the served bytes, not what the network panel reports.
Why does my open-data portal rank for nothing despite thousands of rows?: If the rows are inlined into the HTML, they are probably the reason. A 2.4 MB payload placed before the license notice, related-dataset links, and schema.org Dataset JSON-LD pushes all three past byte 2,097,152, so Google indexes a heading and a truncated script tag. The rows do not help you rank; they are actively hiding the content that would.
What HTML size does pseolint flag as a warning versus an error?: Documents under 1,572,864 bytes (1.5 MB) are skipped entirely and never produce a finding. From 1.5 MB to just under 2 MB the rule reports a high-confidence warning that the page is approaching the cutoff. At 2,097,152 bytes and above it reports an error, because content past that point is already being dropped rather than merely at risk.

How this shows up in practice

Cascade County's open-data team inlines each dataset table into the document it belongs to. Portal manager Hollis Ferrantino pulled a run on July 9, 2026 and found 1,840 dataset pages past the 1,572,864-byte warning threshold, 512 of them also past 2,097,152 bytes and therefore reported as errors. The street-tree inventory served 2.6 MB, its 86,400-row JSON block taking 2.4 MB on its own and pushing the licence notice, the 40 sibling-dataset links and the schema.org Dataset JSON-LD beyond the point Googlebot stops reading a single fetched file. Ferrantino moved the table behind a paged API call, cutting the served document to 94 KB, and confirmed it on 2026-07-28. Sibling links reappeared in the crawl within 6 hours of the first refetch, internal-link coverage across the portal climbed from 52.7% to 91.3%, and the $118,000 transparency grant kept its published-access condition.

Sources

Google Search Central: Googlebot and its crawl limits: Google's Googlebot documentation sets the crawl limit at the first 2 MB of each fetched file, uncompressed, with 64 MB for PDFs; the February 2026 revision replaced the older 15 MB figure, and tech/html-size flags against the current number by warning at 1.5 MB and erroring at 2,097,152 bytes.
Google Search Central: Large site owner's guide to managing crawl budget: Crawl-budget guidance for large sites warns that low-information fetches get deprioritised, which is what a 1,840-page open-data portal invites when every dataset document carries a 2.4 MB inline payload: a full pass moves roughly 4.7 GB to deliver content Googlebot stops reading partway through.
web.dev: Web Vitals: Core Web Vitals guidance is where total page weight belongs, and this rule deliberately leaves it there. A dataset page with 118 kB of HTML pulling 21 MB of vector tiles is a performance question, not a truncation one, and tech/html-size stays silent on it.

Related rules

Want to know whether this rule actually fires on your site?

Run pseolint against your sitemap. The audit is free, takes about a minute, and returns a per-URL list of every rule that fired (including this one) with the exact metric values so you can prioritise the fix queue.

Open the spambrain checker All rules