Get a free consultation
Website Design

Googlebot’s File Size Limit: Is Your HTML Too Big to Be Fully Crawled?

By Sigmato Editorial TeamUpdated 15 Nov 20266 min read

Closeup of switch in server with connectors and adapters connected to plastic device in dark room on blurred background

Key takeaways

  • When Googlebot crawls for Google Search, it fetches the first 2 MB of a supported file such as an HTML page. For PDFs the limit is 64 MB.
  • Anything past the limit is not used. Googlebot “stops the fetch and only sends the already downloaded part” for indexing.
  • The limit applies to uncompressed data, so gzip or Brotli does not help you stay under it.
  • Every CSS and JavaScript file is fetched separately and has the same limit, so one huge bundle can be cut off too.
  • Most pages are nowhere near 2 MB. The ones at risk carry inline images, huge script state, giant tables or long lists rendered all at once.
Closeup of switch in server with connectors and adapters connected to plastic device in dark room on blurred background

What Google actually says

Google documents file size limits in two places, and it helps to keep them apart.

  • Google’s crawlers in general: by default they crawl only the first 15 MB of a file, and content after that is ignored. Individual crawlers can use different limits.
  • Googlebot for Google Search: the Googlebot page says it “crawls the first 2MB of a supported file type” and the first 64 MB of a PDF. Each resource referenced in the page, such as a stylesheet or a script, is fetched on its own and is bound by the same limit.

Google moved its file size information into the crawler documentation on 3 February 2026. Third-party coverage at the time described this as a cut from 15 MB to 2 MB, while Google’s wording suggests these limits were being documented in more detail. Whether it was a change or a clarification does not matter for your site. What matters is that 2 MB of uncompressed HTML is the number to design for.

Why compression does not save you

The limit applies to uncompressed data. A page that travels as 300 KB of gzip but expands to 3 MB of HTML is a 3 MB page as far as this limit is concerned. So the number that matters is the size of the HTML after your server’s compression is undone, not the “transferred” figure in the Network panel.

What gets lost if a page goes over

Googlebot reads from the top of the file downwards and stops at the cutoff. Whatever it has downloaded is what Search can use. In practice, the content at the end of an oversized page is the part at risk:

  • text and headings placed late in the document
  • internal links in the footer or in a long list, which can stop Google discovering those URLs
  • structured data (JSON-LD) that was printed at the bottom of the page
  • anything a script prints after the cutoff

Who is actually at risk?

Almost every normal content page is well under the limit; a typical article or service page is a few hundred kilobytes at most, and often far less. The risk shows up on pages like these:

Culprit Why it gets big Fix
Inline base64 images Image data pasted straight into the HTML or CSS Serve images as normal files with width and height
Framework state printed in the page React or Next.js apps can embed a large JSON blob so the page can hydrate Send only the data the page needs; fetch the rest after load
Huge inline scripts or styles Whole libraries or framework CSS pasted inline Move them to external files, and keep each file lean
Very long lists and tables Thousands of rows or products rendered at once Paginate with real links, or split into sensible category pages
Deep page-builder markup Many nested wrappers around every block Simplify sections; use a lean template for key pages
Repeated mega menus The same large navigation printed in several places Print each menu once; use CSS for responsive behaviour
Giant JSON-LD blocks Every product or location embedded on one page Mark up what the page is about, not the whole catalogue

How to measure your HTML size

Ask for the page without compression and count the bytes. From a terminal:

curl -s -H "Accept-Encoding: identity" https://example.com/page/ | wc -c

That prints the uncompressed size in bytes. To check many URLs at once, this small Python script reads your sitemap and flags pages above a warning threshold:

import re, urllib.request

SITEMAP = "https://example.com/sitemap.xml"
WARN = 1_000_000   # flag anything over about 1 MB of uncompressed HTML

def get(url):
    req = urllib.request.Request(url, headers={"Accept-Encoding": "identity", "User-Agent": "size-check"})
    return urllib.request.urlopen(req, timeout=30).read()

urls = re.findall(r"<loc>([^<]+)</loc>", get(SITEMAP).decode("utf8", "ignore"))
for u in urls[:200]:
    size = len(get(u))
    if size > WARN:
        print(f"{size:>10,} bytes  {u}")

Run it against your most important templates first: the home page, a category page, a long article and a product page. You can also open Chrome DevTools, choose the Network tab and compare the “size” and “resource size” columns for the document request. The larger number is the uncompressed size.

How to bring a heavy page down safely

  1. Find what is big. View the page source and look for long runs of inline data: base64 strings, huge JSON, large inline scripts or very repetitive markup.
  2. Move heavy assets out of the HTML. Put images, large scripts and styles in separate files. Remember that each file has its own limit too, so keep bundles reasonable.
  3. Do not hide real content behind a “load more” button that only works through script. If you have a long list, paginate with normal links so every item has a crawlable page.
  4. Trim framework state. Pass the page only the data it renders, and avoid serialising whole application stores into the document.
  5. Put the important things early. Title, main heading, primary content, canonical link and key structured data belong near the top of the document, not at the end.
  6. Reduce nesting. Simplify sections and remove empty wrappers; this also helps rendering speed and the Interaction to Next Paint metric.
  7. Set an internal budget. We suggest treating about 500 KB of uncompressed HTML as a “look closer” line and 1 MB as a hard warning, well below the 2 MB cutoff. That is our own safety margin, not a Google number.

How to check what Google actually saw

Use the URL Inspection tool in Search Console and open the crawled page and the rendered result. If a section, link or block of structured data is missing from what Google fetched, compare it with the position of that content in your HTML. Content that starts after a very large block of inline code is the usual suspect.

A quick audit checklist

  • Measure the uncompressed HTML of your five most important templates.
  • Search the source for data:image and for very long inline JSON.
  • Check that every important internal link appears early enough in the document and as a normal <a href> link.
  • Confirm structured data is in the head or near the top of the body, not at the very end.
  • Measure your largest JavaScript and CSS files separately.
  • Re-test after every theme or plugin update, because bloat usually arrives with an update.

Where this fits with everything else

A small, clean document helps crawling, helps speed and helps responsiveness, so none of this work is wasted. Pair it with our guides on Core Web Vitals and WordPress speed optimisation, and with our post on improving INP. If you would like a size and crawl audit of your own site, our website maintenance team can run one, and you can get in touch.

Frequently asked questions

What is Googlebot’s file size limit?

For Google Search, Googlebot fetches the first 2 MB of a supported file type such as HTML, and the first 64 MB of a PDF. Google’s crawlers in general default to the first 15 MB of a file. Content beyond the limit is not used.

Does gzip or Brotli compression help me stay under the limit?

No. The limit applies to uncompressed data, so a page that transfers as 300 KB but expands to 3 MB still counts as 3 MB.

Do CSS and JavaScript files count towards the page limit?

Each referenced resource is fetched separately and is bound by the same limit on its own. They do not add up with the HTML, but a single oversized script can be cut off.

Will my normal blog posts hit the limit?

Almost certainly not. A typical page is a small fraction of 2 MB. The limit matters for pages with inline images, huge script state, very long lists or deeply nested markup.

How do I check my page size?

Run curl with Accept-Encoding set to identity and count the bytes, use Chrome DevTools to compare size with resource size, or script a check across your sitemap.

Sources and further reading

Sigmato Editorial Team

Written by

Sigmato Editorial Team

The Sigmato Editorial Team is made up of software engineers, UI/UX designers and AI specialists at Sigmato Solutions Pvt Ltd in Bengaluru, India and the UK. We write practical guides on web, mobile, cloud and AI development, drawing on the products we design and build for startups and enterprises worldwide.

Last updated 15 November 2026 · Talk to our team