SEO: Analyzing Googlebot Crawl Budget Problems and Inefficiencies

This article is based on original reporting from Practical Ecommerce. Methodology and examples reference that site’s own log files and sitemaps.

Why Crawl Budget Matters for Large Sites

Crawl budget inefficiencies can quietly undermine organic search performance: if Googlebot never crawls a new or updated page, that page cannot be indexed, and it will not rank. For most small sites this is a non-issue, but high-volume ecommerce and publishing sites — with tens or hundreds of thousands of URLs — can run into real problems.

Google’s Webmaster Central Blog post “What Crawl Budget Means for Googlebot” defines crawl budget as the combination of two factors:

• Crawl rate — how fast Googlebot can fetch pages without overloading the server.
• Crawl demand — how much Google actually wants to crawl a given URL based on its perceived importance and freshness.

Google states that sites with fewer than a few thousand pages are unlikely to notice crawl budget constraints. However, ecommerce sites routinely exceed that threshold, making crawl budget analysis a worthwhile periodic exercise.

The Goal: Find Pages That Should Be Crawled But Aren’t

The analysis described here aims to answer two questions:

1. Are there pages in the XML sitemap that Googlebot has never crawled?
2. How quickly does Googlebot pick up page modifications after the sitemap is updated?

To answer both questions, you need two data sources: your XML sitemaps (which represent what should be crawled) and your web server log files (which record what was actually crawled). Comparing the two surfaces gaps.

Step 1 — Isolating Real Googlebot Requests in Server Logs

Server logs contain requests from many different bots — Bingbot, various SEO crawlers, and tools that spoof the Googlebot user-agent string. Before drawing any conclusions, you must filter down to verified Googlebot traffic only.

Screaming Frog’s Log File Analyser makes this straightforward:

1. Open Log File Analyser and drag your log file(s) into the “Drag and Drop Log Files Here” zone.
2. In the Project tab, go to New > User Agents.
3. Tick “Verify Bots When Importing Logs (Slows down Import)”. This performs a double-reverse DNS lookup against Google’s published IP ranges, confirming each request genuinely originated from Google’s infrastructure — the method Google describes in its Search Console help article “Verifying Googlebot.”

Bot verification can dramatically reduce your dataset. In the Practical Ecommerce example, enabling verification dropped the Googlebot request count from 306,960 to 112,308 — meaning roughly half of the apparent Googlebot requests were spoofed or tool-generated.

Exporting a Clean CSV from Log File Analyser

Once Log File Analyser has finished processing:

1. Set Verification Status to Show Verified to exclude all unverified bot entries.
2. Confirm that the log timestamps are correctly parsed — set your local timezone in the project settings if needed.
3. Export to CSV. This file will be your crawl-activity dataset for comparison.

Step 2 — Extracting URLs and Modification Dates from XML Sitemaps

Your XML sitemaps represent the set of URLs you want Googlebot to crawl and index. For this analysis to be meaningful, two assumptions must hold:

• The sitemaps are comprehensive and contain only canonical, indexable URLs.
• The <lastmod> dates in the sitemap entries are accurate and updated whenever content changes.

Screaming Frog’s SEO Spider can download sitemap URLs but discards the <lastmod> timestamp — which is essential for answering question two. Use Python instead.

Parsing Sitemaps to a Pandas DataFrame with Python

The following approach expands an XML sitemap index, fetches each child sitemap, and loads all URLs plus their modification dates into a Pandas DataFrame — a tabular data structure well-suited to the join operations that follow.

import requests
import pandas as pd
from xml.etree import ElementTree as ET

SITEMAP_NS = '{http://www.sitemaps.org/schemas/sitemap/0.9}'

def fetch_sitemap(url):
    """Fetch an XML sitemap or sitemap index and return a DataFrame of (loc, lastmod)."""
    response = requests.get(url, timeout=30)
    response.raise_for_status()
    root = ET.fromstring(response.content)

    records = []

    # Sitemap index — recurse into child sitemaps
    for sitemap_tag in root.findall(f'{SITEMAP_NS}sitemap'):
        child_url = sitemap_tag.findtext(f'{SITEMAP_NS}loc')
        if child_url:
            records.extend(fetch_sitemap(child_url).to_dict('records'))

    # Standard sitemap — collect URL entries
    for url_tag in root.findall(f'{SITEMAP_NS}url'):
        loc     = url_tag.findtext(f'{SITEMAP_NS}loc', default='')
        lastmod = url_tag.findtext(f'{SITEMAP_NS}lastmod', default='')
        records.append({'url': loc, 'lastmod': lastmod})

    return pd.DataFrame(records)

# Replace with your sitemap index URL
sitemap_df = fetch_sitemap('https://www.example.com/sitemap_index.xml')
sitemap_df['lastmod'] = pd.to_datetime(sitemap_df['lastmod'], errors='coerce')
sitemap_df.to_csv('sitemap_urls.csv', index=False)
print(sitemap_df.head())
Code language: PHP (php)

Export sitemap_urls.csv and import it into Log File Analyser under File > Import URLs.

Step 3 — Identifying Uncrawled URLs

With the sitemap URLs loaded into Log File Analyser alongside your verified Googlebot log data:

1. Navigate to the URLs tab.
2. Select “Not in Log File” from the filter dropdown.

The resulting list shows every URL that exists in your sitemap but received zero verified Googlebot visits during the log period. These are your highest-priority pages to investigate: Googlebot is either unaware of them, deprioritising them, or blocked from reaching them.

Common causes for URLs appearing here include:

• Pages blocked by robots.txt or a noindex directive contradicting the sitemap entry.
• Pages with very low PageRank receiving low crawl demand.
• Crawl budget consumed by low-value URLs (parameter duplicates, thin paginated pages, etc.) before Googlebot reaches these pages.
• Sitemap errors that prevent Googlebot from discovering the URLs at all.

Step 4 — Measuring How Quickly Googlebot Picks Up Changes

Log File Analyser does not directly support comparing <lastmod> dates from sitemaps to crawl timestamps. Return to Python for this step.

Load the CSV export from Log File Analyser into a second DataFrame, then merge the two on the URL column using a left join. A left join retains every sitemap URL (including those never crawled) while capturing crawl timestamps for those that were.

import pandas as pd

# Sitemap DataFrame from Step 2
sitemap_df = pd.read_csv('sitemap_urls.csv', parse_dates=['lastmod'])

# Log File Analyser CSV export
# Adjust column names to match the actual export headers from your version of the tool
log_df = pd.read_csv('log_file_analyser_export.csv', parse_dates=['Date'])
log_df = log_df.rename(columns={'Address': 'url', 'Date': 'crawl_date'})

# Keep only the most recent crawl per URL
latest_crawl = log_df.groupby('url', as_index=False)['crawl_date'].max()

# Left join: keep all sitemap URLs, attach the latest crawl date where available
merged_df = sitemap_df.merge(latest_crawl, on='url', how='left')

# Calculate lag between last modification and most recent crawl
merged_df['days_to_crawl'] = (
    merged_df['crawl_date'] - merged_df['lastmod']
).dt.days

# Surface pages modified but not yet re-crawled
uncrawled_updates = merged_df[
    merged_df['crawl_date'].isna() | (merged_df['crawl_date'] < merged_df['lastmod'])
]

print(f"Pages modified but not re-crawled: {len(uncrawled_updates)}")
uncrawled_updates.to_csv('uncrawled_updates.csv', index=False)
Code language: PHP (php)

The output CSV gives you:

• Every URL whose lastmod date is more recent than the most recent Googlebot crawl — pages updated since they were last crawled.
• Every URL with no crawl record at all.
• A days_to_crawl column showing the lag for pages that were eventually re-crawled.

Interpreting and Acting on the Results

A large list of uncrawled or stale URLs does not by itself confirm a crawl budget problem — it could mean your log retention window is too short, or that the URLs were added very recently. Cross-reference the following before drawing conclusions:

• Log period length: Googlebot may not crawl every page every day. A 30-day log window gives a more reliable picture than a 7-day window.
• Google Search Console Coverage report: Corroborate uncrawled URLs against the “Discovered – currently not indexed” and “Crawled – currently not indexed” buckets.
• robots.txt and meta robots: Verify that no directives are accidentally blocking the uncrawled pages.
• Low-value URL waste: Use the log file to find URL patterns Googlebot is spending time on that deliver little value — infinite scroll parameters, session IDs, filter combinations — and block or consolidate them to reclaim budget.

Summary

Diagnosing Googlebot crawl budget problems requires comparing intent (XML sitemaps) against reality (server logs). The workflow is:

1. Import and verify server logs in Screaming Frog Log File Analyser, filtering to confirmed Googlebot IPs only.
2. Parse XML sitemaps — including <lastmod> dates — into a structured CSV using Python and Pandas.
3. Use Log File Analyser’s “Not in Log File” filter to surface URLs that have never been crawled.
4. Use a Pandas left join to measure the lag between content updates and Googlebot re-crawls.
5. Prioritise fixing blocking issues, improve internal linking to low-crawl pages, and remove or consolidate low-value URLs wasting crawl budget.

For further context on using server logs for SEO diagnostics, search for Google’s Webmaster Central Blog post “What Crawl Budget Means for Googlebot” and the Search Console help article “Verifying Googlebot.”