How to Use Screaming Frog to Improve On-Page SEO (Without Wasting 6 Hours on the Wrong Reports)

Before you crawl a single URL, the configuration you choose will determine whether your audit data is accurate or dangerously misleading. This is the step that most tutorials rush past in two sentences. The default Screaming Frog settings are designed for general crawling — they are not optimised for an on-page SEO audit.

Here is how to configure a crawl that gives you clean, actionable data.

First, set your user agent to Googlebot. This matters because some sites serve different content to different user agents. If you crawl as Screaming Frog's default spider and your site's CDN or JavaScript rendering behaves differently for Googlebot, you are auditing a version of your site that Google never sees.

Go to Configuration > User-Agent and select Googlebot.

Second, enable JavaScript rendering if your site uses a JavaScript framework such as React, Vue, or Next.js. Go to Configuration > Spider > Rendering and switch from 'Text Only' to 'JavaScript'. This adds crawl time but gives you the rendered DOM — the actual content Google evaluates.

Auditing a JavaScript-heavy site without rendering enabled is one of the most common causes of 'I can't find any issues but rankings are stuck' situations.

Third, configure your crawl to respect or ignore canonicals depending on your audit goal. If you want to see what Google sees (de-duplicated), keep canonical respect enabled. If you want to find canonical misconfigurations, disable it temporarily and compare the two crawl outputs.

Fourth, connect your Google Search Console and Google Analytics integrations before you crawl. In Screaming Frog, go to Configuration > API Access. Connecting GSC pulls impression and click data per URL directly into your crawl export.

This is the single configuration change that transforms your audit from a technical exercise into a revenue intelligence report. You will be able to see which pages have high impressions but low clicks — a direct signal of title tag and meta description optimisation opportunity — and which pages rank for nothing despite being fully indexed.

Finally, set your crawl depth limit and page limit according to your site's architecture. For most sites under 10,000 pages, crawl everything. For larger sites, set a crawl depth of five to seven levels and prioritise the most commercially important sections first.

One of the core problems with standard Screaming Frog guides is that they give you a list of things to check without giving you an order of operations. The result is that most people fix what is easy — title tags, meta descriptions — and never address the issues causing the most ranking damage. The CANOPY Framework solves this by organising on-page audit findings into six layers, ordered by SEO impact and logical dependency.

CANOPY stands for: Content, Architecture, Navigation, Orphans, Performance, and Your-On-Page-Signals.

C — Content: Start here. Use Screaming Frog's 'Content' tab and enable custom extraction (covered in the next section) to identify thin content, duplicate body text, and pages with missing or inadequate heading structures. Pages that have fewer than 300 words on commercially important topics should be flagged for content expansion before any other on-page work is done.

Improving thin content gives Google more signal to work with and directly raises topical depth scores.

A — Architecture: Next, audit your site's internal linking structure and crawl depth. Use the 'Crawl Analysis' feature under Reports > Crawl Analysis to generate a crawl depth report. Any commercially important page sitting at crawl depth five or deeper is under-resourced from a crawl budget and internal authority perspective.

Move these pages shallower through internal linking before touching their on-page elements.

N — Navigation: Audit your navigation-level internal links. Export your inlink report and identify which pages receive the most internal links from your navigation and header elements. These pages signal topical authority to Google.

If your most internally-linked page is not your most strategically important page, you have a navigation structure problem — not an on-page problem.

O — Orphans: Orphaned pages are pages with no inlinks that Screaming Frog discovers through your sitemap but not through crawling. These pages are essentially invisible to Google. Export your sitemap URL list and your crawled URL list, then compare them.

Any URL in the sitemap but not found via crawl is orphaned. This is frequently where old blog posts and valuable resource pages go to die.

P — Performance: Use Screaming Frog's integration with PageSpeed Insights to pull Core Web Vitals data per URL. On-page SEO is not just about content — page speed and rendering performance affect how Google evaluates and ranks individual URLs.

Y — Your On-Page Signals: Finally, address title tags, meta descriptions, H1 and H2 structure, image alt attributes, and schema markup. This is the layer most guides start with. We do it last — because it only has full impact once the layers above it are resolved.

Custom extraction is the most underused feature in Screaming Frog, and arguably the most powerful for on-page SEO. It allows you to pull specific elements from every crawled page using CSS selectors, XPath, or regex — turning your crawl into a content audit at scale.

Here is how to access it: Configuration > Custom > Extraction. You can add up to three custom extraction fields in the free version and unlimited in the paid version.

Use Case 1: Heading Hierarchy Audit Create a custom extraction using CSS selector `h1` to pull every H1 tag from every page. Export this alongside your page titles. Now you can check in a single spreadsheet whether every page has exactly one H1, whether the H1 and title tag are aligned (they should be related but not necessarily identical), and whether any pages are missing H1s entirely.

Sort by H1 count to instantly flag pages with multiple H1s — a common issue on templated CMS sites.

Use Case 2: Schema Markup Detection Create an extraction using XPath: `//script[@type='application/ld+json']`. This pulls every JSON-LD schema block from every page. Now you can identify which pages have schema, which do not, and whether schema types are consistent with content type.

A product page without Product schema is a missed structured data opportunity. A blog post with no Article schema is leaving rich result eligibility on the table.

Use Case 3: Word Count Approximation Use `//body` as your XPath selector and extract the full body text. While Screaming Frog does not provide a native word count column, you can paste the exported body text into a word count formula in your spreadsheet. For a faster approach, the 'Text' tab in Screaming Frog shows 'Word Count' when you enable it under Configuration > Spider > Advanced — tick 'Check Spelling' or use the readability metrics.

Pages under a threshold you define (typically 300 words for informational content, 500 for commercial pages) should be flagged for content expansion.

Use Case 4: CTA Presence Check If you have a consistent call-to-action element with a class name across your site, you can extract it to verify it appears on every key commercial page. Use `.your-cta-class-name` as your CSS selector. Any page returning empty on this extraction is missing its primary conversion element.

Custom extraction essentially lets you build bespoke audit logic on top of Screaming Frog's crawl engine — without writing a custom crawler or using a separate tool.

This is the framework I wish someone had given me earlier in my career. It reframes crawl depth — traditionally a dry technical metric — as a direct predictor of which pages are being suppressed from their ranking potential.

The Crawl Depth Revenue Matrix works like this: export your crawl depth report from Screaming Frog (Reports > Crawl Analysis > Crawl Depth), then merge it with your GSC data (clicks and impressions per URL) and your Google Analytics data (sessions and conversions per URL). The resulting matrix has three dimensions:

- Crawl Depth (1–7+ levels from homepage) - Commercial Value (revenue attribution, conversion rate, or strategic priority) - Current Organic Visibility (GSC impressions and position)

Pages in the matrix quadrant of high commercial value + deep crawl depth + low organic visibility are your highest-priority internal linking opportunities. These are pages that likely rank poorly not because of poor on-page content, but because they are structurally under-resourced — Google is not crawling them frequently enough, and they receive too little internal link equity to compete.

The fix is not to rewrite the page. The fix is to add three to five contextually relevant internal links from shallower, higher-authority pages — and in many cases, to add the page to a relevant section of your main navigation or to a topic cluster hub page.

In our experience, this intervention alone — adding internal links from pages with strong crawl frequency to commercially important deep pages — produces measurable ranking improvement without any content changes. It works because you are solving the actual problem: crawl frequency and internal authority distribution.

How to build the matrix in Screaming Frog: 1. Run your crawl and connect GSC and GA4 APIs before crawling 2. Go to Reports > Crawl Analysis and export the depth report 3.

Export the full URL list with GSC data (clicks, impressions, position) 4. In your spreadsheet, create a column for crawl depth, one for impressions, one for commercial value (manual score or revenue data), and one for current average position 5. Colour-code by quadrant: green (shallow + visible), amber (shallow + invisible or deep + visible), red (deep + invisible + commercially important) 6.

Prioritise every red URL for internal linking sprint work before any other on-page intervention

Title tags and meta descriptions are the on-page elements that get the most attention in Screaming Frog guides — and the most superficial treatment. The standard advice is: fix duplicates, keep titles under 60 characters, include your keyword. That advice is not wrong, but it misses the nuance that separates average performance from measurable click-through rate improvement.

The Title Tag Audit Process in Screaming Frog In the 'Page Titles' tab, Screaming Frog flags issues as: Missing, Duplicate, Over 60 Characters, Under 200 Pixels, and Multiple. Start with missing titles — these are your most urgent fixes. A page without a title tag will have its title auto-generated by Google, often from the H1 or even from anchor text pointing to the page.

Next, look at duplicates. Export the duplicate title list and group by URL pattern. If you have 40 duplicate titles, they are almost certainly coming from one template — a category page, a tag archive, a product variant.

Fix the template, not the individual pages.

For title length, do not treat 60 characters as a hard rule. Screaming Frog flags titles over 60 characters, but what matters is pixel width, not character count. A title with narrow characters (i and l) at 65 characters may display fully in SERPs, while a title with wide characters (M and W) at 55 characters may get truncated.

The 'Title Pixel Width' column is more reliable than character count for this reason.

The CTR Optimisation Layer Here is what most guides skip: once your GSC integration is active, filter your page title tab to show pages with more than 200 impressions and a click-through rate below your site average. These pages are indexed, visible, and failing to earn clicks. Their title tags (and meta descriptions) are not compelling enough for the query context.

For these pages, open your GSC data and find the top queries driving impressions. Does the current title tag align with the language of those queries? Often it does not — the page was written with one keyword intent in mind and is actually ranking for a related but slightly different query.

Rewriting the title to align with the actual ranking query language (not the intended target keyword) frequently produces click-through rate improvement within two to four weeks — a faster feedback loop than most on-page changes.

Meta descriptions do not directly affect rankings, but they directly affect clicks. Treat meta descriptions as ad copy. Every meta description should answer: what does this page give me that others do not?

Internal linking is consistently undervalued in on-page SEO — and Screaming Frog is one of the best tools in existence for auditing and improving it, yet almost no guide covers this in meaningful depth.

The internal linking picture in Screaming Frog lives in two places: the 'Inlinks' tab (accessible by clicking any URL and viewing its inlink data) and the bulk export under Reports > All Inlinks. It is the bulk export that gives you the full picture.

Auditing Internal Link Distribution Export Reports > All Inlinks. This gives you every internal link on your site — source URL, destination URL, anchor text, and whether the link is follow or nofollow. Sort by destination URL and count inlinks per page.

This inlink count is a proxy for how much internal authority each page receives.

Now compare your inlink count per page against your GSC impressions per page. Pages with high impressions but low inlink counts are ranking despite limited internal support — they have organic merit but are being under-resourced. These are candidates for internal link building that will compound existing performance.

Pages with high inlink counts but low impressions have the opposite problem — they are internally well-resourced but failing on content or on-page signal quality. These need content work, not more internal links.

Anchor Text Audit In the same export, analyse your anchor text distribution for key commercial pages. If your most important service page receives 20 internal links but 18 of them use the anchor text 'click here' or 'learn more', you are wasting significant internal linking opportunity. Contextual, keyword-relevant anchor text in internal links is a direct on-page SEO signal.

Re-anchor existing links to use descriptive, topically relevant phrases.

Finding Link Gap Opportunities Screening Frog's 'Link Opportunities' report (available in paid version under Reports > Link Opportunities) cross-references your page content with anchor text of other pages to suggest internal link placements. This is particularly useful on sites with hundreds of blog posts where manual discovery of link opportunities would take days.

For sites on the free version, export all crawled page titles and URLs, then use a content search to identify pages that mention a topic but do not link to your target page on that topic. These are your link gap pages.

On-page SEO does not exist in isolation from technical SEO — and Screaming Frog is the junction point between the two. Three technical areas have direct, measurable impact on on-page SEO performance: redirect chains, canonical misconfigurations, and indexation issues.

Redirect Chains Go to Reports > Redirect Chains. Any URL that routes through more than one redirect before reaching its destination is a redirect chain. Google will follow redirect chains, but each hop dilutes the PageRank passed to the final destination and slows crawl response time.

More importantly, if a redirected URL has external backlinks pointing to it, those backlinks are losing their value through each additional hop.

For on-page SEO, the most impactful redirect chain fix is consolidating chains to single 301 redirects. This preserves more link equity on the final destination page and makes the value of any external links pointing to old URLs more effective.

Canonical Misconfigurations In the 'Canonicals' tab, look for three issues: pages with no canonical tag, pages with self-referencing canonicals (correct), and pages where the canonical points to a different URL. The third case is the most dangerous. If page A has a canonical pointing to page B, Google will consolidate ranking signals to page B — which means page A will not rank, regardless of how well-optimised its on-page content is.

A common misconfiguration: paginated pages (page-2, page-3 of a blog archive) canonical-ing back to the first page of the archive. This is usually intentional but sometimes applied to pages that should be independently indexed — such as paginated product category pages with unique products on each page.

Index Coverage via Response Codes Screening Frog's 'Response Codes' tab shows every URL and its HTTP response. Filter for non-200 responses. 404s on internally-linked pages mean you are wasting internal link equity on dead pages. 301s that cascade into other 301s create the chains discussed above. 5XX errors on pages with significant GSC impressions are a direct ranking emergency — these pages are periodically unavailable to Googlebot.

For every 404 found on a page with historical GSC impressions, either restore the page or redirect it to the most relevant live equivalent. Never let a page that once received clicks simply return a 404 without action.

The biggest missed opportunity with Screaming Frog is treating it as a one-time audit tool. Most teams run a crawl, fix the findings, and revisit the tool six months later when someone notices rankings have dropped. By then, you are doing damage control, not proactive optimisation.

Building Screaming Frog into a continuous monitoring cadence transforms it from an audit tool into an SEO health system. Here is how to operationalise this.

Scheduled Crawl Cadence Screening Frog allows you to schedule automated crawls and export reports automatically in the paid version. Set up a monthly crawl for small to medium sites (under 5,000 pages) and a bi-weekly crawl for large sites or high-velocity content operations. Save your configuration including API connections so every scheduled crawl pulls fresh GSC and GA4 data.

Crawl Comparison Reports Screening Frog's 'Crawl Comparison' feature (Reports > Crawl Comparison) allows you to load two crawl files and compare them. This shows you what has changed between crawls — new pages added, pages that changed status codes, pages where title tags changed, new redirects introduced. This is invaluable for catching regressions introduced by CMS updates, developer deployments, or content team changes that inadvertently broke on-page elements.

Issue Velocity Tracking Create a simple tracking sheet that records the count of key issue types after every crawl: missing title tags, broken inlinks, redirect chains, pages at depth five or deeper, orphaned pages. Plotting these numbers over time gives you an issue velocity metric — whether your on-page health is improving or degrading month-over-month. A rising orphan count, for example, indicates that content is being published without internal linking strategy — a process problem, not just an SEO problem.

The Quarterly CANOPY Review Every quarter, run the full CANOPY Framework (Content, Architecture, Navigation, Orphans, Performance, Your On-Page Signals) against your crawl data. The quarterly cadence aligns with most content publishing calendars and gives Google time to register changes from the previous quarter's fixes before you triage the next round.

The teams that see compounding SEO growth are the ones that treat on-page SEO as an operational system — not a project with a start and end date. Screaming Frog, run on a structured cadence with consistent configuration and benchmarked data, is the infrastructure for that system.