Technical SEO Scan and Site Crawl
Find technical issues. Know what to fix first.
Run a crawl and technical scan that checks meta tags, robots.txt, sitemaps, structured data, and Core Web Vitals. Findings are categorized by severity so teams know where to focus. Results roll into client reporting with clear summaries and next steps.
6
Check Categories
5-10 min
Scan Time
Prioritized
Issue Severity
“The technical scan found issues our previous tools missed. The severity ratings helped us prioritize fixes that actually moved the needle.”
— Tom H., Technical SEO Lead
What the Technical Scan Checks
The scan runs a series of modules that cover crawl health, performance, and on-page technical fundamentals. Here's what's included.
Crawl and Indexability Checks
The crawl starts at your specified URL and discovers linked pages across the site.
What it checks:
- URL discovery and crawl coverage
- HTTP status codes (redirects, 4xx errors, 5xx errors)
- Indexability blockers (noindex tags, canonical issues, blocked by robots)
- Internal linking structure and orphan page detection
What you get:
- A list of crawled URLs with status
- Categorized crawl issues grouped by type and severity
Core Web Vitals and Performance
Performance metrics are pulled via PageSpeed Insights for the URLs you specify.
What it checks:
- LCP (Largest Contentful Paint): How fast the main content loads
- INP (Interaction to Next Paint): How responsive the page is to user input
- CLS (Cumulative Layout Shift): How stable the layout is during loading
What you get:
- CWV scores per URL with pass/fail indicators
- Performance insights and recommendations
Important: CWV data is refreshed on demand. Lab data (simulated tests) can differ from field data (real user metrics), and field data reflects a rolling window of actual visits. Treat scores as directional indicators, not absolute measurements.
Meta Tags and On-Page Technical Basics
The scan analyzes meta elements that affect how pages appear in search results and how search engines understand content.
What it checks:
- Title tags (presence, length, duplicates)
- Meta descriptions (presence, length, duplicates)
- Canonical tags (presence, self-referencing, conflicts)
- Heading structure (H1 presence and hierarchy)
What you get:
- Page-level findings for meta tag issues
- Recommendations for missing or problematic elements
robots.txt Checks
The scan fetches and analyzes your robots.txt file to identify rules that might block crawling.
What it checks:
- Disallow rules and their scope
- Accidental blocks on important sections
- Crawl-delay directives
- Sitemap references in robots.txt
What you get:
- A breakdown of robots.txt rules and their impact
- Warnings for potentially problematic directives
Sitemap Checks
The scan locates and validates your XML sitemap to ensure search engines can discover your pages efficiently.
What it checks:
- Sitemap presence and accessibility
- URLs included in the sitemap
- Broken or redirecting URLs in the sitemap
- Non-indexable URLs included in the sitemap
What you get:
- Sitemap health summary
- List of problematic URLs that shouldn't be in the sitemap
Structured Data Checks
The scan detects and validates structured data (schema markup) on your pages.
What it checks:
- Schema types present (Article, Product, LocalBusiness, FAQPage, etc.)
- JSON-LD syntax validity
- Required and recommended properties
- Common implementation errors
What you get:
- Schema detection results per page
- Validation warnings and errors for each schema type
How Findings Are Categorized
Not every issue needs immediate attention. Findings are categorized by severity and type so teams can prioritize effectively.
By Severity
Prevents crawling, indexing, or basic page access
Site-wide noindex, homepage returning 5xx, robots.txt blocking entire site
Materially harms performance or prevents key pages from competing
Missing title tags, Core Web Vitals failures, broken canonical chains
Reduces efficiency or weakens quality signals
Duplicate meta descriptions, missing H1s, minor CLS issues
Best-practice improvements with limited immediate impact
Meta description length, schema enhancements, minor redirect chains
By Type
HTTP errors, redirects, noindex tags, orphan pages
LCP, INP, CLS scores and thresholds
Titles, descriptions, canonicals, heading structure
Disallow rules, accidental blocks, crawl directives
Missing sitemap, broken URLs, non-indexable URLs
Schema validation, missing properties, syntax errors
Using the Categories
Start with Critical and High severity issues. These are the blockers that prevent pages from being crawled, indexed, or performing well. Once those are resolved, move to Medium and Low for incremental improvements.
How Technical Findings Appear in Reports
Technical scan results don't stay in a silo. They feed into client-ready reports so stakeholders see what's happening and what needs attention.
Run the Crawl and Technical Scan
Initiate the scan for a client. The system crawls the site, runs technical checks, and categorizes findings.
Findings Are Summarized
Issues are grouped by severity and type, then summarized into actionable priorities. The scan generates recommendations based on what was found—not generic advice, but specific next steps tied to the actual issues detected.
Reports Present Technical Highlights
When you generate a report, technical findings appear alongside traffic, keyword, and competitor data.
What Clients See in Reports
What This Delivers
Clients understand the technical state of their site without wading through raw data. They see what changed, what's broken, and what to do next—in the same report that covers their SEO performance.
Limitations and What Can Affect Results
Technical scans depend on site accessibility and external APIs. Here's what can affect results.
Blocked Crawling
If the site blocks the crawler (via firewall, bot protection, server rules, or robots.txt restrictions), the scan will be incomplete or unable to start. Some sites require whitelisting specific crawlers or adjusting security settings.
Rate Limits and API Quotas
PageSpeed Insights and other external providers can limit requests. During heavy usage periods, scans may be queued or return partial results. If CWV data is missing, try again later or reduce the number of URLs being checked.
Site Complexity
Very large sites (thousands of pages) may require scoped crawling. Starting with a specific section or priority pages produces faster, more focused results than attempting a full-site crawl.
Dynamic Rendering
Sites that rely heavily on JavaScript rendering may produce incomplete extraction depending on how content is loaded. If key content isn't visible to the crawler, it won't appear in the findings. Server-side rendering or pre-rendering improves crawl coverage.
CWV Variability
Core Web Vitals scores are not static. Lab data (simulated tests) can differ from field data (real user metrics). Field data reflects a rolling window and depends on actual traffic. Treat CWV results as indicators of performance, not definitive scores.
Troubleshooting Crawl and Scan Issues
When scans don't complete or results look wrong, here's how to diagnose and fix.
Crawl Will Not Start or Returns No URLs
Likely cause:
Crawler is blocked by firewall or bot protection, URL is incorrect, DNS or SSL issue prevents access.
Fix:
Confirm the site loads publicly in a browser. Check if bot protection is blocking automated access. Whitelist the crawler if needed. Verify the URL is correct and uses the right protocol (http vs https). Retry the scan.
Crawl Issues Do Not Appear or Look Incomplete
Likely cause:
Crawl is still in progress, site blocks parts of the crawl, scope is too narrow.
Fix:
Wait for the crawl to complete—large sites take longer. If parts of the site are blocked, check robots.txt or server access rules. Retry with a broader starting URL or reduce scope to a specific section.
Core Web Vitals Fails or Is Blank
Likely cause:
API quota limits reached, URL is invalid or not publicly accessible, temporary provider issues.
Fix:
Rerun the CWV check later. Test a single URL first to confirm it works. Reduce the number of URLs being checked if quota limits are a concern.
robots.txt Findings Seem Wrong
Likely cause:
Multiple robots.txt files at different locations, cached version differs from live file.
Fix:
Fetch robots.txt directly in your browser (domain.com/robots.txt) to see the current version. Clear caches if you recently made changes. Rerun the scan.
Structured Data Validation Errors
Likely cause:
Invalid JSON-LD syntax, missing required properties for the schema type, incorrect nesting.
Fix:
Review the specific errors reported. Adjust the schema markup to fix syntax issues or add missing properties. Rerun validation. Consider testing in Google's Rich Results Test for additional context.
Technical SEO Scan FAQs
Common questions about the technical scan and site crawl.
Still have questions? The best way to understand the technical scan is to run one.
Run My Technical ScanRun a Technical Scan and Get Clear Priorities
Find what's broken, understand why it matters, and know what to fix first. Technical findings roll into client reports so everyone stays aligned.
Run My Technical ScanResults in 5-10 minutes • Prioritized by severity