Which onpage SEO tools you need depends on the problem: a checker grades one page, a crawler maps your whole site, and an AI agent writes and publishes content for you. Pick the category before the brand, because the right first tool for a small team is often one you can already use for free.
- Screaming Frog’s free version crawls 500 URLs; Ahrefs Free audits verified sites with 5,000 monthly crawl credits.
- A page checker like Yoast can’t spot orphan pages, redirect chains or duplicate titles, only a crawler maps those.
- Semrush and Ahrefs report problems, while Claire (from €199/month) writes and publishes content but still needs a crawler.
- AI crawlers skip JavaScript, so client-side content can stay invisible to ChatGPT despite Google rankings.
Checker, Crawler or Agent: Which Onpage SEO Tools Match Your Problem?
If your question is “is this one page okay?” a checker answers it. If it’s “what’s broken across my site?” you need a crawler. If it’s “who’s going to fix all of this?” you’re shopping for an agent.
| Category | What it does | What it can’t do |
|---|---|---|
| Page checker (Yoast, Rank Math, PageSpeed Insights) | Grades a single URL: title, meta description, headings, keyword use, load speed | Can’t see how your pages link to each other, so site-wide problems stay hidden |
| Site crawler (Screaming Frog, Sitebulb, audit modules in Semrush, Ahrefs, SE Ranking, Seobility) | Follows every internal link the way a search bot does and lists broken links, duplicates, redirects and indexability issues | Doesn’t fix anything, so you or a developer work through the list |
| AI content agent (Claire) | Researches keywords, writes and optimizes articles, publishes them to your CMS | Doesn’t crawl your site or run technical audits |
That’s also why a red score from a plugin tells you so little. It grades the page you’re looking at, with zero visibility into the rest of your site. Google Search Console sits next to all three as your free baseline. It shows indexing status, Core Web Vitals and search performance straight from Google, though only for your own verified site.
How Far Do Free Tools Get You, and Where Exactly Do They Stop?
For a small site, a free stack covers almost everything a small team needs. The limits are specific, though, and they hit at different points.
| Tool | Free up to | What’s missing |
|---|---|---|
| Google Search Console | No limit, your verified site only | No crawl of your internal link structure, so orphan pages don’t surface, and no competitor data |
| PageSpeed Insights | One URL per test | No site-wide view |
| Screaming Frog (free) | 500 URLs per crawl, no sign-up | JavaScript rendering, scheduled crawls, GA/GSC integration, custom extraction |
| Ahrefs Free | 5,000 crawl credits per project per month, unlimited verified sites, 170+ checks | Can’t audit sites you haven’t verified, so no competitor audits |
| Semrush (free account) | One project, 100 audited pages per month | Too small for a full 150 to 200-page company site |
| Yoast / Rank Math (free) | Checks inside the WordPress editor | Only sees the page you’re editing |
Ahrefs Free is the quiet hero here. For a blog with a few thousand pages, 5,000 credits usually cover one full audit a month, which is enough rhythm for most small teams.
Free stops being enough at three clear points. Your site outgrows the 500-URL Screaming Frog limit. Your content relies on JavaScript to render. Or you want crawls running on a schedule without anyone remembering to press start. Hit none of those and you can keep your money for now.
What Only a Crawler Catches: Orphan Pages, Redirect Chains and Duplicate Content
A crawler sees your site the way Googlebot does: as a web of links. That’s why it catches problems a single-page checker structurally can’t, because those problems live between pages, never on one.
Orphan page: A page that no other page on your site links to. Visitors and bots can only find it via the sitemap, a direct link or luck.
An orphan page costs you on four fronts: it receives no link equity, gets crawled less often, is indexed more slowly and often fails to rank even when the content itself is strong. In practice, that’s a well-written article doing almost nothing for you. The fix is usually simple: add internal links from related pages, merge or redirect the page if it duplicates another, or deliberately set it to noindex if it shouldn’t rank at all.
A redirect chain happens when page A redirects to B, which redirects to C. Each extra hop may reduce the SEO value passed through to the final page and slows loading for every visitor and bot. The fix is to flatten the chain into one direct redirect from A to C.
Duplicate titles and conflicting canonical tags are the third classic. They send Google mixed signals about which of your pages to show, so two of your pages compete for the same query and the one you’d rather rank can lose out. Broken internal links and pages accidentally set to noindex round out a typical crawl report.
So what does each issue cost in rankings? The cost scales with the page it hits. A money page accidentally set to noindex loses all of its Google traffic, because it can’t appear in results at all. An orphaned article keeps its quality but misses the link equity and crawl attention it needs to rank. A redirect chain leaks a little value per hop and adds load time on every visit.
Your own reference point is Search Console. Note the clicks and impressions of each affected page before you fix it, compare again four weeks later, and fix pages first that already earn clicks. That gives you a real number for your site instead of an industry average nobody has measured reliably.
Which crawler fits? Screaming Frog is the industry standard. Sitebulb starts at around $18 a month and explains issues as plain-English “hints” with visual crawl maps, which helps if redirect chains still feel abstract. If you already pay for Semrush, Ahrefs, SE Ranking or Seobility, their audit modules do the same job inside the suite.
Report, Suggest or Publish: Where Each Tool Stops Doing Work for You
Most on-page tools stop at the to-do list. The real split is how much of that list the tool works off itself. Here they fall into three groups:
- Report tools (Screaming Frog, Sitebulb, Semrush, Ahrefs, SE Ranking, Seobility, Search Console) only flag what’s wrong.
- Suggestion tools (Yoast, Rank Math) flag missing meta descriptions or weak headings while you edit.
- Agent tools like Claire’s Blog-Writer research keywords, write articles and publish to WordPress, Webflow or Shopify.
Be clear about where that third group stops. Claire isn’t a crawler or audit tool. It won’t find your orphan pages, fix server-side issues or clean up redirect chains. For those you still need one of the crawlers above, and someone with hosting access for the deeper fixes. If you want the bigger picture first, here’s how SEO now includes AI search.
What Does Each Tool Cost This Month, in the Currency You’ll Actually Pay?
If you’re budgeting from Austria, only some of these vendors bill in euro: Seobility and Claire publish euro prices, and SE Ranking shows EU visitors euro prices. Semrush, Ahrefs and Sitebulb bill in US dollars, Screaming Frog in pounds, with no separate Austrian price list, so your card statement shows a converted amount that moves with the exchange rate.
| Tool | Price (vendor currency) | Limits and terms |
|---|---|---|
| Screaming Frog | £199 per user per year (roughly €245, depending on the exchange rate) | Annual billing only, no monthly option |
| Sitebulb | About $18/month (Lite, 10,000 URLs per audit) to $42/month (Pro, 500,000 URLs), Cloud from about $125/month | 14-day full trial without credit card, no permanent free tier |
| Semrush | About $139.95 (Pro), $249.95 (Guru), $499.95 (Business) per month | Audits 100,000 to 1,000,000 pages/month, free account for testing |
| Ahrefs Free | Free | Verified sites only, 5,000 crawl credits per project/month |
| SE Ranking | €109/month (Core), €235/month (Growth) for EU visitors | Core: 250,000 audit pages/month, 2,000 tracked keywords, 100 AI-prompt checks a day |
| Seobility | Basic free, Premium €59.90/month (€47.92 billed annually), Professional €99.90/month | Monthly or annual billing |
| Claire | €199/month (Start), €299/month (Growth) | Up to 30 articles a month, no setup fee, cancel monthly |
Put that in perspective for a 200-page site. Semrush Pro costs about $140 a month for audit capacity you’ll never use, while Screaming Frog works out to under £17 a month. The suites earn their price through keyword tracking and competitor research, so pay for them only if you’ll use those parts.
Budget note: Seobility’s prices are net of VAT, and Screaming Frog charges per user. If two people need to crawl, budget two licences.
Need client-ready dashboards on top of your audits? That’s a separate tool category, covered in this comparison of SEO reporting software.
Is Your Site Even Readable by ChatGPT and Perplexity?
Maybe not, and your usual audit probably won’t tell you. Two things decide it. Whether your robots.txt lets AI crawlers in. Whether your main text is in the raw HTML your server sends.
The second point catches a lot of modern sites. AI crawlers fetch only the HTML your server sends and don’t run scripts. Googlebot renders JavaScript in a second wave, so a site built on client-side rendering (like a React app without server-side rendering) can rank on Google while looking almost empty to ChatGPT. Independent tests from February 2026 estimate that 42% of JS-rendered content never gets indexed by AI systems.
You can check this yourself in about ten minutes:
- Open yourdomain.com/robots.txt and look for Disallow rules under GPTBot, ClaudeBot, PerplexityBot, Google-Extended or “User-agent: *”.
- Run curl -A “GPTBot/1.0” https://yourdomain.com/your-page and check whether your main text appears.
- Check whether yourdomain.com/llms.txt exists.
If step two shows only an empty div instead of your text, that page depends on client-side rendering and needs a developer. If step three returns a 404, you simply haven’t created the file yet, and that’s quick to change.
What llms.txt is: A plain text file in your site’s root folder that summarizes what your site covers and points to its key pages. It’s a low-effort first step before you tackle bigger rendering changes.
The classic audits in Yoast, Semrush and Ahrefs weren’t built around this check. Small dedicated tools fill the gap: this AI crawler robots.txt access checker tests your rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, and looks for llms.txt or llms-full.txt. For the rendering side, the JavaScript rendering in Screaming Frog’s paid licence and on every Sitebulb plan lets you compare what a browser sees with the raw HTML.
Want a wider check? Claire’s free GEO audit tests your site against 70+ points that decide whether ChatGPT and other AI systems find, understand and cite it. For how this connects to classic rankings, see how Google ranks pages in the AI era.
Your Pick by Team Situation, and the First Week With It
Match the stack to your hours and your access to a developer. Your page count matters less.
| If you are… | Pick | Budget line | First week |
|---|---|---|---|
| A solo marketer with a 50-page site, no dev | Search Console + Ahrefs Free + Screaming Frog free | €0 | Day 1: GSC and PageSpeed Insights. Day 2: verify Ahrefs, run Site Audit. Day 3: sort the top 10 issues by impact. Day 4: fix titles and meta descriptions. Day 5: robots.txt check for AI crawlers. |
| A two-person team with a growing blog | Screaming Frog licence + Claire Start | £199/year + €199/month | Run a full crawl, fix orphan pages with internal links, set Claire to copilot mode so you approve every article. |
| An agency-supported company with a developer | Sitebulb Cloud for visual stakeholder reports, dev on technical fixes | From about $125/month | Crawl, share the report, hand redirect chains and canonical conflicts to the dev with a deadline. |
Whichever row you’re in, the split of work stays the same. Titles, descriptions, headings, links and alt text are yours to fix in the CMS. Redirect chains, server configuration and JavaScript rendering changes need a developer or hosting access.
The costly mistakes are avoidable too. Size the tool to the site: a 40-page site doesn’t need Semrush Business at $499.95 a month when Ahrefs Free covers it. Work the top issues by impact and let low-priority warnings wait. Keep AI tools in approval mode until you trust their output.
The Real Gap Is the Follow-Through
Most teams have enough tools already. What’s missing is someone working through what those tools find, and anyone checking whether AI crawlers can see the site at all. So before you buy anything new, run one crawl and one GPTBot fetch this week. Fix the top five issues you find, then decide whether the bottleneck is finding problems (buy a crawler) or fixing them (bring in an agent).
FAQ
Is Semrush or Ahrefs more accurate for a technical site audit?
Neither wins clearly on accuracy, since both run over 140 technical checks. For a small site the bigger difference is access: Ahrefs Free audits verified sites with 5,000 crawl credits a month, while Semrush’s free account stops at 100 pages.
Should I block AI crawlers in my robots.txt?
Rarely, if AI visibility is the goal. Blocking GPTBot, ClaudeBot or PerplexityBot stops them from fetching your pages, so ChatGPT, Claude and Perplexity can’t draw on that content. Block only sections you deliberately want kept out.
Should I use Screaming Frog or Sitebulb if I’m not technical?
Sitebulb, in most cases. It explains each issue as a plain-English hint with visual crawl maps and includes JavaScript rendering on every plan. Its 14-day trial needs no credit card, so you can test it on your own site before paying from about $18 a month.
Does fixing every warning a tool flags actually improve rankings?
No. Many warnings are cosmetic, and working through hundreds of them eats the time you need for the issues that matter. Start with indexability, orphan pages, redirect chains and duplicate titles, then move down the list.
How often should I re-run a full crawl once my site is fixed?
Every four to six weeks for a small site of around 50 pages. A growing blog benefits from a crawl after each larger batch of new content, and after a redesign or migration you crawl immediately after launch.
How do I show my boss progress without using the tool’s own score?
Use numbers from outside the tool: organic clicks in Search Console before and after each batch of fixes, the count of resolved 4xx and 5xx errors, and how quickly Google reindexes corrected pages. Those hold up in a meeting.
