A technical SEO checklist for AI crawlers starts with access: to be visible in AI answers, AI crawlers must be allowed to reach your pages, receive fast server-rendered HTML, and find clear signals about which URL is canonical. Decide which bots you allow in robots.txt, confirm your firewall isn't silently blocking them, and check server logs to prove they're actually visiting.
This checklist is written for business owners and the developers they work with. Work through each list, and pass anything you can't verify to whoever manages your hosting. For the strategy behind it, start with our pillar guide to AEO and GEO.
Which AI crawlers should you know about?
Each AI company runs separate user agents for training, search indexing and user-requested fetches, and you can treat them differently. A user agent is the name a bot announces when it requests a page; robots.txt rules target those names.
| User agent / token | Company | What it's for | Blocking it affects |
|---|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train models | Training use, not ChatGPT search results |
| OAI-SearchBot | OpenAI | Indexing pages for ChatGPT search | Whether you can appear as a ChatGPT search source |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to | Live visits on a user's request |
| ClaudeBot | Anthropic | Collecting content for model training | Training use |
| Claude-SearchBot / Claude-User | Anthropic | Search indexing and user-requested fetches | Appearing in Claude's search answers |
| Google-Extended | A robots.txt token, not a separate crawler: controls use of content for Gemini models | Gemini training and grounding, not Google Search or AI Overviews | |
| Googlebot | Google Search, including AI Overviews and AI Mode | Everything in Google Search | |
| Bingbot | Microsoft | Bing index, which also supports Microsoft Copilot | Bing and Copilot visibility |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers | Appearing as a Perplexity source |
| Applebot-Extended | Apple | Token controlling use of content for Apple's AI models | Apple's AI training use, not Siri or Spotlight search |
| CCBot | Common Crawl | Open web archive used by many AI researchers | Inclusion in Common Crawl datasets |
Bot names and purposes change. Before editing rules, check each company's official crawler documentation for the current list.
The key decision: blocking training bots is a content-licensing choice; blocking search bots is a visibility choice. Many businesses that want leads from AI search allow the search and user-fetch agents even if they block training.
Checklist 1: robots.txt
Your robots.txt file sits at yourdomain.com/robots.txt and tells well-behaved bots what they may crawl. It's a request, not a lock; reputable AI crawlers say they honour it, but it won't stop bad actors.
- robots.txt loads with a 200 status code (a 5xx error can make crawlers back off entirely).
- No leftover
Disallow: /from a staging site. - Explicit rules for each AI agent you've made a decision about.
- Your sitemap location is listed.
- Rules are tested after every deploy.
A pattern for a business that wants AI search visibility but no training use might look like this:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Disallow: /admin/ Sitemap: https://www.example.com/sitemap.xml
Adjust to your own policy. Many businesses simply allow everything, since being part of training data can help models recognise their brand.
Checklist 2: firewall, CDN and hosting
Robots.txt can say "allow" while your firewall says "block", and the firewall wins. Several CDNs and security services now offer one-click or default settings that block AI bots, and some hosts block unfamiliar crawlers to save bandwidth.
- Check bot-management settings in your CDN or firewall (look for "AI bots", "AI scrapers" or "bot fight" options).
- Confirm rate limits don't return 429 errors to legitimate crawlers during busy periods.
- Verify bots by the IP ranges or reverse DNS that each company publishes, not by user-agent string alone, which anyone can fake.
- Make sure rate limiting relies on the real client IP, not a header a visitor can set.
That last point comes from our own work. On grww.tech we found and fixed a login rate-limit bypass caused by trusting a spoofable IP header. The same mistake in bot rules lets scrapers dodge limits while genuine crawlers get blocked.
Checklist 3: rendering and HTML
Serve your important content in the initial HTML, because many AI crawlers fetch pages without running JavaScript the way a browser does. Googlebot renders JavaScript, but it's safer not to assume every AI fetcher will.
- View the page source (not the inspector) and confirm headings, body text, prices and FAQs are present.
- Use server-side rendering or static generation for marketing pages on frameworks like Next.js.
- Don't load FAQ answers or product details only after a click.
- Use real headings (H2, H3) in a logical order, plus tables and lists for structured information.
- Add descriptive alt text to images that carry meaning.
Checklist 4: speed and reliability
Crawlers have time and resource budgets; slow or heavy pages get fetched less often and less completely. Speed also matters for the humans who click through from an AI answer.
- Compress images to modern formats. On grww.tech, converting portfolio images to WebP cut their weight from 1.66 MB to 456 KB.
- Keep video files lean; our site's scroll videos are about 1.4 MB each.
- Load analytics after the page renders so tracking doesn't delay first paint.
- Watch server response times in your hosting dashboard.
- Return correct status codes: 200 for live pages, 301 for moved pages, 404 or 410 for removed ones.
Our explainer on Core Web Vitals covers the user-facing speed metrics in plain language.
Checklist 5: canonicals, indexing and duplication
Tell crawlers clearly which version of each page counts, and keep low-value pages out of the index. Mixed signals waste crawl effort and can surface the wrong URL in answers.
- Every page has a self-referencing canonical tag on your live domain.
- No canonicals pointing at old domains, staging sites or HTTP versions.
- Templated or thin pages are improved, merged or set to noindex.
- Your XML sitemap lists only canonical, indexable URLs with accurate last-modified dates.
We've had to fix both of these. Canonical tags on our site once pointed at an old domain, which told search engines to credit a different site. And we set 167 templated city pages to noindex to avoid scaled-content risk.
Checklist 6: snippet controls and structured data
Snippet controls let you limit how much text Google can show, and structured data labels what your content is. Use both deliberately.
- Understand that Google applies
nosnippetandmax-snippetto AI Overviews and AI Mode as well as classic results; overusing them can remove you from AI answers. - Add Organization, LocalBusiness, Product, Article or FAQPage markup that matches visible content.
- Validate structured data after each template change.
- Keep business name, address and phone consistent between markup and the page.
Checklist 7: llms.txt and machine-readable extras
llms.txt is an optional, proposed file that points language models to your most useful pages; it's low effort but unproven. No major AI company has confirmed using it for ranking or citations.
- Decide whether to publish one after reading our explainer on llms.txt.
- If you do, keep it short, accurate and updated when key pages change.
- Don't let it replace the fundamentals above.
Checklist 8: verify with logs and monitoring
Server logs are the only proof that AI crawlers actually visit. Analytics tools that rely on JavaScript usually don't record bots at all.
- Search raw access logs (or your CDN's bot analytics) for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Bingbot.
- Note which pages they request and which status codes they receive.
- Investigate any spike in 403, 429 or 5xx responses to bots.
- Re-check after every hosting, CDN or framework change.
- Pair crawl data with answer monitoring, as described in our guide to tracking your brand's visibility in AI answers.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
Not according to OpenAI's documentation, which separates GPTBot (training) from OAI-SearchBot (search). To appear as a source in ChatGPT search, allow OAI-SearchBot. Check OpenAI's current crawler documentation, as details can change.
Does blocking Google-Extended remove me from AI Overviews?
No. Google states that Google-Extended doesn't affect inclusion in Google Search. AI Overviews and AI Mode use Googlebot and are controlled through normal Search settings such as noindex and snippet controls.
Should I block AI crawlers to protect my content?
It depends on your business model. Publishers who sell content often block training bots. Most service businesses benefit from being visible and understood by AI systems, so they allow at least the search crawlers.
How do I know if my firewall is blocking AI bots?
Check your CDN or firewall bot settings and look for blocked requests from AI user agents in its logs. Then confirm in your server logs that those bots receive 200 responses on key pages.
Do AI crawlers respect robots.txt?
The major companies say their named crawlers do. Robots.txt is voluntary, though, so it won't stop scrapers that ignore it; firewall rules are the tool for enforcement.
Get a technical review
Most AI visibility problems we see are mundane: a blocked bot, a stray canonical, content hidden behind JavaScript. If you'd like a developer to work through this checklist on your site, see our services or request a fixed-price audit.