A Complete Guide to the AEOTrack Crawlability Page: Manage AI Crawler Access

Introduction

Before an AI model can mention your brand or cite your website, its underlying web crawler must first be permitted to visit your site. If your robots.txt file accidentally blocks AI bots like GPTBot, ClaudeBot, or PerplexityBot, AI search engines won't be able to index or cite your content — regardless of how well-optimized your pages are.

The AEOTrack Crawlability page (found under Your site → Crawlability in the sidebar) scans your website's live robots.txt file and audits permission rules for every major AI search crawler and training bot.

This guide walks through every metric, filter, and table on the Crawlability page, using a fictional example company — Acme Analytics.

---

What Is the AEOTrack Crawlability Page?

The Crawlability page provides a real-time audit of which AI crawlers your robots.txt file permits, restricts, or blocks. It separates what your rules permit from which crawlers have actually visited your site.

It helps you answer critical questions:

• Are any major AI search crawlers currently blocked by my robots.txt file?
• Which crawlers have explicit rules written for them versus inheriting global User-agent: * rules?
• What is the specific purpose of each crawler (search indexing vs. model training)?
• Which AI platform owns each crawler (OpenAI, Anthropic, Google, Perplexity, Cohere, etc.)?

---

Summary Metrics & Actions

At the top of the page, four summary cards present an at-a-glance count of your crawler permissions, alongside action controls.

Summary Cards

• Blocked: The number of AI crawlers explicitly forbidden from accessing your site (e.g., 0).
• Partial: The number of crawlers permitted under partial or conditional rules (e.g., 28).
• Open: The number of crawlers granted completely unrestricted access (e.g., 0).
• Crawlers Checked: The total number of AI crawlers evaluated in the audit (e.g., 28).

Action Controls

• View robots.txt: Opens a view of your site's current live robots.txt file.
• Re-read robots.txt: Triggers a fresh live fetch of your robots.txt file to update status after making server or file changes.

---

The Crawlers Audit Table

The main section of the page lists every checked AI crawler, categorized by rule type.

Table Sections

1. Named in your robots.txt: Crawlers that have explicit User-agent directives specified in your file (e.g., GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, cohere-ai).
2. Following your global rules: Crawlers that have no specific rule named, so they inherit whatever directives apply to all user-agents (User-agent: *).

Table Columns

• Crawler: The official name of the bot (e.g., GPTBot, ClaudeBot).
• Purpose: Indicates whether the bot is used for live search retrieval or model training.
• Platform: The company or AI model operating the bot (e.g., ChatGPT · OpenAI, Claude · Anthropic, Gemini · Google, Perplexity, Cohere).
• Status: The permission status badge (Open, Partial, or Blocked).

Example Table Data (fictional):

The AEOTrack Crawlability audit page, showing summary status counts and crawler permission tables.

---

How to Read the Crawlability Page: Step-by-Step

1. Check your Blocked count — ensure critical search crawlers (like OAI-SearchBot or PerplexityBot) are not blocked if you want AI engines to cite your site.
2. Filter by status — use the status pills (Blocked, Partial, Open) to quickly isolate restricted crawlers.
3. Distinguish Search vs. Training bots — determine whether you want to allow live search indexing bots while restricting model training bots.
4. Use "Re-read robots.txt" — click re-read after updating your server configuration to confirm your changes took effect immediately.

---

Frequently Asked Questions

What's the difference between a search crawler and a training crawler?
Search crawlers (like OAI-SearchBot or PerplexityBot) fetch live web content to synthesize real-time answers and citations. Training crawlers (like GPTBot or ClaudeBot) collect data to train future AI model versions.

What does "Partial" status mean?
It means the crawler is permitted to access some sections of your site, but specific subdirectories or file paths are disallow-restricted in your robots.txt.

Why does AEOTrack separate robots.txt rules from actual crawler visits?
Crawlability shows what your rules allow, while server log plugins (like the AEOTrack WordPress plugin) show which bots actually visited.

---

Conclusion

The Crawlability page ensures that your technical infrastructure isn't silently blocking AI search engines from finding and citing your brand. By regularly auditing crawler permissions, you maintain complete control over how AI models access your site.