Allow AEOTrackBot to read your site

Why you see "we could not read this page"

AEOTrack reads your pages the way an AI engine does: it fetches the URL, parses the HTML and grades what it finds. Some hosts and firewalls (Cloudflare, Akamai, Sucuri, Wordfence and similar) answer that fetch with a challenge page or a block instead of the page. The product retries with a browser-style request and then with a reader service; if all three fail, the Site Analyzer and the Content Optimizer show "We could not read this page" with the reason (blocked, bot challenge, or timeout).

This is a setting on your side, not a fault in the page. Nothing about your AI visibility score depends on it: the engines fetch your site with their own crawlers.

Option 1: allow the crawler (recommended)

Every request AEOTrack makes to your site identifies itself with this user agent:

AEOTrackBot/1.0

Add a rule that allows that user agent. Where to do it:

• Cloudflare: Security → WAF → Custom rules → Create rule. Expression: (http.user_agent contains "AEOTrackBot"), action Skip (all remaining custom rules, rate limiting and bot fight mode). If Bot Fight Mode or Super Bot Fight Mode is on, the skip rule is what lets the request through.
• Wordfence (WordPress): Firewall → Blocking → Allowlisted user agents, or Tools → Live Traffic to find the blocked request and click "Whitelist".
• Sucuri: Firewall → Access Control → Allow User Agents.
• Akamai / Imperva / AWS WAF: add an allow rule matching the User-Agent header containing AEOTrackBot.
• robots.txt: not needed. AEOTrackBot only fetches the pages you ask it to analyse; it does not crawl your site.

After the rule is in place, run the analysis again. There is nothing to reset on our side.

Option 2: paste the page

If you cannot change the firewall, the same screen offers Paste the page text or HTML. Paste either the rendered text of the page or its HTML source (up to 200 KB) and press Analyse. The analysis then grades what you pasted instead of fetching. Pasted text is graded on the content checks (length, headings, structure); the HTML checks (schema, meta tags, links) need the HTML source, which you can copy from your browser with View Page Source.

The Content Optimizer accepts a pasted page in the same way and rewrites from it.

What the crawler fetches

• Only the URL you submit (the homepage during onboarding, the page you pick in the Site Analyzer or Content Optimizer). It follows redirects on that URL and nothing else.
• One request per analysis, plus at most two retries. It does not download images, scripts or stylesheets.
• It never submits forms or signs in.

If a page still fails after the rule is added, reply to any AEOTrack email with the URL and the message you saw and we will look at the request log with you.