A Complete Guide to AEOTrack AI Crawler Setup: Server Log Ingestion & Bot Detection

Introduction

Knowing whether your robots.txt permits AI crawlers is only half the battle. The other half is knowing whether AI bots — like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot — are actually visiting your website in real life.

Standard web analytics tools (such as Google Analytics or frontend JavaScript pixels) cannot detect AI crawlers. Why? Because AI crawlers request your HTML file directly and leave without executing JavaScript.

The AEOTrack AI Crawler Setup page (found under Your site → AI crawler setup in the sidebar) solves this by enabling server-side log ingestion (via Vercel, Cloudflare, or WordPress), letting you capture raw HTTP requests from AI bots before pages are served.

This guide walks through every diagnostic step, integration method, and configuration step on the AI Crawler Setup page.

---

Why JavaScript Tracking Snippets Don't Detect Bots

Standard web analytics rely on JavaScript snippets executing inside a browser. AI crawlers behave fundamentally differently:

• No JavaScript execution: When GPTBot, ClaudeBot, or PerplexityBot accesses a URL on your site, it downloads the raw HTML markup, extracts textual content, and immediately exits.
• Bypassed client scripts: Because AI bots never render JavaScript or run <script> tags, client-side tracking pixels are never fired.
• The server log solution: Server-side log ingestion captures every incoming HTTP request at the CDN or web server layer before the response is sent to the crawler.

Why JavaScript analytics snippets fail to track AI bots, making server log ingestion necessary.

---

Diagnosing "Needs Attention" & No Bot Visits

When you first open the AI Crawler Setup page, AEOTrack checks whether any AI bot visits have been logged for your account.

"Needs Attention: No AI crawler has ever reached your site"

If no visits are recorded, AEOTrack flags a diagnostic notice:

Engines can only cite pages they have fetched. Nothing has been recorded here, which either means the tracking snippet/log drain is not installed yet, or the crawlers are genuinely being turned away. Until one of those is resolved, everything else is downstream of it.

Recommended Action Steps

1. Confirm bot tracking is installed: Rule out missing log drains or plugins first. Missing log drains and blocked crawlers produce identical zero-visit reports, but require opposite solutions.
2. Allow AI crawlers in robots.txt: Check the Crawlability page to confirm that GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot are not disallowed. A blanket Disallow: / blocks AI engines along with traditional search engines.

---

Server Log Ingestion Setup

To capture real AI bot visits across your domain, connect your web host or CDN access logs to AEOTrack using your account's unique Endpoint URL and Secret key.

Unique Credentials

• ENDPOINT URL: https://api.aeotrack.io/bot-detection/logs/dom_pj9d1k1dc30redcxxoku (Example endpoint for Acme Analytics)
• SECRET: •••••••••••••••••••••••••••••••• (A signed secret token used by your server to authenticate log payloads)

The AEOTrack AI Crawler Setup page, featuring diagnostic alerts, log ingestion endpoint details, and Vercel setup instructions.

---

Integration Guides: Vercel, Cloudflare, & WordPress

AEOTrack provides native integration paths for popular web hosts and CDNs:

1. Vercel Log Drains

1. Navigate to your Vercel project dashboard → Settings → Log Drains.
2. Click Add Drain, select the JSON (NDJSON) format, and paste your AEOTrack ENDPOINT URL.
3. Set the Custom Secret field to your AEOTrack SECRET value (Vercel signs each log delivery with this key).
4. Click Save. Vercel immediately sends a verification request, which AEOTrack answers automatically.

2. Cloudflare Worker / Logpush

1. Enable Cloudflare Logpush or deploy the AEOTrack Cloudflare Worker snippet.
2. Forward HTTP request headers (including User-Agent and request URI) to your AEOTrack Endpoint URL.

3. WordPress Plugin

For WordPress sites, AEOTrack offers a lightweight plugin that hooks into early server execution:

WordPress plugin setup card for recording real AI crawler visits.

1. Install and activate the AEOTrack Bot Detection plugin in your WordPress admin dashboard.
2. Enter your account API key.
3. Once activated, the plugin records incoming HTTP requests from recognized AI crawlers and syncs them directly to your dashboard.

---

How to Verify Your Setup: Step-by-Step

1. Copy your Endpoint URL and Secret from the Server Log Ingestion card.
2. Configure your CDN log drain (Vercel, Cloudflare, or WordPress plugin).
3. Trigger a verification check — save your log drain configuration to allow AEOTrack's automated handler to respond.
4. Monitor recorded bot visits — check back after scheduled AI checks to view real-time crawler logs and hit counts.

---

Frequently Asked Questions

Why can't I just use Google Analytics to track AI bots?
Google Analytics relies on a JavaScript tag running in a browser. AI crawlers do not execute JavaScript, so Google Analytics never sees them.

Will log ingestion slow down my website performance?
No. Log drains deliver HTTP access logs asynchronously in the background from your CDN or server layer — your site's response time to visitors and crawlers is completely unaffected.

What happens if I block GPTBot in robots.txt?
If GPTBot is blocked in robots.txt, OpenAI's crawler will respect your rules and avoid visiting your site, resulting in zero logged visits and reduced citation opportunities in ChatGPT.

---

Conclusion

Setting up server log ingestion bridges the gap between what your robots.txt rules allow and what AI crawlers actually do. By capturing raw server logs via Vercel, Cloudflare, or WordPress, you gain complete visibility into AI search engine activity across your domain.