AEO Crawler FAQ: scoring, crawling and AI citations

AEO Crawler is an Answer Engine Optimization scoring platform that analyzes your web content and measures how well it is structured for AI-powered search engines like ChatGPT, Perplexity, Google SGE, and similar tools.

It crawls your site, scores each page across nine key dimensions, and delivers actionable recommendations to improve your visibility in AI-generated answers.

Answer Engine Optimization is the practice of structuring your content so that AI-powered answer engines can easily find, understand, cite, and surface it in response to user queries.

Unlike traditional SEO which optimizes for keyword rankings on a results page, AEO optimizes for being selected as the direct answer — making factors like clarity, structure, entity recognition, and citation potential critically important.

AEO Crawler fetches each page and evaluates it against nine weighted scoring dimensions:

• Direct Answer (20%)
• Citation Potential (18%)
• Structured Data (12%)
• Entity Clarity (12%)
• Query Coverage (10%)
• Semantic Coverage (10%)
• Conciseness (7%)
• Readability (6%)
• Content Freshness (5%)

Each dimension receives a score from 0–100. Scores are color-coded:

• 80+ — Excellent
• 60–79 — Good
• 40–59 — Fair
• Below 40 — Poor

Traditional SEO focuses on ranking in a list of blue links by optimizing keywords, backlinks, and technical page health.

AEO focuses on being the answer — the snippet, the cited source, the direct response in an AI-generated reply. AEO emphasizes:

• Clear questions and answers
• Structured data markup
• Authoritative entity signals
• Content conciseness

• Direct Answer (20%) — Does the page directly answer the question it implies? Content that leads with a clear, concise answer scores highest.
• Citation Potential (18%) — Is the content likely to be cited by an AI? Factual, authoritative, and specific content scores better.
• Structured Data (12%) — Does the page use Schema.org markup (FAQ, HowTo, Article, etc.)? Structured data gives AI engines explicit signals.
• Entity Clarity (12%) — Are the key entities (people, places, products, concepts) clearly defined and unambiguous?
• Query Coverage (10%) — Does the page address the full breadth of query intents — what, how, why, when, who, and comparison questions?
• Semantic Coverage (10%) — Does the content demonstrate genuine topic depth through vocabulary richness, technical terminology, and supporting evidence?
• Conciseness (7%) — Is the content appropriately focused without unnecessary filler?
• Readability (6%) — Is the language clear, well-structured with headings, and accessible?
• Content Freshness (5%) — How recently was the content updated? Pages updated within 2 months get 28% more AI citations. Detected from Schema.org dates, HTTP headers, meta tags, and in-text patterns.

• 80–100 — Excellent — Your content is well-optimized for AI answer engines.
• 60–79 — Good — Solid optimization with room for targeted improvements.
• 40–59 — Fair — Important gaps that are likely costing you AI visibility.
• Below 40 — Poor — Significant structural and content issues to address.

Most sites start between 40–60 and can reach 70+ with targeted changes.

The weights reflect the relative importance of each signal in how modern AI answer engines select and cite content.

Direct Answer is weighted highest (20%) because AI engines fundamentally need to find a clear answer. Citation Potential (18%) matters because AI systems prioritize authoritative, citable content.

The remaining seven dimensions each contribute at 5–12% based on their observed impact on AI citation rates. The newest — Content Freshness (5%) — was added because research shows pages updated within 2 months receive 28% more AI citations. Query Coverage and Semantic Coverage capture how AI engines evaluate topical breadth and depth.

In addition to the 9-dimension breakdown, every analysis returns 6 Unified AEO Metrics — a higher-level view of your AI search readiness:

• AI Visibility — Your overall composite score across all dimensions
• Citation Probability — How likely AI systems are to cite your content as a source
• Answer Extraction — How easily an AI can extract clear, direct answers from your page
• Entity Authority — The strength of your entity definitions and authority signals
• Query Coverage — The breadth of user queries your content addresses
• Semantic Coverage — The depth and richness of your topic coverage

These metrics are available on all tiers, including Free.

Multi-Platform Citation Tracking analyzes how your content performs across different AI search platforms, because each one cites content differently:

• ChatGPT — Favors encyclopedic, Wikipedia-style content with clear definitions and structured explanations
• Perplexity — Favors authentic, experience-based content with first-person expertise and community validation
• Google AI Overviews — Favors multimodal signals — YouTube references, descriptive image alt text, VideoObject schema, and strong E-E-A-T
• Claude — Favors well-structured arguments, high citation density, and nuanced analysis with evidence

Per-platform scores are generated by the CITEr AI agent and are available on Accelerator and Apex plans with AI analysis in full mode.

Citation verification queries live answer engines with questions your page should win, then records whether your URL was actually cited. It is a measurement of the outcome, not a prediction of it. Engines checked per plan:

• Explorer — No live citation checks
• Pathfinder — 1 engine — ChatGPT
• Accelerator — 3 engines — ChatGPT, Perplexity, Gemini
• Apex — 3 engines — ChatGPT, Perplexity, Gemini

Verification runs against the live engines at the moment you request it, so results move as the engines do. A page can score well on all nine dimensions and still not be cited yet — that gap is the point of measuring it.

We recommend re-crawling:

• After any significant content update
• After implementing recommended fixes
• At minimum once per month for active sites

AEO is an evolving field — as AI search engines update their algorithms, re-scoring helps you stay ahead. The crawl history in your dashboard makes it easy to track score changes over time.

The Score Forecast Engine is a predictive intelligence system that projects how your AEO score will change over time based on which recommendations you choose to implement.

It uses gap analysis combined with impact modeling to calculate projected improvements at three time horizons:

• 7 days — Quick wins
• 30 days — Medium-effort changes
• 90 days — Full implementation

Toggle recommendations on and off to see your projected score update in real-time.

The radar chart provides an instant visual fingerprint of your content's AEO performance across all nine scoring dimensions.

The filled polygon represents your current scores, while the dashed green overlay shows your projected scores after implementing selected recommendations.

This makes it immediately clear where your content excels and where the largest opportunities for improvement exist.

The branded PDF report generates a professional, consultant-grade A4 document you can share with clients, stakeholders, or your content team. It includes:

• Branded header with your score and letter grade (A+ through F)
• The radar chart visualization
• Complete breakdown table with all nine dimension scores, weights, and progress bars
• Prioritized recommendations with effort estimates and impact projections
• Forecast section showing your projected improvement trajectory

The report is generated entirely client-side — your data never leaves your browser during export.

Impact Points represent the estimated score improvement (in points) you can expect from implementing a specific recommendation.

They are calculated using gap analysis: for each dimension affected, we measure the gap between your current score and the realistic ceiling, then multiply by the recommendation's impact factor.

Example: If your Structured Data score is 20 and the ceiling is 95, the 75-point gap × 0.70 impact factor = +52.5 impact points.

• Quick Win — Change can be made in hours, results appear within a week. Examples: restructuring your opening paragraph, improving conciseness.
• ~30 days — Medium-effort changes like adding structured data markup or improving entity clarity, requiring more planning.
• ~90 days — Substantial content strategy work like expanding query coverage or deepening semantic coverage across your site.

The forecast timeline uses these effort levels to project realistic improvement curves.

• A+ (90–100) — Exceptionally well-optimized for AI answer engines
• A (80–89) — Strong optimization with minor refinements possible
• B (70–79) — Good, with clear opportunities to improve
• C (60–69) — Significant gaps in AI readiness
• D (50–59) — Content needs substantial work
• F (below 50) — Critical structural and content issues

Most sites start at C or D; reaching B or above puts you ahead of the vast majority of competing content.

When you submit a URL, AEO Crawler starts from that page, fetches the HTML, scores it, then follows internal links to discover and score additional pages — up to your plan's page limit.

The crawler:

• Respects your robots.txt by default
• Follows redirect chains
• Handles JavaScript-rendered pages where possible
• Uses a polite crawl delay to avoid overloading your server

The page limit depends on your plan:

• Explorer — Up to 10 pages per crawl
• Pathfinder — Up to 25 pages per crawl
• Accelerator — Up to 100 pages per crawl
• Apex — Up to 500 pages per crawl

You can also set a custom page limit below your plan maximum when starting a crawl.

• 50 pages — 1–3 minutes
• 500 pages — 5–15 minutes
• Large agency crawls — 20–40 minutes

You'll receive a notification when your crawl is complete, and results appear in your dashboard in real-time as pages are processed.

Yes. AEO Crawler can analyze any publicly accessible website — there is no restriction to your own domains.

Analyzing competitor content can reveal how well they are positioned for AI search and inform your own AEO strategy.

All crawling must comply with the target site's robots.txt and our Acceptable Use Policy.

Yes. The Free tier (Explorer) includes:

• 3 crawls per day
• Up to 10 pages per crawl
• Full AEO scoring across all nine dimensions
• Basic recommendations

No credit card required. Start crawling immediately after creating your account. API access is available on Apex.

You can upgrade at any time from Settings → Subscription.

Click "Upgrade Plan", select your desired plan, and you'll be redirected to our secure Stripe checkout. Upgrades take effect immediately after payment confirmation.

Your unused crawl balance from the current cycle does not carry over.

Yes. Cancel your subscription at any time from Settings → Subscription. There are no cancellation fees or long-term commitments.

After canceling, you retain access to your paid plan features until the end of your current billing period, then your account reverts to the Free tier. Your crawl history and data remain accessible.

All payments are processed securely through Stripe. We accept:

• Visa, Mastercard, American Express
• Local payment methods supported by Stripe (varies by region)

We do not store your card details — Stripe handles all payment data under PCI DSS compliance.

Yes. AEO Crawler respects the robots.txt file of every site it crawls. If a page is disallowed, it will be skipped and noted in your results.

We identify ourselves with a descriptive User-Agent string so site owners can identify our crawler in their server logs.

We store:

• The URL crawled
• AEO scores and sub-scores for each page
• Recommendations generated
• Metadata such as crawl timestamp and page title

We do not store the full HTML or body content of the pages we analyze. See our Privacy Policy for full details.

Yes — on Apex. The API is included with the Apex plan. With it you can:

• Trigger crawls programmatically
• Retrieve results
• Integrate AEO scores into your own tools and dashboards

Apex also includes Direct AI / MCP integration for Claude Code and Cursor. API keys are managed under Settings → API Keys. Rate limits apply.

Yes. To ensure fair use and service stability, API requests are rate-limited per account.

• Free tier — Up to 60 API requests per minute, 10 crawl jobs per month
• Paid plans — Higher API rate limits commensurate with crawl allowances

Crawl jobs themselves use polite per-domain delays to avoid overloading target servers.