Can AI crawlers see your website? The JavaScript problem

Open your site, turn off JavaScript, and reload. Whatever is left is roughly what many AI crawlers read.

What the data shows

In December 2024 Vercel and MERJ published an analysis of AI crawler traffic across Vercel's network. Their key finding: none of the major AI crawlers they measured rendered JavaScript. That covered OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta, ByteDance and PerplexityBot. The exceptions were Gemini, which uses Googlebot's rendering, and Applebot, which renders pages in a browser.

ChatGPT's and Claude's crawlers did download JavaScript files (11.5% and 23.8% of their requests) but didn't execute them. The study also found ChatGPT's crawler spent 34.82% of its fetches on 404 pages, a reminder to keep redirects and sitemaps tidy.

That study is a snapshot from late 2024. Crawlers change, but it's the best public measurement we know of, and server-rendered content is the safe assumption.

Who's at risk

• Single-page apps (React, Vue, Angular) that render everything in the browser
• Site builders or themes that load text through scripts after the page arrives
• Menus, prices or reviews pulled in by third-party widgets
• Content hidden behind "click to expand" that's only fetched on click

How to check

1. Run the SEO Health Snapshot. Its "Text in the HTML" check counts the words in the raw HTML, before any JavaScript runs.
2. In Chrome, view the page source (not Inspect) and search for a sentence from your page. If it's not there, crawlers that don't render won't see it.

How to fix it

• Use server-side rendering or static generation for important pages. Most modern frameworks support it (Next.js, Nuxt, SvelteKit, Astro).
• Keep the title, meta description, main text, prices and contact details in the initial HTML.
• Use JavaScript for enhancements (chat widgets, counters, animations), not core content.

Sources

• Vercel and MERJ: The rise of the AI crawler (Dec 2024)