Can AI Crawlers Actually Read Your Website?

We found this on our own site last week, which is the only reason this article is specific rather than theoretical.

Our homepage has a services section. A list of six services on the left, and a panel on the right showing the details of whichever one you are pointing at. It looks good. It works fine. It had been live for months.

It was also putting exactly one service into the page's HTML. The other five existed as button labels and nothing else. Their descriptions, their feature lists, and their prices were generated in the browser when you moved your mouse. To anything that doesn't move a mouse, our homepage claimed we offered one service.

Why this happens

Most AI crawlers do not execute JavaScript. They fetch the HTML your server sends and take the text out of it. That is the whole process.

Googlebot is the exception, because it runs a full rendering pass, and that is why this kind of problem can sit there for years without anyone noticing. Rendering isn't free even for Google: in our own Search Console crawl stats, 53 percent of Googlebot's requests were for JavaScript files and only 12 percent for HTML. You rank fine. Search Console looks healthy. Meanwhile ChatGPT, Claude, and Perplexity are reading a version of your page with most of what you sell missing from it.

The patterns that cost you

Not every interactive element is a problem. The only question is whether the text exists in the HTML before any JavaScript runs.

Usually broken:

• Tabbed panels that swap content on click, which is the classic case and the one we had
• Carousels and sliders that render only the visible slide
• "Load more" and infinite scroll, where everything past the first batch is invisible
• Content fetched from an API after the page loads
• Modals and popovers whose contents are built when they open

Usually fine:

• <details> and <summary> accordions, where collapsed is a display state and the text is still present
• FAQ dropdowns built the same way
• Anything hidden with CSS alone, including display: none
• Scroll-triggered fade-ins, as long as the text is in the markup and only the opacity animates

That distinction is the useful part, because the intuition most people have is that hidden content is bad for SEO, and that idea is both out of date and pointed at the wrong thing. A collapsed FAQ is fine. A visible tab panel that generates its siblings on demand is not.

How to check your own page in a minute

Three ways, cheapest first.

View source, not inspect element. This trips up nearly everyone. "Inspect" shows you the live DOM after JavaScript has run. "View page source" shows what the server actually sent. They are different documents, and you want the second one, which is Ctrl+U in most browsers.

Once it is open, search it for a sentence that appears on your second tab, your third slide, or below the "load more" button. If it isn't there, a crawler that doesn't render isn't seeing it.

Disable JavaScript and reload. In Chrome DevTools press Ctrl+Shift+P, type "JavaScript", choose "Disable JavaScript", then reload. You are now looking at roughly what an AI crawler sees. Click through your main pages. Whatever survives is what you are actually publishing.

Fetch it the way a crawler does. From a terminal:

curl -s https://yoursite.com | wc -w

That is the word count of what a crawler receives. Compare it to what you think is on the page. On a content-heavy homepage the two numbers should be in the same range. If you get a few hundred words from a page that visibly has two thousand, you have found your problem.

What fixing it looks like

The rule is to render everything and hide what isn't active with CSS.

Rather than swapping which panel exists based on state, put all of them in the page and control visibility with opacity or display. The text ships in the HTML, the interaction still feels the same, and a crawler that never runs a line of JavaScript reads all of it.

Two things to get right while you are in there. Keep the inactive content out of the keyboard tab order, because five hidden panels that are still focusable means keyboard users tabbing through content they can't see. The inert attribute handles this by removing an element from focus and from the accessibility tree.

Then prefer inert to hidden or aria-hidden for this particular job. All three hide content from assistive technology, but text extractors commonly treat hidden and aria-hidden="true" as an instruction to discard the section, which would undo the whole fix. inert gives you the accessibility behaviour without the extraction penalty.

For us the change moved five services worth of copy into a homepage that had never contained it. Descriptions, feature lists, starting prices. Same design, same interaction, same animation.

Check the pages that matter, not all of them

You don't need to audit a hundred URLs. Check the handful that answer the questions you want to be cited for:

1. Your homepage, which is the page most likely to be fetched to answer "what does this company do"
2. Your main service or product pages
3. Any page with pricing on it
4. Your FAQ, if the answers aren't in <details> elements

If those four are clean you have covered most of your exposure.

The wider point

It is possible to have a fast, accessible, well-ranking, well-designed site that AI search cannot read. Those qualities are not the same thing, and the tools that measure the first four will not warn you about the fifth.

Lighthouse won't catch it, Search Console won't catch it, and your analytics won't catch it either, because traffic you never got doesn't show up anywhere. The only way to find it is to look at what your server actually sends, which takes a minute and which almost nobody does.

If you would rather have someone check it properly, it is part of what our AI SEO and analysis audits cover, and the performance work inside web design and development tends to surface the same issues from the other direction. Sites we build from scratch render their content on the server by default, which is the version of this problem you never have to fix.