How AI Sees Your Page: Content the Crawler Can't Find

A product page in an online store: the name and price above the photo, three tabs below it – "Description", "Specifications", "Delivery" – and further down a reviews section that loads when the user scrolls to it. The description runs to two thousand words, the specs list thirty parameters, there are two hundred reviews. Everything needed to answer the question "is this lamp suitable for a small office?".

Someone asks exactly that question to an AI assistant with web search. The answer cites your page – and is based solely on two sentences from the meta description and the price. Not a word about dimensions, light output or reviews, even though they're on the page.

The first suspicion usually falls on CSS: "the tabs have display: none, so the bot doesn't see them". That's almost never the cause. To understand where the content really gets lost, you need to know what a crawler actually does with a page.

A crawler doesn't look at the page, it reads HTML

A browser downloads HTML, then CSS and JavaScript, builds the DOM tree, runs scripts, computes layout and paints pixels. The user sees the result of that whole process. A crawler usually does only the first step: it downloads the HTML and extracts text from it.

Googlebot is the exception – for years it has rendered pages in a browser, JavaScript included, albeit with a delay. But the crawlers feeding AI assistants and their search mostly don't. Analyses of traffic from such bots, published by Vercel among others, showed that they download JavaScript files but don't execute them. In practice that means one simple rule on which everything else depends: an AI crawler sees what's in the HTML sent by the server – and nothing else.

Two things follow from that rule at once. Content hidden with styles but present in the HTML is as visible to the crawler as anything else. And content that only comes into existence in the browser – after a click, after a scroll, after a script runs – doesn't exist for the crawler.

The pitfall: display: none isn't the problem

A crawler that doesn't compute layout doesn't even know that some CSS rule hides an element. To it, a tab with display: none, hidden or visibility: hidden is just another piece of text in the HTML. Search engines may weight such content differently, but they see it.

The real problem lies one step earlier – in whether the hidden content made it into the HTML at all. And here what decides isn't CSS, but the way the component hides it. Compare two implementations of the same tabs in React:

tsx

// "Conditional" tabs: only the active panel is in the HTML
function ProductTabs({ tabs }: { tabs: Tab[] }) {
const [active, setActive] = useState(tabs[0].id)
const current = tabs.find((t) => t.id === active)!

return (
  <div>
    <TabList tabs={tabs} active={active} onChange={setActive} />
    <div role="tabpanel">{current.content}</div>
  </div>
)
}

It looks innocent and works fine for the user. But the server renders only the first tab. The specs and delivery information are only created in the browser after a click – and a crawler never clicks. To it, the product page has a description and nothing more.

tsx

// Tabs hidden with an attribute: all panels are in the HTML
function ProductTabs({ tabs }: { tabs: Tab[] }) {
const [active, setActive] = useState(tabs[0].id)

return (
  <div>
    <TabList tabs={tabs} active={active} onChange={setActive} />
    {tabs.map((t) => (
      <div
        key={t.id}
        role="tabpanel"
        id={"panel-" + t.id}
        aria-labelledby={"tab-" + t.id}
        hidden={t.id !== active}
      >
        {t.content}
      </div>
    ))}
  </div>
)
}

The same tabs for the user, the same state, the same click handling. There's only one difference: all panels are rendered, and the inactive ones are hidden by the hidden attribute. The server sends the whole description, the whole spec table and the delivery information in the HTML. The crawler reads it all.

The same applies to FAQ accordions, "More information" sections and every component that shows content on demand. The rule is simple: hide with a style or attribute, not a condition. {open && <Panel />} removes the content from the HTML, hidden={!open} only hides it.

hidden="until-found": hidden, but findable

There's a variant that goes a step further. In the content-visibility article I described hidden="until-found" – an attribute that hides an element like hidden, but lets the browser reveal it when the user searches its text with Ctrl+F or lands on it via a text-fragment link.

html

<div role="tabpanel" id="panel-spec" hidden="until-found">
<table>…</table>
</div>

To a crawler it's exactly the same as plain hidden – the text is in the HTML. To the user, it solves a problem tabs have always had: in-browser search doesn't find the word "weight" because the spec table is in a tab nobody opened. With until-found, the browser reveals the panel itself and fires a beforematch event on the element, where you can switch the active tab so the interface stays consistent. A browser that doesn't know this value treats it as regular hidden – so you lose nothing.

The native <details> element gives you the same effect for free: the contents of a closed element are in the HTML, and newer browsers reveal them during search.

Text that isn't in the HTML: CSS content and images

The second category is content that's visible on the page but isn't text in the document at all. The sneakiest case is content in pseudo-elements:

css

.price::after {
content: " incl. VAT";
}

.product-card--new::before {
content: "New";
}

.shipping-info::before {
content: "Free delivery on orders over $50";
}

The user sees "Free delivery on orders over $50". The crawler doesn't, because that text isn't in the HTML – it exists only in the stylesheet, and the crawler doesn't apply CSS. It's not in textContent either, it can't be copied, and a page excerpt pasted into an assistant won't contain it. Pseudo-elements are fine for decoration – icons, arrows, quotation marks. Any information someone might quote should be real text in the HTML.

The same rule applies to text in images. A "−30% on the whole collection until Sunday" banner exported as a single JPG is, to a crawler, just its alt – if it has one. It's better to do the same thing with real text laid over the photo with CSS: position: absolute on the text, object-fit: cover on the image and a gradient underneath for contrast. It looks identical, scales with the screen, can be translated – and can be read.

Lazy-loaded content: reviews, "show more", infinite scroll

The third category is content that only arrives on the page after a user action or after scrolling. In e-commerce that's most often reviews – and it's a double loss, because reviews are exactly what people ask AI assistants about: "is this product any good?", "what do buyers complain about?".

tsx

// Reviews fetched when the section scrolls into view
function Reviews({ productId }: { productId: string }) {
const [reviews, setReviews] = useState<Review[]>([])
const ref = useInView(() => {
  fetch("/api/reviews/" + productId).then((r) => r.json()).then(setReviews)
})
return <section ref={ref}>{reviews.map(renderReview)}</section>
}

For performance, that's sensible: you don't fetch two hundred reviews before anyone scrolls to them. But a crawler doesn't scroll, so to it the reviews section is empty. The compromise that usually works best: the first few reviews and a summary (average rating, number of reviews, most common pros and cons) in the server HTML, the rest loaded on scroll or after clicking "Show more". For the user the difference is unnoticeable; for the crawler it's the difference between "no reviews" and concrete arguments.

The same applies to truncating text. A description shortened with styles – line-clamp, max-height with a gradient and a "Read more" button – is fully in the HTML. A description shortened in code – description.slice(0, 300) with the rest fetched on click – has three hundred characters in the HTML.

css

/* The whole description is in the HTML, the first 4 lines are visible */
.description:not(.is-expanded) {
display: -webkit-box;
-webkit-line-clamp: 4;
-webkit-box-orient: vertical;
overflow: hidden;
}

Next.js: what ends up in the HTML and what doesn't

In Next.js with the App Router, you get a lot of this for free, but not all of it. Four cases are worth knowing:

• Server Components render on the server, and their output goes into the HTML. Data fetched in a Server Component – product description, specs, the first reviews – is visible to every crawler.
• Client Components ("use client") are also rendered on the server for the first request. It's a common misconception: "use client" doesn't mean "browser only". The tabs from the example above can be a client component and still send every panel in the HTML.
• Data fetched in useEffect, or by a library that only runs in the browser, won't make it into the HTML – effects don't run on the server. The same goes for components loaded with dynamic(..., { ssr: false }).
• Streaming with Suspense – described in the use() and Suspense article – sends content in the same HTTP response, just later. A crawler that downloads the whole response also gets the streamed content, even though it's placed at the end of the document and moved into position by a small script. The text is in the HTML, so the crawler will read it.

The safest pattern for a product page: data fetched in Server Components, interactivity in small client components that receive content as props or children rather than fetching it themselves.

How to check what the crawler sees

No need to guess. The simplest test is to download the page without a browser and look for a specific phrase from the hidden content:

bash

# Is a phrase from the spec table in the HTML sent by the server?
curl -s https://example.com/product/office-lamp | grep -c "Luminous flux"

# The same with one of the bots' User-Agent headers
curl -s -A "GPTBot" https://example.com/product/office-lamp | grep -c "Luminous flux"

0 means the crawler won't see that phrase. The second way doesn't need a terminal: in devtools, disable JavaScript (in Chrome: command palette, "Disable JavaScript") and reload. What you see then – or don't see – is very close to what the crawler gets. The third: "View page source" instead of the Elements tab. Elements shows the DOM after scripts have run; the source shows the HTML the server sent.

Pitfalls and best practices

• Hiding with styles doesn't hurt; being absent from the HTML does. display: none and hidden keep the text in the document. {open && <Panel />} removes it from the HTML sent by the server.
• Render tabs and accordions in full. Hide inactive panels with the hidden attribute, or hidden="until-found", which also lets users find content with Ctrl+F.
• Information is text, not CSS content or an image. Leave pseudo-elements to decoration; lay banner text over the photo with CSS.
• The most important reviews and a summary in the server HTML. Loading on scroll is fine for the rest, but a crawler will never scroll.
• Truncate text with styles, not code. line-clamp keeps the whole description in the HTML; slice() doesn't.
• "use client" doesn't rule out SSR; useEffect and ssr: false do. Fetch content in Server Components and pass it to interactive components.
• Test with curl or with JavaScript disabled. The Elements tab in devtools shows the page after scripts; the crawler sees it before them.

Summary

The crawlers feeding AI assistants mostly don't run JavaScript and don't apply CSS – they read the HTML sent by the server and extract text from it. That's why hiding content with styles, which we worry about so much, is almost never the problem: a tab with display: none is as readable to a crawler as the rest of the page. The problem is content that isn't in that HTML – conditionally rendered panels, reviews fetched on scroll, descriptions truncated in code, information inserted via CSS content or baked into an image. The fixes are simple and usually improve accessibility and in-browser search too: render everything and hide with an attribute, fetch the most important content on the server, write information as text. And when you're not sure, curl and a single grep will tell you in a second what a crawler will see of your page.