<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Bidvix]]></title><description><![CDATA[Bidvix]]></description><link>https://bidvix.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 14:33:48 GMT</lastBuildDate><atom:link href="https://bidvix.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[I audited 111 sites for AI-search readiness. The 19 biggest content sites scored zero.]]></title><description><![CDATA[9 AI crawlers, 111 sites, 4 assets. The 19 highest-authority content sites publish zero llms.txt files - and get mentioned 13x more than the ones that do.]]></description><link>https://bidvix.hashnode.dev/i-audited-111-sites-for-ai-search-readiness-the-19-biggest-content-sites-scored-zero</link><guid isPermaLink="true">https://bidvix.hashnode.dev/i-audited-111-sites-for-ai-search-readiness-the-19-biggest-content-sites-scored-zero</guid><category><![CDATA[aiseo]]></category><category><![CDATA[marketing]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Thu, 12 Nov 2026 11:25:02 GMT</pubDate><content:encoded><![CDATA[<p>Last week I published a smaller version of this audit — 30 sites, four AI crawlers. The most common reply was "sample too small." Fair.</p>
<p>So I rebuilt it: <strong>111 sites, 9 AI crawlers, 4 site-level assets, plus two external signals</strong> (Tranco traffic rank, Hacker News mention count). English and Chinese sites. And I split the sample by <em>business model</em> instead of grouping everything into "big sites vs small sites."</p>
<p>That last split is where the data stops being a curiosity and starts being useful.</p>
<p>Because the top of the web is not doing what the GEO industry says it's doing. <strong>Of the 19 content sites ranked inside the global top 1,000, exactly zero publish an <code>llms.txt</code> file.</strong> Not one. Meanwhile 7 of their top-1,000 neighbours do — and all 7 sell software or website builders.</p>
<p>Here's the full dataset.</p>
<h2>What I measured</h2>
<p><strong>Site-level (fetched with a normal desktop Chrome user agent, no special treatment):</strong></p>
<table>
<thead>
<tr>
<th>Signal</th>
<th>How it was judged</th>
</tr>
</thead>
<tbody><tr>
<td>9 crawler rules</td>
<td>Parsed <code>robots.txt</code> groups properly (wildcard fallback, <code>Allow: /</code> override), for GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent</td>
</tr>
<tr>
<td><code>llms.txt</code></td>
<td>Must return 200, start with <code>#</code>, and be &gt;40 chars. A SPA catch-all page does not count</td>
</tr>
<tr>
<td><code>sitemap.xml</code></td>
<td>Must contain <code>&lt;urlset</code> or <code>&lt;sitemapindex</code></td>
</tr>
<tr>
<td>JSON-LD</td>
<td><code>"@type"</code> present in server-rendered homepage HTML. FAQPage / Organization recorded separately</td>
</tr>
</tbody></table>
<p><strong>External:</strong></p>
<ul>
<li><strong>Tranco rank</strong> — a research-grade top-1M list, used as a proxy for authority. Sites are bucketed ≤1k / 1k–10k / &gt;10k.</li>
<li><strong>Hacker News mentions</strong> — <code>nbHits</code> for <code>"domain.com"</code> via the public Algolia API. This is a proxy for <em>third-party written coverage</em>, not for AI citations. I'm explicit about that because it matters for the conclusion.</li>
</ul>
<p><strong>Sample:</strong> 111 sites (68 English, 43 Chinese). 91 of them returned a usable <code>robots.txt</code>; the other 20 (Wikipedia, Stack Overflow, Reddit, Medium, Quora, Forbes…) were unreachable from this network and are excluded from all <code>robots.txt</code> percentages rather than guessed at. They stay in the external-signal analysis.</p>
<h2>Finding 1: the top 1,000 publish almost no GEO assets — and get mentioned the most</h2>
<p>Top-1,000 sites (Tranco ≤ 1,000), split by what they sell:</p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Sites</th>
<th>Have <code>llms.txt</code></th>
<th>Have JSON-LD</th>
<th>Median HN mentions</th>
</tr>
</thead>
<tbody><tr>
<td>Content / media / community</td>
<td>19</td>
<td><strong>0 (0%)</strong></td>
<td>3 (16%)</td>
<td><strong>12,963</strong></td>
</tr>
<tr>
<td>Software / SaaS / dev tools</td>
<td>13</td>
<td>4 (31%)</td>
<td>6 (46%)</td>
<td>991</td>
</tr>
<tr>
<td>Commerce / marketplaces</td>
<td>8</td>
<td>3 (38%)</td>
<td>3 (38%)</td>
<td>242</td>
</tr>
</tbody></table>
<p>Read that again. The 19 highest-authority content sites — Wikipedia, Stack Overflow, Reddit, Medium, Quora, TechCrunch, The Verge, Wired, CNET, Forbes, HBR, McKinsey, Baidu, Sina, Sohu, 163, QQ, Xiaohongshu, Bilibili — have <strong>zero</strong> <code>llms.txt</code> files between them. And they are mentioned on Hacker News <strong>13× more often</strong> than the top-1,000 software companies that do publish one.</p>
<p>The 7 sites in the top 1,000 that do have an <code>llms.txt</code>: HubSpot, Atlassian, Slack, Mailchimp, Shopify, Squarespace, Wix. Every one of them is selling you a tool, a store, or a website. Not one is in the business of owning content.</p>
<p>This is the cleanest version of the argument I can put in a table: <strong><code>llms.txt</code> correlates with wanting to be found. It does not correlate with being cited.</strong> If it did, Wikipedia would be invisible and Wix would dominate AI answers.</p>
<h2>Finding 2: blocking AI crawlers is a business-model decision, not a technical one</h2>
<p>This is the part that surprised me. Everyone frames <code>robots.txt</code> AI-blocking as a capability question — "do you have the technical maturity to control crawlers?" The data says it's a revenue question.</p>
<p>Sites blocking <strong>at least one</strong> AI crawler, by type:</p>
<table>
<thead>
<tr>
<th>Type</th>
<th>Blocking ≥1 crawler</th>
<th>Blocking GPTBot</th>
</tr>
</thead>
<tbody><tr>
<td>Content / media / community</td>
<td><strong>12/36 — 33%</strong></td>
<td>9/36 — 25%</td>
</tr>
<tr>
<td>Software / SaaS / dev tools</td>
<td>4/42 — 10%</td>
<td>2/42 — 5%</td>
</tr>
<tr>
<td>Commerce / marketplaces</td>
<td>0/13 — 0%</td>
<td>0/13 — 0%</td>
</tr>
</tbody></table>
<p>And it holds across all nine crawlers, not just OpenAI's:</p>
<table>
<thead>
<tr>
<th>Crawler</th>
<th>Content sites</th>
<th>Software sites</th>
<th>Gap</th>
</tr>
</thead>
<tbody><tr>
<td>GPTBot</td>
<td>25%</td>
<td>5%</td>
<td>+20 pts</td>
</tr>
<tr>
<td>ClaudeBot</td>
<td>28%</td>
<td>2%</td>
<td>+25 pts</td>
</tr>
<tr>
<td>PerplexityBot</td>
<td>25%</td>
<td>2%</td>
<td>+23 pts</td>
</tr>
<tr>
<td>CCBot</td>
<td>28%</td>
<td>2%</td>
<td>+25 pts</td>
</tr>
<tr>
<td>Google-Extended</td>
<td>25%</td>
<td>2%</td>
<td>+23 pts</td>
</tr>
<tr>
<td>Applebot-Extended</td>
<td>25%</td>
<td>0%</td>
<td>+25 pts</td>
</tr>
<tr>
<td>meta-externalagent</td>
<td>22%</td>
<td>0%</td>
<td>+22 pts</td>
</tr>
</tbody></table>
<p>A consistent 20–25 point gap on every single crawler is not noise. It's a policy.</p>
<p>The logic is embarrassingly simple once you see it. If your content <em>is</em> the product, an AI reading it for free is a competitor. If your product sits behind a signup, an AI quoting your docs is a sales channel. Nobody at Figma or Wix is worried that ChatGPT will clone their pricing page — they want ChatGPT to <em>name</em> them when someone asks for a design tool.</p>
<p>The sites blocking hardest are the ones with something to lose: Baidu (9/9 crawlers blocked), Xiaohongshu (9/9), Sohu (8/9), Zhihu (9/9), Wired (8/9), CNET (8/9), The Verge (7/9), TechCrunch (6/9).</p>
<h2>Finding 3: the middle tier works hardest, the top tier doesn't bother</h2>
<p><code>llms.txt</code> adoption by authority tier:</p>
<table>
<thead>
<tr>
<th>Tier</th>
<th>Sites</th>
<th><code>llms.txt</code></th>
<th>JSON-LD</th>
</tr>
</thead>
<tbody><tr>
<td>≤ 1k</td>
<td>40</td>
<td>18%</td>
<td>30%</td>
</tr>
<tr>
<td>1k–10k</td>
<td>45</td>
<td><strong>44%</strong></td>
<td>49%</td>
</tr>
<tr>
<td>&gt; 10k</td>
<td>16</td>
<td>6%</td>
<td>25%</td>
</tr>
</tbody></table>
<p>The 1k–10k band publishes <code>llms.txt</code> at <strong>2.4× the rate of the top 1,000</strong>. That band is exactly where companies are climbing and can feel the traffic moving — Vercel, Supabase, Linear, Asana, ClickUp, Zapier, Intercom, Product Hunt, Webflow, Semrush.</p>
<p>The top 1,000 don't need the file. The bottom doesn't know it exists. The middle is where the anxiety lives, and anxiety produces checklists.</p>
<p>If you're a mid-size SaaS reading this: <strong>your competitors have probably already shipped it.</strong> 44% is not an early-adopter number anymore. That's the entry ticket, not the advantage.</p>
<h2>Finding 4: the Chinese web is a five-year-old gap, not a one-year one</h2>
<table>
<thead>
<tr>
<th>Signal</th>
<th>English sites (68)</th>
<th>Chinese sites (43)</th>
</tr>
</thead>
<tbody><tr>
<td><code>llms.txt</code></td>
<td>26 — <strong>38%</strong></td>
<td>3 — <strong>7%</strong></td>
</tr>
<tr>
<td>JSON-LD on homepage</td>
<td>33 — <strong>49%</strong></td>
<td>7 — <strong>16%</strong></td>
</tr>
<tr>
<td>Median HN mentions</td>
<td>1,045</td>
<td>10</td>
</tr>
<tr>
<td>Blocking ≥1 AI crawler</td>
<td>21%</td>
<td>13%</td>
</tr>
</tbody></table>
<p>The whole Chinese-language sample produced <strong>three</strong> <code>llms.txt</code> files: ifeng.com, gitee.com, ctrip.com. Three, out of 43, including every major tech media brand and every major cloud provider. Alibaba Cloud, Tencent Cloud, Huawei Cloud, Volcengine, Juejin, CSDN, 36Kr — none.</p>
<p>Structural markup is barely better: 16% versus 49%.</p>
<p>Chinese teams block AI crawlers <em>less</em> than English teams (13% vs 21%) — so the access door is open, but almost nothing is published in a form an engine can parse. That's an unusual combination: <strong>permissive robots, empty markup.</strong> The crawl budget is being spent on pages that give the model nothing structured to lift.</p>
<p>For anyone working in the Chinese market this is the actual finding. It's not that competition is fierce. It's that the floor is empty.</p>
<h2>The uncomfortable aggregate</h2>
<ul>
<li><strong>50 of 111 sites</strong> have none of the three assets (<code>llms.txt</code>, <code>sitemap.xml</code>, JSON-LD) in place.</li>
<li><strong>15 of 111</strong> have all three.</li>
<li>Median HN mentions across the whole sample: 310. Mean: 50,505. The mean is 160× the median, because four domains (GitHub, Medium, Reddit, TechCrunch) sit above 500,000 and drag it there. Any "average mentions" figure in the GEO space is almost certainly built on a mean and is therefore worthless.</li>
</ul>
<h2>What I'd actually do, by type</h2>
<p><strong>If you sell software</strong> — publish the assets, but don't expect them to move citations. 44% of your tier already has <code>llms.txt</code>; skipping it is a self-inflicted wound, shipping it is table stakes. Your real leverage is being written about somewhere a model already trusts, which is a distribution problem, not a markup problem.</p>
<p><strong>If you sell content or media</strong> — decide the crawler policy on purpose instead of by default. Blocking all 9 crawlers keeps your archive intact and removes you from AI answers entirely. Allowing them gives away the archive and buys brand recall. There's no free option; pick one and write it down. Right now most publishers have chosen "block" without running the numbers.</p>
<p><strong>If you're in the Chinese market</strong> — the 7% is real, and it's the cheapest gap in this entire dataset. Structured data plus an <code>llms.txt</code> costs an afternoon. Almost nobody on your side of the web has done it.</p>
<h2>Method limits, stated plainly</h2>
<ol>
<li><strong><code>robots.txt</code> blocking ≠ exclusion from AI answers.</strong> An engine can cite a page it crawled earlier, or via a licensed feed. Treat this as policy intent, not outcome.</li>
<li><strong>HN mentions are a tech-community proxy.</strong> It over-weights developer tools and under-weights consumer and Chinese sites (median 10 for CN). It is not an AI citation count. Measuring actual citations across 111 domains × 5 engines is a different, much more expensive study — that's what my product does, and I didn't want to launder my own tool's output into "independent research."</li>
<li><strong>20 sites were unreachable</strong> from this network and are excluded from <code>robots.txt</code> percentages. The exclusion skews toward very large English sites — Wikipedia, Reddit, Stack Overflow, Medium, Quora, Forbes — so the top-1,000 content numbers are computed on the 19 I could reach, not on an ideal sample.</li>
<li><strong>Type classification is mine</strong>, based on how each site makes money. Reasonable people could file McKinsey as consulting rather than content; the direction of every finding survives those reclassifications, which is why I kept the buckets coarse.</li>
</ol>
<p>Raw data: one CSV row per domain, 9 crawler verdicts and 4 asset flags each. The whole collection script is 120 lines of Python with nothing beyond the standard library — drop your own domain list in and re-run it. That beats trusting my numbers.</p>
<hr />
<p><em>Disclosure: I build <a href="https://bidvix.com/answers/do-i-need-llms-txt?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=geo-audit-111">Bidvix</a>, a tool that checks how often a brand gets cited by ChatGPT, Gemini, Claude, Perplexity and DeepSeek. The audit above was collected independently of the product, and the product's own output is deliberately not part of it. If you want the underlying checklist rather than the tool, <a href="https://bidvix.com/guides/what-is-generative-engine-optimization?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=geo-audit-111">here's the version with no signup</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.]]></title><description><![CDATA[I pulled robots.txt, llms.txt and JSON-LD from 30 sites. Only 1 blocks GPTBot - and the most-cited sites are the least optimized.]]></description><link>https://bidvix.hashnode.dev/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst</link><guid isPermaLink="true">https://bidvix.hashnode.dev/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><category><![CDATA[marketing]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Thu, 12 Nov 2026 02:59:36 GMT</pubDate><content:encoded><![CDATA[<p>Everyone "knows" that the web is locking out AI crawlers. Cloudflare flips the default to block, robots.txt blocklists go viral, every SEO newsletter screams about GPTBot.</p>
<p>So I ran the actual test. I pulled <code>robots.txt</code>, <code>llms.txt</code>, <code>sitemap.xml</code>, and the homepage JSON-LD from 30 well-known sites — and the results were the opposite of the narrative.</p>
<p><strong>Only 1 site out of 30 blocks GPTBot. Zero block ClaudeBot or PerplexityBot.</strong></p>
<p>And the most uncomfortable finding: the sites AI engines cite the most — Wikipedia, GitHub, MDN, Stack Overflow — are the <em>worst</em> at "AI optimization." If technical setup decided citations, that shouldn't be possible.</p>
<p>Here's the full data, the method, and what it actually means if your site isn't Wikipedia.</p>
<h2>The method (reproducible in 10 minutes)</h2>
<p>Two groups of 15, chosen for a reason:</p>
<ul>
<li><strong>Group A — heavy AI-citation sites:</strong> Wikipedia, GitHub, Stack Overflow, MDN, Python.org, React, Next.js, Tailwind, Kubernetes, Stripe, AWS docs, Microsoft Learn, Ruby, Go, Rust</li>
<li><strong>Group B — commercial SaaS sites:</strong> Asana, ClickUp, Notion, Linear, Atlassian, Salesforce, Zoho, Wix, Typeform, Calendly, Loom, Miro, Basecamp, Zendesk, GitLab</li>
</ul>
<p>For each site I fetched with a plain browser user agent (no special treatment), and recorded:</p>
<ol>
<li>Does <code>robots.txt</code> explicitly block GPTBot / ClaudeBot / PerplexityBot / CCBot?</li>
<li>Does <code>/llms.txt</code> return 200?</li>
<li>Does <code>/sitemap.xml</code> return a valid urlset?</li>
<li>Does the homepage ship JSON-LD (<code>"@type"</code>)?</li>
</ol>
<p>Nothing exotic. You can re-run this yourself; the script is a couple of <code>curl</code>s per domain.</p>
<h2>Finding 1: the AI-crawler blockade is mostly a myth at the top</h2>
<table>
<thead>
<tr>
<th>Bot</th>
<th>Explicitly blocked</th>
</tr>
</thead>
<tbody><tr>
<td>GPTBot</td>
<td><strong>1 / 30</strong> (Loom)</td>
</tr>
<tr>
<td>ClaudeBot</td>
<td><strong>0 / 30</strong></td>
</tr>
<tr>
<td>PerplexityBot</td>
<td><strong>0 / 30</strong></td>
</tr>
<tr>
<td>CCBot</td>
<td><strong>0 / 30</strong></td>
</tr>
</tbody></table>
<p>One. Loom was the only site in either group with an explicit GPTBot block. Everyone else — including every major media-adjacent property you'd expect to be defensive — allows all four crawlers by default.</p>
<p>The "web is closing itself to AI" story is real at the margins (some news sites, some forums), but among the sites that actually matter for citations, the door is wide open.</p>
<p>Which leads directly to the uncomfortable part.</p>
<h2>Finding 2: the most-cited sites are the least "optimized"</h2>
<p>Cross the two groups and the pattern flips everything GEO content tells you:</p>
<table>
<thead>
<tr>
<th>Signal</th>
<th>Group A (heavily cited)</th>
<th>Group B (commercial SaaS)</th>
</tr>
</thead>
<tbody><tr>
<td><code>llms.txt</code> present</td>
<td>6 / 15 (40%)</td>
<td><strong>9 / 15 (60%)</strong></td>
</tr>
<tr>
<td>Homepage JSON-LD</td>
<td>4 / 15 (27%)</td>
<td><strong>11 / 15 (73%)</strong></td>
</tr>
<tr>
<td>Valid <code>sitemap.xml</code></td>
<td>4 / 15</td>
<td><strong>12 / 15</strong></td>
</tr>
</tbody></table>
<p>The SaaS group has spent the last two years buying "AI visibility" — and it shows in their plumbing. The sites LLMs actually cite? Wikipedia and Stack Overflow don't ship llms.txt at all. Most dev-doc sites have no JSON-LD on the homepage.</p>
<p><strong>If llms.txt and schema markup drove citations, Wikipedia would be invisible. It is the single most-cited domain in AI answers.</strong></p>
<h2>Finding 3: what actually separates cited from ignored</h2>
<p>Put the three findings together and one model explains all of it:</p>
<p><strong>Citations are decided by third-party trust, not on-site configuration.</strong></p>
<p>ChatGPT cites Wikipedia because thousands of independent sources corroborate it. It cites GitHub and Stack Overflow because developers quote them constantly across the open web. The model doesn't need your schema.org markup to know what these sites are — the rest of the web has already told it, thousands of times.</p>
<p>The SaaS group's pristine llms.txt files are the entry ticket, not the win condition. They make a site <em>parseable</em>; they don't make it <em>trustworthy</em>. Nothing on your own domain can substitute for other people talking about you.</p>
<p>I saw the same thing from the other direction in an earlier experiment — logging every URL five models cited across ~1,400 citation events for 200 buyer prompts. The pattern was infrastructure, not prose: extractable pages, consistent entities, crawlability. But all of it only mattered <em>after</em> the brand existed in third-party sources the models already trusted.</p>
<h2>So what should a normal site actually do?</h2>
<p>If you're not Wikipedia, your leverage looks like this, in order:</p>
<ol>
<li><strong>Get your robots.txt out of the way</strong> — it's the cheapest check you'll ever pass, and being the one site that blocks GPTBot is the one way to guarantee zero AI traffic.</li>
<li><strong>Get into third-party sources the models already trust</strong> — this is where citations are actually decided. Dev.to, Hashnode, community answers, comparison pages, directory listings. One well-placed answer in a trusted community beats a month of on-site tweaks.</li>
<li><strong>Make pages extractable</strong> — answers in one piece, consistent brand entity, clean chunks. This is what converts a mention into a citation.</li>
<li><strong>llms.txt and JSON-LD</strong> — do them, they cost an afternoon, but understand what they are: plumbing, not strategy.</li>
</ol>
<p>The uncomfortable summary: on-site "GEO optimization" is table stakes that the biggest winners barely bother with, because their real asset lives on other people's domains.</p>
<hr />
<p><em>Data collected 2026-11-11 with plain browser user agents, all requests reproducible with <code>curl</code>. Two of the 30 sites (Wikipedia, GitHub) return errors to some automated fetches for <code>robots.txt</code>; their crawler permissions are documented publicly and are permissive — I've excluded them from the llms.txt percentage rather than guess.</em></p>
<p><em>If you want to run this audit against any domain — crawlability, entity consistency, extractability, share of voice against competitors — that's what I built <a href="https://bidvix.com/?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=crawler-survey">Bidvix</a> for. The 5-minute <a href="https://bidvix.com/guides/geo-audit-checklist?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=crawler-survey">AI visibility audit checklist</a> covers the same four checks I used above.</em></p>
]]></content:encoded></item><item><title><![CDATA[I logged every URL five LLMs cited for 200 buyer prompts]]></title><description><![CDATA[Five models, 200 buyer prompts, ~1,400 citation events. The deciding factor was crawlability and extractability, not writing quality.]]></description><link>https://bidvix.hashnode.dev/i-logged-every-url-five-llms-cited-for-200-buyer-prompts</link><guid isPermaLink="true">https://bidvix.hashnode.dev/i-logged-every-url-five-llms-cited-for-200-buyer-prompts</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><category><![CDATA[marketing]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Wed, 11 Nov 2026 03:24:14 GMT</pubDate><content:encoded><![CDATA[<p>Everyone wants the content trick that gets them cited by ChatGPT. After logging citations across 200 buyer prompts and five models, I'm fairly convinced the deciding factor is duller than that: whether a crawler can fetch your page and whether a chunk of it survives extraction.</p>
<p>Here is what I actually did, what I found, and the checklist I now run before writing a single word of new content.</p>
<p>The setup</p>
<p>I took 200 prompts that real buyers type — comparison questions, "best X for Y" questions, troubleshooting questions, and pricing questions — across three categories I know well: developer tools, B2B SaaS, and e-commerce operations.</p>
<p>Each prompt was run three times, on different days, against five systems: ChatGPT, Gemini, Claude, Perplexity, and DeepSeek. Whenever the answer included a citation or a linked source, I logged the URL. When there was no citation, I logged the brand names mentioned and searched for the page that most likely seeded them.</p>
<p>That produced roughly 1,400 logged citation events. Not a peer-reviewed study. But large enough that the same patterns kept repeating.</p>
<p>Finding 1: citations cluster into a small number of page types</p>
<p>Sorting the cited URLs by structure rather than topic, they collapsed into six buckets:</p>
<ol>
<li>Comparison and alternative pages ("X vs Y", "alternatives to X")</li>
<li>Documentation and reference pages with stable URLs</li>
<li>Listicles and roundups that name many products</li>
<li>Forum and community threads with first-hand answers</li>
<li>Pricing and feature pages with explicit, structured facts</li>
<li>Third-party directories and review aggregators</li>
</ol>
<p>In my sample, those six buckets accounted for the overwhelming majority of cited URLs. Original thought-leadership essays were almost never cited, even when they ranked well in Google. That stung, because essays are what everyone wants to write.</p>
<p>The practical translation: if you want to be quoted, publish pages whose structure matches what the model is trying to answer. The model is assembling a comparison; give it a comparison page. It is verifying a price; give it a page where the price is a fact, not a paragraph of positioning.</p>
<p>Finding 2: crawlability is a hard gate, and most teams fail it silently</p>
<p>A model cannot cite a page it never fetched. This sounds obvious, and yet the most common failure I found was not "bad content" but "content that was never retrieved."</p>
<p>Three failure modes, in order of how often I hit them:</p>
<p>The robots.txt block. A surprising number of sites still disallow GPTBot, ClaudeBot, PerplexityBot, or Google-Extended — often inherited from a template, or added during a "block AI scrapers" panic in 2023 and forgotten. Blocking AI crawlers is a legitimate business decision. Doing it by accident is not.</p>
<p>The CDN or WAF block. This one is invisible in your analytics. Cloudflare, Akamai, and bot-management rules frequently block unfamiliar user agents at the edge, before the request ever reaches your origin. Your server logs show nothing because nothing arrived. Your 404 logs show nothing because it was never a valid request. Everything looks fine until you notice you are absent from every AI answer in your category.</p>
<p>The JavaScript-only render. If your content requires client-side rendering and executes slowly, some retrieval pipelines will time out or capture an empty shell.</p>
<p>The diagnostic that actually works: stop reading dashboards, read raw logs. Filter for the specific user agents over a week, excluding your own IPs. If GPTBot appears zero times across 10,000 requests, you have your answer, and it is not a content problem.</p>
<p>Finding 3: extraction happens in chunks, not pages</p>
<p>This is the part that changed how I write.</p>
<p>Retrieval systems split pages into passages and match those passages against the query. A page can rank well, be crawled successfully, and still never be cited — because no single chunk contains a clean, self-contained answer.</p>
<p>What survives chunking looks like this:</p>
<p>One question per page, or one question per H2 section. Two topics on one page means both lose clarity at extraction time.</p>
<p>The answer in the first 40 words of the section. If your H2 is "How much does it cost?" and the next 200 words are context before the number appears, the chunk boundary will often land before the number.</p>
<p>Tables over prose for anything with more than two variables. A pricing table compresses into a citable fact; a pricing paragraph compresses into mush.</p>
<p>Full entity names instead of pronouns. A chunk that says "it integrates with the other tool" is worthless once separated from its page. A chunk that says "Acme Analytics integrates with Snowflake" survives.</p>
<p>I rewrote one product page along those four rules and nothing else — same URL, same design, same keyword targeting. Within a month it started appearing as a cited source on prompts where it had previously been invisible. That is a sample size of one, so treat it as a signal, not a law. But the mechanism is at least coherent: I made the page easier to lift.</p>
<p>Finding 4: entity consistency is the cheapest fix nobody does</p>
<p>Ask five models what your company does, and you will often get five different descriptions. Mine differed on category, founding year, and even the primary use case.</p>
<p>That happens because models assemble an entity from whatever sources they retrieve: your site, your LinkedIn, a directory listing from 2021, a podcast description written by a host who half-remembered your pitch. Inconsistent sources produce an inconsistent entity, and an inconsistent entity is a weak one — harder to match, harder to trust, less likely to be surfaced.</p>
<p>The fix costs almost nothing: write down the canonical facts about your company in one place — one-sentence description, category, founding year, headquarters, pricing model, primary competitor set — and then push those exact strings everywhere you control. Your homepage, your about page, your schema.org Organization markup, your LinkedIn, your Crunchbase, your G2 profile, your directory listings.</p>
<p>Then verify by asking each model the same question once a week and diffing the answers. When the descriptions converge, you know the entity is stabilizing.</p>
<p>Finding 5: recency matters less than people think, but staleness is fatal</p>
<p>I expected a strong recency bias. What I found was closer to a staleness penalty: recent pages were not automatically favored, but pages with visibly outdated facts (a 2022 price, a discontinued feature, a dead competitor still listed) were consistently skipped.</p>
<p>The rule I use: any page that makes a factual claim about price, availability, or product capability needs a visible last-updated date and an actual update every quarter. Not a rewrite — an update.</p>
<p>What I would do in week one</p>
<p>If I were starting from zero tomorrow, in priority order:</p>
<ol>
<li>Run your ten buyer prompts across all five models. Record whether you are mentioned and which URLs are cited instead.</li>
<li>Pull a week of raw server logs and grep for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Zero hits means a block, not a content problem.</li>
<li>Check robots.txt and your CDN rules for the same four user agents.</li>
<li>Write the canonical entity paragraph and push it to every profile you control.</li>
<li>Pick your three most important questions and rewrite those pages so each section answers one question in its first sentence.</li>
<li>Re-run the ten prompts in 30 days and compare. Ignore week-to-week noise; a 10-15% swing is not a trend.</li>
</ol>
<p>The uncomfortable conclusion</p>
<p>The work that moves AI citations is unglamorous: unblocking crawlers, structuring facts, making pages quotable, and keeping the story about your company identical everywhere. None of it produces a satisfying before-and-after screenshot. All of it compounds, because once a page becomes a reliable source for one query, it tends to get reused for adjacent ones.</p>
<p>Content is still the input. But in my logs, the pages that won were the ones that were reachable and liftable — not the ones that were best written.</p>
<p>I build a tool called Bidvix that runs the prompt-and-citation audit described above across ChatGPT, Gemini, Claude, Perplexity, and DeepSeek. The logging method in this post is the same one it automates. If you'd rather do it by hand, the six steps above are enough to start this week.</p>
]]></content:encoded></item><item><title><![CDATA[I shipped 30 pages and the AI cited none of them]]></title><description><![CDATA[Chunk-level extraction explains it: one page per question, answer in the first 40 words, tables over prose, full entity names.]]></description><link>https://bidvix.hashnode.dev/i-shipped-30-pages-and-the-ai-cited-none-of-them</link><guid isPermaLink="true">https://bidvix.hashnode.dev/i-shipped-30-pages-and-the-ai-cited-none-of-them</guid><category><![CDATA[aiseo]]></category><category><![CDATA[webdev]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Wed, 11 Nov 2026 00:08:12 GMT</pubDate><content:encoded><![CDATA[<p>Last year we had a documentation set we were genuinely proud of. Thorough, well-written, nicely laid out.</p>
<p>Then I ran it through the models. Every answer that mentioned us cited third-party pages. Not a single one of our own docs made it into an answer.</p>
<p>The reason turned out to be embarrassingly simple: <strong>those pages were structured as essays.</strong> To pull one clean answer out of them, the model had to assemble it from three sections. Given a choice, models prefer sources where the answer is already sitting there in one piece.</p>
<p>Here's what I changed, and how to retrofit it in an afternoon.</p>
<h2>How a model actually reads your page</h2>
<p>Retrieval-augmented answering works roughly like this:</p>
<ol>
<li>Your page gets retrieved, then <strong>split into chunks</strong> (a few hundred words each)</li>
<li>Each chunk is scored on "can this answer the question on its own?"</li>
<li>Self-sufficient chunks become candidates, ranked by relevance and trust</li>
<li>The answer is generated by <strong>copying or lightly paraphrasing</strong> sentences from those chunks</li>
</ol>
<p>Two consequences follow directly:</p>
<p><strong>Chunks must be self-contained.</strong> If your conclusion is in the intro and the supporting facts are 800 words later, the middle chunk contains an argument with no conclusion. The model extracts it and doesn't know what to do with it.</p>
<p><strong>Key sentences must look like answers.</strong> Generation prefers sentences that can be dropped into a response verbatim. "Let's now explore the various aspects of this topic" contributes nothing.</p>
<h2>Four rules that fixed it</h2>
<h3>1. One page, one question</h3>
<p>This was my biggest mistake. We had a single "product overview" page covering what it is, pricing, competitor differences, and architecture. Four questions on one page means every chunk is half an answer.</p>
<p>Split it: <code>/what-is-x</code>, <code>/x-pricing</code>, <code>/x-vs-y</code>, <code>/x-architecture</code> — each title <em>is</em> the question. A month later, the pricing page started showing up in citations.</p>
<h3>2. Answer in the first 40 words</h3>
<p>Every page should state the conclusion in its opening sentence, written to be quotable.</p>
<p>Before:</p>
<blockquote>
<p>Before we discuss pricing, it's worth understanding the market context. In recent years, the SaaS landscape has become increasingly competitive...</p>
</blockquote>
<p>After:</p>
<blockquote>
<p>Bidvix's free tier covers 20 AI visibility checks per day across 5 models. Paid plans start at $29/month and add historical trends and citation clustering.</p>
</blockquote>
<p>The answer exists by word 40. Everything after is context — useful for the model, but don't expect extraction from it.</p>
<h3>3. Tables and lists over prose</h3>
<p>Models extract facts from structured content far more reliably. Same information, different container:</p>
<table>
<thead>
<tr>
<th>Format</th>
<th>Extraction difficulty</th>
</tr>
</thead>
<tbody><tr>
<td>Three plans described in flowing paragraphs</td>
<td>High</td>
</tr>
<tr>
<td>Three-row comparison table</td>
<td>Low</td>
</tr>
<tr>
<td>FAQ, one Q per block</td>
<td>Lowest</td>
</tr>
</tbody></table>
<p>FAQs deserve special mention. The question <em>is</em> the user's prompt and the answer <em>is</em> the sentence the model wants to quote. That's not coincidence — the FAQ structure is isomorphic to the structure of a generated answer.</p>
<h3>4. Full entity names, with context on first use</h3>
<p>"We", "it", "this product" break at chunk boundaries. A chunk that reads "it supports 5 models" tells the model nothing about who "it" is.</p>
<p>Use the product name in every key block, and on first mention add a short descriptor: "Bidvix (an AI visibility monitoring tool)". This is entity consistency applied <em>inside</em> a page — same principle as making sure your one-liner matches everywhere else on the web.</p>
<h2>The one-afternoon retrofit</h2>
<p>Don't rewrite the site. Fix the five pages that matter:</p>
<ol>
<li><strong>Pick 5 pages</strong> — the five questions users ask most, one page each. Create the missing ones.</li>
<li><strong>Rewrite each first paragraph</strong> — a direct answer, quotable, under 40 words.</li>
<li><strong>Convert prose facts into tables</strong> — pricing, supported scope, comparisons.</li>
<li><strong>Add an FAQ block</strong> — 5-8 real questions, 2-3 sentences each, no filler.</li>
<li><strong>Add matching structured data</strong> — FAQPage / Product / HowTo schema, worded identically to the page.</li>
</ol>
<p>Verify by re-running the same prompts two to four weeks later and checking what share of cited URLs are pages you control. Look at the trend, not a single run.</p>
<h2>One caveat</h2>
<p>"Extractable" is not "written for robots". These rules make pages <em>better for humans</em> too — direct answers, fewer pleasantries, cleaner tables. That's just good technical writing. Where extraction preferences and human reading preferences conflict, it's rarer than people assume, and you don't need separate content for the two audiences.</p>
<p>I built <a href="https://bidvix.com/?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=extractable">bidvix</a> to handle the measurement half — mention rate and cited URLs across five models, so I can see whether the retrofit actually moved anything. The rewriting above needs no tooling at all.</p>
<hr />
<p><strong>Also in this series:</strong> <a href="https://dev.to/dongdongdev/your-product-doesnt-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself-3ki1">The 20-minute AI visibility audit</a> · <a href="https://dev.to/dongdongdev/the-six-kinds-of-pages-llms-actually-cite-and-how-to-find-yours-in-an-afternoon-8i1">The six kinds of pages LLMs actually cite</a> · <a href="https://dev.to/dongdongdev/hiding-you-from-ai-answers-a-5-minute-check-4eb8">Your robots.txt might be hiding you</a></p>
]]></content:encoded></item><item><title><![CDATA[Your brand has five different descriptions across five models. That's why AI skips you.]]></title><description><![CDATA[Entity consistency is the cheapest fix in AI visibility: no new content, just make every source say the same thing.]]></description><link>https://bidvix.hashnode.dev/your-brand-has-five-different-descriptions-across-five-models-that-s-why-ai-skips-you</link><guid isPermaLink="true">https://bidvix.hashnode.dev/your-brand-has-five-different-descriptions-across-five-models-that-s-why-ai-skips-you</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><category><![CDATA[marketing]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Wed, 11 Nov 2026 00:06:56 GMT</pubDate><content:encoded><![CDATA[<p>Most AI visibility advice is about being <em>mentioned</em>. That is the second problem. The first one is being mentioned <strong>as the wrong thing</strong>.</p>
<p>Here is the mechanic that people skip: a language model does not look up your brand in a table. At answer time it reconstructs a description of you from whatever fragments appear across its training data and its retrieval results. If those fragments disagree — different category, different audience, different founder, different spelling — the model does not resolve the conflict. It averages it, or it picks the most repeated fragment and discards yours.</p>
<p>So the question is not "am I visible?" It is "am I <em>the same</em> thing everywhere?"</p>
<h2>Three tests you can run in ten minutes</h2>
<p>Open ChatGPT, Claude, or Perplexity in a fresh session (no memory, no custom instructions) and run these. Write down the exact wording, not your impression.</p>
<p><strong>Test 1 — Category stability.</strong> Ask <code>What is &lt;your brand&gt;?</code> five times in five separate sessions. Do not ask follow-ups.</p>
<p>You are looking for the noun. If it comes back as "a tool", "a platform", "a service", "a company", and "a website" across five runs, you have no stable category. Category is the single strongest signal a model uses to decide <em>when</em> to surface you. An unstable category means you are eligible for every query and selected for none.</p>
<p><strong>Test 2 — Boundary test.</strong> Ask <code>Is &lt;your brand&gt; good for &lt;adjacent job&gt;?</code> — the job one step outside your actual scope.</p>
<p>A model with a clean model of you will say no, and explain what you actually do. A model with a muddy model of you will say yes and invent a reason. That invention is the tell: it means the model has no boundary for you, only a vibe.</p>
<p><strong>Test 3 — Attribution test.</strong> Ask <code>Who makes &lt;your brand&gt;?</code> and <code>What else does &lt;founder or company&gt; make?</code></p>
<p>Wrong answers here are the most expensive kind, because they attach your brand to someone else's reputation. This is also the fastest way to discover that your brand name collides with an unrelated product — common for short, generic names.</p>
<h2>Why the fragments disagree</h2>
<p>Almost always one of four causes.</p>
<p><strong>1. Your pages describe you at different altitudes.</strong> The homepage sells a vision ("the future of X"), the product page lists features, the About page tells a founding story, and the pricing page names a category. Four writers, four categories. A model reading all four has to pick one, and it picks the one that appears most often in third-party sources, not the one you consider canonical.</p>
<p><strong>2. Directory listings froze years ago.</strong> Every directory you submitted to captured a tagline on a specific day. Some of those directories are still what the model retrieves. If your tagline changed and the listings did not, you are permanently described by your earliest, least accurate sentence.</p>
<p><strong>3. Structured data is missing or contradicts the page.</strong> <code>Organization</code> and <code>SoftwareApplication</code> schema with a <code>description</code> that differs from the visible copy is worse than no schema at all — you have now given the model two official-sounding descriptions that disagree.</p>
<p><strong>4. No <code>sameAs</code> graph.</strong> If your site does not explicitly link your entity to its LinkedIn, Crunchbase, GitHub, and app store pages, the model has to infer that these belong to the same company. It often gets this wrong for brands with common words in their name.</p>
<h2>The fix, in the right order</h2>
<p>Order matters, because each step depends on the previous one.</p>
<p><strong>Step 1 — Write one sentence and freeze it.</strong> Format: <code>&lt;Brand&gt; is a &lt;category&gt; that &lt;does the specific job&gt; for &lt;specific audience&gt;.</code> No adjectives, no superlatives, no "AI-powered". This sentence is now the canonical description. Every page, every listing, every profile uses a close variant of it — not identical (that reads as spam), but the same category noun and the same audience.</p>
<p><strong>Step 2 — Make the category noun appear in the places models actually read.</strong> Title tag, H1, first sentence of the meta description, <code>Organization.description</code>, and the first line of your About page. Five slots, one noun.</p>
<p><strong>Step 3 — Add the <code>sameAs</code> graph.</strong> A single <code>Organization</code> block listing every profile URL that represents the same entity. This is cheap and it is the highest-leverage structured data change most small brands have not made.</p>
<p><strong>Step 4 — Re-run the three tests monthly, same questions, same model settings.</strong> What you are tracking is not the answer text, it is the category noun. When it stops moving across five runs, you have an entity. Until then you have a rumour.</p>
<h2>What this looks like when it is broken</h2>
<p>A pattern I see often: a tool whose homepage says "AI visibility platform", whose pricing page says "SEO audit plans", and whose directory listings say "ChatGPT brand monitoring". Three categories. Ask four models what it is and you get three different answers, all defensible, none stable. The fix is not more content — it is deleting two of the three categories.</p>
<p>That is the counterintuitive part of this whole discipline. AI visibility work is mostly <strong>subtraction</strong>: fewer claims, fewer categories, fewer synonyms for what you are. The brands that get cited consistently are boringly consistent.</p>
<hr />
<p>I build a small tracker for the measurement side of this (bidvix), but the three tests above are free, they take ten minutes, and they will tell you more than a dashboard will. Run them before you buy anything.</p>
<hr />
<p><strong>Also in this series:</strong> <a href="https://dev.to/dongdongdev/your-product-doesnt-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself-3ki1">The 20-minute AI visibility audit</a> · <a href="https://dev.to/dongdongdev/the-six-kinds-of-pages-llms-actually-cite-and-how-to-find-yours-in-an-afternoon-8i1">The six kinds of pages LLMs actually cite</a> · <a href="https://dev.to/dongdongdev/hiding-you-from-ai-answers-a-5-minute-check-4eb8">Your robots.txt might be hiding you from AI answers</a></p>
]]></content:encoded></item><item><title><![CDATA[Your robots.txt might be hiding you from AI answers: a 5-minute check]]></title><description><![CDATA[Check GPTBot, ClaudeBot, PerplexityBot and Google-Extended in five minutes - plus the CDN-level blocks that never show up in your server logs.]]></description><link>https://bidvix.hashnode.dev/your-robots-txt-might-be-hiding-you-from-ai-answers-a-5-minute-check</link><guid isPermaLink="true">https://bidvix.hashnode.dev/your-robots-txt-might-be-hiding-you-from-ai-answers-a-5-minute-check</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><category><![CDATA[webdev]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Wed, 11 Nov 2026 00:05:40 GMT</pubDate><content:encoded><![CDATA[<p>Every audit I run starts the same way, and it is not with content. It is with <code>robots.txt</code>.</p>
<p>The pattern: someone has spent six months publishing comparison pages and wondering why ChatGPT never mentions them, and somewhere in the site config a <code>Disallow: /</code> rule was copied from a staging template, or a security plugin added <code>User-agent: *</code> with a blocklist, or a developer blocked "AI scrapers" in 2023 and never revisited it. Two years of content, invisible to the engines.</p>
<p>Here is the check, and the nuance people get wrong.</p>
<h2>1. Which bots matter, and what each one actually does</h2>
<p>These are not interchangeable. Blocking one does not block the others:</p>
<table>
<thead>
<tr>
<th>Bot</th>
<th>Used for</th>
<th>Blocking it costs you</th>
</tr>
</thead>
<tbody><tr>
<td><code>GPTBot</code></td>
<td>OpenAI's crawling for model training</td>
<td>Training signal, but not necessarily ChatGPT search visibility</td>
</tr>
<tr>
<td><code>OAI-SearchBot</code></td>
<td>Building the ChatGPT search index</td>
<td><strong>Appearing in ChatGPT search answers</strong></td>
</tr>
<tr>
<td><code>ChatGPT-User</code></td>
<td>User-initiated fetch ("open this URL")</td>
<td>ChatGPT being able to read your page on demand</td>
</tr>
<tr>
<td><code>ClaudeBot</code></td>
<td>Anthropic's crawler</td>
<td>Claude understanding/citing your pages</td>
</tr>
<tr>
<td><code>PerplexityBot</code></td>
<td>Perplexity's index</td>
<td>Perplexity citations</td>
</tr>
<tr>
<td><code>Google-Extended</code></td>
<td>Gemini / Vertex training controls</td>
<td>Gemini training signal; <strong>does not affect Search ranking</strong></td>
</tr>
<tr>
<td><code>Bingbot</code></td>
<td>Bing index — which Copilot runs on</td>
<td>Copilot visibility entirely</td>
</tr>
</tbody></table>
<p>The one people get wrong most often: they block <code>GPTBot</code> to "opt out of training", feel principled, and don't realise <code>OAI-SearchBot</code> is a separate switch that governs whether they show up in ChatGPT search. Opting out of training is a legitimate choice. Opting out of being <em>cited</em> is usually an accident.</p>
<h2>2. Read your own file</h2>
<pre><code class="language-bash">curl -s https://yourdomain.com/robots.txt
</code></pre>
<p>Then look for four things:</p>
<ol>
<li>A blanket <code>Disallow: /</code> under <code>User-agent: *</code> — kills everything below it unless a specific agent overrides it.</li>
<li>Named agent groups with <code>Disallow: /</code> (<code>User-agent: GPTBot</code> followed by <code>Disallow: /</code> is the classic "we blocked AI in 2023" leftover).</li>
<li>Whole-site blocks hidden in a plugin's "security" tab, which you won't see in the file's first ten lines.</li>
<li>A <code>Sitemap:</code> line that points to a sitemap returning 404 — common after a domain or framework migration.</li>
</ol>
<h2>3. The minimum viable allowlist</h2>
<p>If you want to be readable by assistants and search engines, you need explicit allows (specific agent groups are matched before <code>*</code>, so being explicit is safer than relying on the wildcard):</p>
<pre><code>User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
</code></pre>
<p>Keep the training-crawler decision (<code>GPTBot</code>, <code>CCBot</code>) as a separate, deliberate choice. Just make it deliberately, not by inheritance.</p>
<h2>4. Check the engine that powers the answer</h2>
<p>Copilot runs on the Bing index, and a surprising number of sites are simply not in it. Verify with:</p>
<pre><code>https://www.bing.com/search?q=site:yourdomain.com
</code></pre>
<p>Zero results means no amount of content work will help that slice of users. Submit the sitemap in Bing Webmaster Tools and wait; it is usually days, not weeks.</p>
<h2>5. Then, and only then, work on content</h2>
<p>Order matters. Crawler access → indexing → entity clarity → content → earned mentions. If you skip to step five, you are optimising pages nobody is allowed to read.</p>
<p>This is the boring part of AI visibility, and it is also the part that produces the biggest jumps, because it is binary: either the crawler gets a 200 and reads the page, or nothing else you do matters.</p>
<p>If you want to measure the result afterwards, I wrote about <a href="https://dev.to/dongdongdev/your-product-doesnt-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself-3ki1">auditing whether AI mentions you at all</a> and about <a href="https://dev.to/dongdongdev/the-six-kinds-of-pages-llms-actually-cite-and-how-to-find-yours-in-an-afternoon-8i1">the six kinds of pages models actually cite</a>. The check above is what I run before either of them, and it takes five minutes.</p>
<p>(Disclosure: I build <a href="https://bidvix.com/?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=launch-phase1">bidvix</a>, which measures mention rates across five models — but crawler access is free to fix and I'd do that first either way.)</p>
]]></content:encoded></item><item><title><![CDATA[The six kinds of pages LLMs actually cite (and how to find yours in an afternoon)]]></title><description><![CDATA[Run your buyer prompts, collect every cited URL, cluster them. In every audit they fall into six buckets - and the gap list is your roadmap.]]></description><link>https://bidvix.hashnode.dev/the-six-kinds-of-pages-llms-actually-cite-and-how-to-find-yours-in-an-afternoon</link><guid isPermaLink="true">https://bidvix.hashnode.dev/the-six-kinds-of-pages-llms-actually-cite-and-how-to-find-yours-in-an-afternoon</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><category><![CDATA[marketing]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Wed, 11 Nov 2026 00:04:09 GMT</pubDate><content:encoded><![CDATA[<p>Last time I wrote about <a href="https://dev.to/dongdongdev/your-product-doesnt-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself-3ki1">auditing whether AI mentions your brand</a>. The part people ask about is always the same: <strong>which pages get cited, and how do I get on them?</strong></p>
<p>Here's the short version: run your buyer prompts, collect every URL the answer cites, and cluster them. In every audit I've done, the citations fall into six buckets. Your job is to find the buckets where you're absent <em>but reachable</em>.</p>
<h2>Step 1: collect the citations (30 minutes)</h2>
<p>Twelve prompts, four families: category, problem, comparison, brand. Run them in ChatGPT (with search), Perplexity and Gemini, logged out. Copy every source URL each answer cites into one column. Don't judge yet, just collect — 60 to 100 rows is normal.</p>
<h2>Step 2: cluster into six buckets</h2>
<p>Every URL will land in one of these:</p>
<table>
<thead>
<tr>
<th>Bucket</th>
<th>What it looks like</th>
<th>How you get in</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Directories</strong></td>
<td>"Top 40 AI tools", niche tool indexes</td>
<td>Free/paid submission forms</td>
</tr>
<tr>
<td><strong>Listicles &amp; comparisons</strong></td>
<td>"Best X for Y in 2026", "A vs B"</td>
<td>Pitch the author, or publish your own comparison page</td>
</tr>
<tr>
<td><strong>Docs &amp; changelogs</strong></td>
<td>Product docs, release notes, pricing pages</td>
<td>Yours to write — make one page answer one question</td>
</tr>
<tr>
<td><strong>Community threads</strong></td>
<td>Reddit, Quora, HN, Indie Hackers, forums</td>
<td>Answer real questions, disclose you're the maker</td>
</tr>
<tr>
<td><strong>News &amp; editorial</strong></td>
<td>Trade press, launch coverage</td>
<td>Harder; usually needs a story, not a product</td>
</tr>
<tr>
<td><strong>Code &amp; lists</strong></td>
<td>GitHub repos, awesome-lists, package docs</td>
<td>A PR, if your tool is developer-adjacent</td>
</tr>
</tbody></table>
<h2>Step 3: read the distribution, not the individual URLs</h2>
<p>Count rows per bucket. Two patterns come up constantly:</p>
<ul>
<li><strong>Directories + listicles dominate.</strong> That means the model isn't reading your site, it's reading the pages <em>about</em> your category. Ten good listings beat one beautiful landing page.</li>
<li><strong>Community threads show up more than people expect</strong>, especially for "what tool should I use" style questions. One useful Quora answer can be pulled into answers for months.</li>
</ul>
<p>Now compare with where you actually exist. The gap list is your roadmap — and it's usually shorter than it feels.</p>
<h2>Step 4: fix the boring blockers first</h2>
<p>Before any of this works, two things are often quietly broken:</p>
<ol>
<li><strong>Crawler access.</strong> Check <code>robots.txt</code> for <code>GPTBot</code>, <code>ClaudeBot</code>, <code>PerplexityBot</code>, <code>Google-Extended</code>. Plenty of sites block them by accident via a copied template. Two years of content behind a crawler block is a common, embarrassing finding.</li>
<li><strong>Indexing in the engine that powers the answer.</strong> Copilot runs on Bing. If you have no Bing index, you don't exist for a slice of users regardless of content quality.</li>
</ol>
<h2>Step 5: pick three buckets, not six</h2>
<p>The failure mode is trying all six at once. I'd pick: one directory pass (10 submissions), one community pass (3 thoughtful answers, disclosure included), and one content pass (5 answer-shaped pages — one question per page, answer in the first 40 words).</p>
<p>Then re-run the same twelve prompts two to four weeks later and count the same buckets. If the distribution shifted toward sources you control, it's working. If it didn't, you were probably working on the wrong bucket — which is exactly the thing measurement is for.</p>
<p>If you'd rather not build the spreadsheet, I made <a href="https://bidvix.com/?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=launch-phase1">bidvix</a> for the measurement half (mention rate + cited URLs across five models). The clustering above is the part that actually changes what you do, and it works fine in a notebook.</p>
<hr />
<p><strong>Also in this series:</strong> <a href="https://dev.to/dongdongdev/your-product-doesnt-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself-3ki1">The 20-minute AI visibility audit</a> · <a href="https://dev.to/dongdongdev/hiding-you-from-ai-answers-a-5-minute-check-4eb8">Your robots.txt might be hiding you from AI answers</a></p>
]]></content:encoded></item><item><title><![CDATA[Your product doesn't exist to ChatGPT: a 20-minute AI visibility audit you can run yourself]]></title><description><![CDATA[A 20-minute checklist to find where you are invisible in AI answers, and which fixes actually moved the needle.]]></description><link>https://bidvix.hashnode.dev/your-product-doesn-t-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself</link><guid isPermaLink="true">https://bidvix.hashnode.dev/your-product-doesn-t-exist-to-chatgpt-a-20-minute-ai-visibility-audit-you-can-run-yourself</guid><category><![CDATA[AI]]></category><category><![CDATA[SEO]]></category><category><![CDATA[marketing]]></category><dc:creator><![CDATA[Ke Gao]]></dc:creator><pubDate>Wed, 11 Nov 2026 00:01:25 GMT</pubDate><content:encoded><![CDATA[<p>Ask ChatGPT "what's the best tool for X" and you'll get three or four brand names. If yours isn't one of them, you have a problem Google Analytics will never show you: <strong>the user never reached your site, so there's no traffic to attribute.</strong></p>
<p>I've been chasing this for my own tool. Here's the checklist I use, the data I got, and the fixes that actually moved the needle.</p>
<h2>Why LLMs ignore your product</h2>
<p>LLMs don't "browse" a category. They answer from:</p>
<ol>
<li><strong>Parametric memory</strong> — whatever was in the training data. You can't change this directly.</li>
<li><strong>Retrieval</strong> — the pages that get pulled in for a query (docs, listicles, directories, Reddit threads, comparison pages).</li>
<li><strong>Entity consistency</strong> — does the model have a stable idea of <em>what you are</em>? One-sentence description, same everywhere, with a category.</li>
</ol>
<p>Most indie products fail on #2 and #3, not #1. The model doesn't know you because nothing on the open web says the same, boring sentence about you ten times.</p>
<h2>The audit (20 minutes)</h2>
<p>Pick <strong>4 prompt families</strong>, and 3 variants each — 12 prompts total:</p>
<table>
<thead>
<tr>
<th>Family</th>
<th>Example</th>
</tr>
</thead>
<tbody><tr>
<td>Category</td>
<td>"best AI visibility tools", "tools to check if ChatGPT mentions my brand"</td>
</tr>
<tr>
<td>Problem</td>
<td>"how do I know if AI search engines recommend my product"</td>
</tr>
<tr>
<td>Comparison</td>
<td>"brand monitoring vs AI visibility trackers"</td>
</tr>
<tr>
<td>Brand</td>
<td>"what is bidvix" (baseline: does the model know you at all?)</td>
</tr>
</tbody></table>
<p>Run each prompt <strong>3 times</strong> in ChatGPT, Gemini, Claude, Perplexity, and DeepSeek. Score:</p>
<ul>
<li><strong>Mention rate</strong> — how often you appear (out of runs).</li>
<li><strong>Position</strong> — first three brands mentioned, or buried in a list.</li>
<li><strong>Sentiment</strong> — recommended, neutral, or described wrongly.</li>
<li><strong>Cited sources</strong> — which URLs the model actually read. This is the gold. Those pages are your real competitors for attention.</li>
</ul>
<p>Track it in a sheet. Repeat weekly. A single check is noise; a trend is a signal.</p>
<h2>What my own audit showed</h2>
<p>For <code>bidvix</code>, brand prompts were fine — the model knew the name. Category prompts were the problem: my tool was absent from "best AI visibility tools" answers, while the cited sources were almost entirely <strong>directory listings and listicles</strong>, not vendor sites.</p>
<p>Two conclusions that changed my plan:</p>
<ul>
<li>Being <em>findable on Google</em> is not the same as being <em>quotable by a model</em>. Directories and comparison pages get retrieved; your beautiful landing page often doesn't.</li>
<li>Consistency beats cleverness. When 6 sources describe you with 6 different categories, the model hedges and leaves you out.</li>
</ul>
<h2>Fixes, in order of effort-to-impact</h2>
<ol>
<li><strong>One canonical sentence.</strong> Write it once: <code>X is a [category] that [does the thing] for [who]</code>. Use it verbatim on your site, your profiles, and every directory submission. Don't get creative — get repetitive.</li>
<li><strong>Answer pages, not blog posts.</strong> One page per question your users actually ask ("does ChatGPT mention my brand", "how to track AI search visibility"). Lead with a 40-word answer, then details. These are the pages retrieval likes.</li>
<li><strong>Get into the sources that get cited.</strong> Directories, "best of" lists, comparison pages, community threads. Not 60 of them — 10 that the models already read for your prompts.</li>
<li><strong>Keep a public changelog / docs page</strong> that states what the product does and doesn't do. Models respect hedged, factual pages far more than marketing copy.</li>
<li><strong>Measure again after 2–4 weeks.</strong> Retrieval-heavy models (Perplexity, ChatGPT with search) move fast; parametric answers move slowly.</li>
</ol>
<h2>The uncomfortable part</h2>
<p>You can't A/B test your way into this. There's no dashboard that tells you "you were cited 4 times today" unless you build or buy one — which is, admittedly, why I built mine. If you want to skip the spreadsheet, <a href="https://bidvix.com/?utm_source=devto&amp;utm_medium=article&amp;utm_campaign=launch-phase1">bidvix</a> runs the same 12-prompt check across five models and gives you a mention rate plus the cited URLs.</p>
<p>But the audit above works with a notebook and 20 minutes. Do that first — the numbers will tell you whether you have a content problem or an entity problem, and those have very different fixes.</p>
<p>If you run it, I'd genuinely like to see your results — especially the cited-URLs column. That list is usually a shock.</p>
<hr />
<p><strong>Next in this series:</strong> <a href="https://dev.to/dongdongdev/the-six-kinds-of-pages-llms-actually-cite-and-how-to-find-yours-in-an-afternoon-8i1">The six kinds of pages LLMs actually cite</a> · <a href="https://dev.to/dongdongdev/hiding-you-from-ai-answers-a-5-minute-check-4eb8">Your robots.txt might be hiding you from AI answers</a></p>
]]></content:encoded></item></channel></rss>