Original Research: This AI Visibility Benchmark evaluated thirteen leading news and media websites using the same testing method on every article. One tool bug turned up mid-testing, and I caught it before it reached this page. The numbers below are the corrected results.
Everyone assumes major news outlets are ready for AI search. Nobody actually checks.
I picked thirteen well-known news and media publishers, one AI-related article from each, and ran every single one through the same tools, on the same day, using the same method. No guessing. No borrowed data. Just the numbers.
What I found broke almost every assumption I walked in with.
Table of Contents
Research Summary
Thirteen news and media publishers tested. Twenty-six AI Visibility and Citability measurements, plus a full seven-bot crawler check on every site. Same method for every article. One tool bug found, caught, and worked around mid-testing.
Benchmark at a Glance
Highest AI Visibility score: 75 (Wired). Highest Citability score: 86 (Wired). Most open to AI crawlers: Fast Company and Inc, zero of seven bots blocked. Most restricted: BBC, all seven bots blocked.
Average AI Visibility: 50. Average Citability: 48. Average crawler blocks: 3.8 of seven bots per site, about 54 percent of the crawler set.
Here is the number that actually surprised me. The two sites that first showed a perfect crawler score turned out to be wrong. I will get to that.
Key Findings
Nine of the thirteen publishers I tested block at least four of the seven AI crawlers I checked. Wired led the whole benchmark on both AI Visibility and Citability, the only site to top both charts.
BBC produced the third-most citable content in the set, 69 out of 100, while blocking every single AI crawler I tested. Fast Company and Inc, both owned by Mansueto Ventures, share a robots.txt file with the exact same outdated bot names, meaning most of their supposed AI blocks do nothing at all.
Reuters, Fast Company, and Inc all scored 6 on Citability, the weakest content-quality results in the set. My own AI robots.txt Checker gave two sites a false perfect score, and I only caught it by checking the raw files by hand.
Why I Built This Benchmark
Every major outlet now covers AI constantly. Fewer of them have checked whether AI can actually read what they publish.
I wanted real numbers, not assumptions. Not “big publishers must have this handled.” Just what showed up when I tested the actual pages.
Benchmark Methodology
Why These Thirteen Websites
I picked thirteen sites that represent a real spread of the news and media industry: general news, business news, and tech-focused outlets. All thirteen have real audiences and real editorial standards. All thirteen have published AI-related coverage recently, which gave me a like-for-like article to test on each one.
Articles Tested
Testing Process
I ran every article through the same tools, in the same order, on the same day. One article per site, no retesting to chase a better number, and no manual scoring.
Testing date: August 2, 2026. All thirteen articles were tested on the same day. Crawler-access findings reflect the robots.txt files available at the time of testing.
The Tools I Used, and the Bug I Found Mid-Testing
AI Visibility Checker
Measures overall AI readiness: technical signals, structural elements, and metadata quality. Returns a score out of 100.
Content Citability Grader
Measures how likely a piece of content is to get cited by an AI system, across Evidence, Structure, Authority, and AI Readability. Each dimension is worth 25 points.
AI Crawler Access, Checked By Hand
I originally planned to run my own AI robots.txt Checker as the third tool, same as I did with Schema on my last benchmark. Partway through testing, I found it misreports certain robots.txt structures, showing blocked bots as allowed. Two sites, The Guardian and Ars Technica, both showed false 100 out of 100 scores before I caught it.
I confirmed the bug using an independent robots.txt parser, then manually checked all seven bots, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, CCBot, and anthropic-ai, on every one of the thirteen sites. The crawler numbers in this piece come from that manual check, not my own tool.
That is worth sitting with for a second. My own site caught its own tool getting this wrong. I would rather tell you that than quietly fix it and pretend it never happened.
Overall Benchmark Results
| Website | AI Visibility | Content Citability | Bots Blocked (of 7) |
|---|---|---|---|
| Wired | 75 | 86 | 4 |
| The Guardian | 69 | 53 | 4 |
| Ars Technica | 65 | 54 | 5 |
| AP | 59 | 52 | 5 |
| Business Insider | 59 | 59 | 3 |
| The Verge | 59 | 66 | 5 |
| CNBC | 48 | 49 | 6 |
| Forbes | 45 | 70 | 1 |
| BBC | 45 | 69 | 7 |
| Reuters | 44 | 6 | 4 |
| TechCrunch | 32 | 49 | 5 |
| Fast Company | 24 | 6 | 0 |
| Inc | 24 | 6 | 0 |
| Average | 50 | 48 | 3.8 |
No two sites showed the same profile. Some led on content. Some led on access. Almost none led on both.
AI Visibility Hall of Fame
Best Overall AI Visibility, Wired, 75. The highest raw score in the whole benchmark, and it held up across every signal I checked.
Best Content Citability, Wired, 86, rated Excellent by my own grader. No other site cleared 70.
Most Open to AI Crawlers, Fast Company and Inc, zero of seven bots blocked. Worth a caveat here: most of those blocks were written against outdated bot names that no longer match real crawlers, so the openness is partly an accident, not a deliberate choice.
Most Restricted, BBC, seven of seven bots blocked, the only fully closed site in the set.
Biggest Surprise, BBC again. Its content scored 69 on Citability, third-best in the whole benchmark, while its crawlers score sits at zero access. One of the most citable articles I tested is also the least reachable one.
Most Interesting Outlier, Fast Company and Inc, sharing one broken robots.txt template between two properties owned by the same parent company, Mansueto Ventures.
Overall AI Visibility Scores
AI Visibility Checker results, thirteen news and media sites, same method, same day.
| Rank | Website | AI Visibility Score |
|---|---|---|
| 1 | Wired | 75 |
| 2 | The Guardian | 69 |
| 3 | Ars Technica | 65 |
| 4 | AP | 59 |
| 4 | Business Insider | 59 |
| 4 | The Verge | 59 |
| 7 | CNBC | 48 |
| 8 | Forbes | 45 |
| 8 | BBC | 45 |
| 10 | Reuters | 44 |
| 11 | TechCrunch | 32 |
| 12 | Fast Company | 24 |
| 13 | Inc | 24 |
| — | Benchmark Average | 50 |
Tested using the Free AI Visibility Checker at nenawow.com, same method, same day, one article per site.
Content Citability Scores
Content Citability Grader results, and this is where the real spread showed up.
| Rank | Website | Content Citability Score |
|---|---|---|
| 1 | Wired | 86 |
| 2 | Forbes | 70 |
| 3 | BBC | 69 |
| 4 | The Verge | 66 |
| 5 | Business Insider | 59 |
| 6 | Ars Technica | 54 |
| 7 | The Guardian | 53 |
| 8 | AP | 52 |
| 9 | TechCrunch | 49 |
| 9 | CNBC | 49 |
| 11 | Fast Company | 6 |
| 12 | Inc | 6 |
| 13 | Reuters | 6 |
| — | Benchmark Average | 48 |
TechCrunch and CNBC landed close together, both near 49, while Fast Company, Inc, and Reuters all landed at exactly 6. That is not noise. That is a real pattern worth a closer look.
Individual Website Analysis
TechCrunch
AI Visibility 32, Citability 49, five of seven bots blocked. TechCrunch writes about AI constantly, including the article I tested here, and still comes back Heavily Restricted on crawler access.
TechCrunch blocks GPTBot, ClaudeBot, Google-Extended, CCBot, and anthropic-ai, while letting Perplexity through. That is a deliberate, specific choice, not a technical accident. A publisher covering AI search as a beat is also one of the harder sites in this set for AI systems to actually reach.
Research conclusion: TechCrunch’s coverage of AI outpaces its own openness to it.
Forbes
AI Visibility 45, Citability 70, just one bot blocked. Forbes produced the second-highest Citability score in the set at 70, one point ahead of BBC.
That combination, strong content and near-full crawler access, makes Forbes one of the more balanced performers here. It never leads a single category, but it never falls near the bottom either.
Research conclusion: Forbes is the steadiest, least dramatic result in the whole benchmark.
Fast Company
AI Visibility 24, Citability 6, zero bots blocked, technically. Fast Company’s raw robots.txt file lists eleven AI-related blocks, but most target crawler names that no longer match how these bots actually identify themselves.
Only its content scores are real weak points here. A 6 on Citability is close to the floor of the whole scale, and 24 on AI Visibility puts it among the lowest in the set.
Research conclusion: Fast Company’s openness may be accidental, since the shared rules target crawler names that no longer match real bots. Its content still needs real rebuilding.
Inc
AI Visibility 24, Citability 6, zero bots blocked, same as Fast Company. Inc shares its parent company, Mansueto Ventures, with Fast Company, and it shares something else too: the identical outdated robots.txt template, right down to the same non-functional bot names.
Two different newsrooms, two different articles, and one shared technical file with the same outdated bot names. That is a rare, specific finding, not a coincidence.
Research conclusion: Inc’s numbers are Fast Company’s numbers, because the underlying setup is the same file.
CNBC
AI Visibility 48, Citability 49, six of seven bots blocked. CNBC’s file is clean and correctly written, using the exact current names for every major crawler, and it blocks nearly all of them anyway.
That is the sharpest contrast in this section of the benchmark. A financial outlet reporting on a 250 billion dollar AI infrastructure story, fully and correctly walling itself off from the AI systems that story is about.
Research conclusion: CNBC’s block is real, deliberate, and almost complete.
The Guardian
AI Visibility 69, Citability 53, four of seven bots blocked, once I corrected my own tool’s false reading. The Guardian’s robots.txt technically allows Google-Extended, but genuinely blocks ClaudeBot and PerplexityBot, not the perfect access my checker first reported.
Even more worth reading closely: the file opens with a plain-English notice stating that AI and LLM use of Guardian content is not permitted without a license. Being crawlable and being authorized are two different things, and this is the clearest example of that gap I found anywhere in the set.
Research conclusion: The Guardian is technically reachable and legally closed at the same time.
Ars Technica
AI Visibility 65, Citability 54, five of seven bots blocked, once corrected. Same story as The Guardian: my own tool first reported a perfect 100, and a manual check found ClaudeBot, PerplexityBot, and Google-Extended all genuinely blocked.
Ars Technica’s raw file stacks a long list of user-agents under one shared Disallow rule, a pattern my checker apparently cannot parse correctly. Worth knowing if you run your own robots.txt audit and see a similar structure.
Research conclusion: Ars Technica is more restricted than any automated first read would tell you.
Reuters
AI Visibility 44, Citability 6, four of seven bots blocked. Reuters ties Fast Company and Inc at the very bottom of the Citability scale, a real, specific weak point given how much AI-industry news Reuters actually breaks.
That gap is worth naming plainly, though with just one Reuters article tested, this reads as a pattern in this sample, not a verdict on wire journalism as a whole.
Research conclusion: this Reuters article was fast. It was not citable.
Associated Press
AI Visibility 59, Citability 52, five of seven bots blocked. AP breaks the low-citability pattern that Reuters, Fast Company, and Inc all showed, landing in the middle of the pack instead of the floor.
So is straight news reporting always weak on citability? Not always. AP’s policy coverage carries more structure and sourcing than the other wire-style pieces I tested.
Research conclusion: AP proves the wire-format weakness is not universal.
Business Insider
AI Visibility 59, Citability 59, three of seven bots blocked. Business Insider is the only site in the whole benchmark where both scores landed on the exact same number.
That symmetry is a small detail, not a major finding, but it is a clean one. Nothing here is a standout, and nothing here is a real weak point either.
Research conclusion: Business Insider is the definition of an average result.
BBC
AI Visibility 45, Citability 69, seven of seven bots blocked. BBC is the sharpest contrast in the entire benchmark. Its content sits second only to Wired on Citability, and it is the only site here blocking every major AI crawler completely.
One of the most citable articles I tested this cycle is also the one AI crawlers have the least chance of reaching. That is worth sitting with.
Research conclusion: BBC proves that great content and AI access are not the same problem.
Wired
AI Visibility 75, Citability 86, four of seven bots blocked. Wired leads the whole benchmark on both major scores, six points clear of The Guardian on Visibility and sixteen points clear of Forbes on Citability.
Something specific about this article’s structure, its named legal sources, its direct definitions of a messy legal question, is driving that gap. This is the strongest single result in the whole set.
Research conclusion: Wired is the benchmark’s clear leader, not a close one.
The Verge
AI Visibility 59, Citability 66, five of seven bots blocked. The Verge lands mid-pack on Visibility but pulls ahead on Citability, its 66 sitting just behind BBC and Forbes.
That gap between the two scores is worth noting. Strong writing does not automatically pull crawler access or technical signals up with it.
Research conclusion: The Verge writes better than its technical setup gives it credit for.
Patterns I Discovered
These are the patterns that held up across thirteen pages, not conclusions about any one newsroom.
Content quality and crawler access measure completely different things
BBC scored 69 on Citability and got fully blocked. Fast Company scored 6 on Citability and left every crawler open. The two numbers do not move together, and treating them as one score would hide the real story on both sites.
A perfect crawler score deserves a second look, not blind trust
Two of the cleanest-looking results in this whole benchmark turned out to be wrong. If your own checker ever hands you a flat 100, read the raw file before you believe it.
Straight news writing varied widely on citability in this sample
Reuters, Fast Company, and Inc all landed at exactly 6. AP and Business Insider, covering similar ground with more structure and sourcing, both cleared 50. Worth watching whether this holds on a larger set, but format looks like it matters as much as topic here.
Shared ownership can mean shared mistakes
Fast Company and Inc post the same broken robots.txt file. If you run more than one property, check whether your technical setup is actually shared on purpose.
No two sites showed the same weakness
Some lead on content and lag on access. Some lead on access and lag on content. Nobody here got everything right.
Do News Websites Block AI Crawlers?
Manually verified results across all seven tracked bots, corrected after finding a bug in my own checker.
| Website | GPTBot | ClaudeBot | PerplexityBot | Google-Extended | OAI-SearchBot | CCBot | anthropic-ai |
|---|---|---|---|---|---|---|---|
| TechCrunch | Disallowed | Disallowed | Allowed | Disallowed | Allowed | Disallowed | Disallowed |
| Forbes | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Disallowed |
| Fast Company | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| Inc | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| CNBC | Disallowed | Disallowed | Disallowed | Allowed | Disallowed | Disallowed | Disallowed |
| The Guardian | Allowed | Disallowed | Disallowed | Allowed | Allowed | Disallowed | Disallowed |
| Reuters | Allowed | Disallowed | Disallowed | Allowed | Allowed | Disallowed | Disallowed |
| Associated Press | Disallowed | Disallowed | Disallowed | Allowed | Allowed | Disallowed | Disallowed |
| Business Insider | Allowed | Disallowed | Allowed | Allowed | Allowed | Disallowed | Disallowed |
| Ars Technica | Allowed | Disallowed | Disallowed | Disallowed | Allowed | Disallowed | Disallowed |
| BBC | Disallowed | Disallowed | Disallowed | Disallowed | Disallowed | Disallowed | Disallowed |
| Wired | Allowed | Disallowed | Disallowed | Disallowed | Allowed | Disallowed | Allowed |
| The Verge | Allowed | Disallowed | Disallowed | Disallowed | Allowed | Disallowed | Disallowed |
Nine of the thirteen publishers blocked at least four of the seven AI crawlers I tracked. Across the full benchmark, the average publisher blocked 3.8 of seven, about 54 percent of the crawler set. That is not a fringe pattern. Nine of the thirteen publishers in this sample blocked a majority of the AI crawlers I tested.
What Website Owners Can Learn
The lessons here apply whether you run a major newsroom or a single review site.
Check your robots.txt by hand, not just through one tool. Two of my own results were wrong until I verified them independently. If a score looks too clean, it probably is.
Great content and open crawler access are two separate jobs. BBC nailed one and missed the other completely. Do not assume fixing one fixes both.
Shared infrastructure needs its own audit. If you run more than one site under one company, check whether your technical files are actually identical, and whether that was ever a real decision.
Straight reporting can still be citable. AP’s result suggests the difference comes down to sourcing and structure more than the news format itself, though this is one article, not a formula.
Limitations
I found a bug in my own AI robots.txt Checker mid-testing. It misreports certain multi-agent block structures in a robots.txt file, showing genuinely blocked bots as allowed. I confirmed this with an independent parser and manually verified all thirteen sites by hand. The crawler numbers in this piece reflect that manual check, not my own tool’s first read.
I tested one article per site. These scores reflect the tested pages, not each publisher’s entire site.
This benchmark reflects a snapshot at one point in time. News sites update content and technical setup constantly. Scores will change.
My toolkit measures practical AI visibility signals. It does not predict actual future citation rates, which depend on live prompts, model versions, and factors no static tool can fully capture.
Read these results as a comparative snapshot of thirteen tested articles on a given date, not a permanent ranking of the newsrooms behind them.
Nena’s Quick Verdict
The biggest surprise here was not which site scored highest. It was catching my own tool telling me two sites were fully open when they were not.
Wired is the clear, real leader of this benchmark, strong on content and strong on Visibility both. BBC is the sharpest cautionary tale, highly citable content sitting behind a fully locked door. Fast Company and Inc share one broken file and one weak set of scores, proof that shared ownership can mean shared blind spots.
Nine of the thirteen sites here blocked at least four of seven major AI crawlers. The average site blocked 3.8 of seven, about 54 percent of the set. That is not an edge case. That is close to the norm across this sample, whether these newsrooms meant it or not.
For any site owner reading this, the real lesson is simple. Check your own robots.txt by hand. Do not assume a good score means a correct one. I run the exact tool that got this wrong, and I still had to catch it myself.
That is the only version of this benchmark I am willing to stand behind.
Related Reading:
How I Found My Own AI Search Visibility Gap
NenaWow AI Visibility Tools: Tools Every Website Owner Can Use
AI Visibility Benchmark 2026: I Tested 9 Leading SEO Websites
Frequently Asked Questions
Do major news websites block AI crawlers?
Yes. Nine of the thirteen sites I tested blocked at least four of the seven major AI crawlers I checked.
Why did two sites show a perfect crawler score at first, then a lower one?
My own AI robots.txt Checker has a bug that misreads certain multi-agent block structures in a robots.txt file, reporting blocked bots as allowed. I caught this by checking The Guardian and Ars Technica’s raw files by hand, then verified the correction with an independent parser.
Does good content mean an AI system can actually read it?
Not always. BBC scored the third-highest Content Citability result in this benchmark while fully blocking every AI crawler I tested. Strong writing and technical access are two separate problems, and one does not fix the other.
Which news site performed best overall?
Wired led the benchmark on both AI Visibility and Content Citability, the only site to top both charts. Its article on AI and legal liability scored 75 on Visibility and 86 on Citability, both the highest results in the set.
Why do Fast Company and Inc show identical results?
Both sites are owned by Mansueto Ventures and appear to share the same robots.txt file, including the same list of outdated AI crawler names that no longer function as real blocks. That shared technical setup produced nearly identical scores across both properties.
