C&EN GEO — Current Posture

This page tracks the state of the site itself, the plumbing that determines whether AI answer engines can reach, read, and cite C&EN. It exists to keep everyone on the same page: robots.txt sat undeployed for weeks while several people assumed it was already live. Each card below states the observed condition today and who owns the next move.

Site state

robots.txt Old version live

The live file still blocks ChatGPT-User and Applebot. Retrieval crawlers and the training crawlers ClaudeBot and Applebot-Extended currently pass under the wildcard rule.

Ticket filed Aug 7 by Mike Denison.
Awaiting ACS technology-leadership approval.
Sujit Boda deploys once approved.
A two-week Cloudflare verified-bot trial begins once the new file is live.

What crawlers receive Truncated at ~500 words

A six-article spot check shows served body text cut off well short of the declared article length on longer pieces.

Article	Declared	Served
Evonik	2,621	491
Nigeria antibiotics	1,629	568
Brazil rare earths	2,631	544
OSTP	791	549
Diploid genome	538	475 complete
Cancer metastasis	509	544 complete

Token budget: roughly 19,000 to 29,000 tokens of scripts and menu markup load before the H1, against a crawl budget of about 10,000 tokens. Cause under diagnosis using a bot-simulation script Sujit has.

Paywall Client-side meter today

Today the site runs a native cookie-based, client-side meter. Piano is installed but not blocking.

The October release, "Phase 2A," is a server-side hard paywall that blocks past the paragraph containing the 100th word.
Open decision: what verified retrieval crawlers receive after the migration.

Measurement Running

The measurement program is live and reporting.

100-query set live; Core scope runs weekly across five engines.
The PFAS guide is the flagship test case.
Perplexity shows the most unbranded wins.

Timeline

When	What
Aug 7	Ticket filed and deliverables handed off.
Aug 10	Code freeze starts.
Week of Aug 10	Editorial session.
~Aug 27	Freeze lifts. Template fixes become eligible: externalize inline scripts, slim served nav, add hub schema.
September	Pilot evaluation, schema, and llms.txt.
October	Phase 2A ships (server-side hard paywall).