This page tracks the state of the site itself, the plumbing that decides whether AI answer engines can reach, read, and cite C&EN. It exists to keep everyone on the same page about what is live, what is decided, and what is still waiting. Each card states the observed condition today and who owns the next move.
robots.txt
Decided, deploys after Oct 8
The live file is unchanged since August: it still blocks ChatGPT-User and Applebot, while Perplexity and Claude retrieval agents pass under the wildcard rule. The policy question is now settled; the deployment is not.
- Sep 3 stakeholder meeting (Sharon Jane Chua, Songul Ozdemir, Sujit Boda, Mike Denison) chose Option B: after the hard paywall launches, AI retrieval crawlers receive metadata, an abstract, and the first 100 words. Full text stays protected. Training crawlers stay blocked.
- Full-text access (Option A) was rejected for post-paywall content. A limited, reversible pre-paywall trial remains possible but is not scheduled.
- Sujit Boda owns the robots.txt update, after the Oct 8 launch. Leadership sign-off on timing sits with Ozdemir and Boda.
- Confirmed Sep 17: no deployment before October.
Why it matters: ChatGPT sends C&EN about five times the referral traffic Perplexity does, and it is the engine whose fetch agent is blocked.
What crawlers receive
Truncated at ~500 words
Unchanged. The six-article spot check from August still stands: served body text is cut off well short of the declared length on longer pieces.
Roughly 19,000 to 29,000 tokens of scripts and menu markup load before the H1, against a crawl budget of about 10,000. The bot-simulation diagnosis is still pending; no results reported. Editorial workarounds shipped meanwhile: a crawler-readable Top 10 table on the Global Top 50 page (Aug 20) and answer-first edits to the flagged pilot articles (Aug 28).
Paywall
Phase 2A launches Oct 8
Today the site runs a native cookie-based, client-side meter. Piano is installed but not blocking.
- Phase 2A, the server-side hard paywall, launches Oct 8. Everything past the paragraph containing the 100th word renders only after access validation.
- What crawlers get after launch is now decided (Option B, above). The AI abstract component that feeds it is in progress, with operational process concerns still being worked through.
- Applying abstracts to the archive is roughly 70,000 articles; AI-drafted with editor approval is the assumed path.
Measurement
Running
The measurement program is live and reporting weekly.
- 100-query set; Core scope runs every Monday across five engines; monthly and quarterly sweeps on schedule. Eight cadence runs archived since the Aug 6 baseline.
- Dashboard retrofitted Sep 21: Over time view, actions log, companion guide. The Aug 28 content changes are on the log as the first treatment to read against.
- AI referral traffic is now tracked from Adobe Analytics: about 28,600 measurable visits year to date, 72% from ChatGPT.
- Gemini model changed Sep 21; Gemini figures before and after are not directly comparable.