Cloudflare has introduced a new feature that lets customers block AI web crawlers from scraping their content without permission or payment, NaijNaira can report.
According to Computer Weekly, this update gives publishers more control over how their intellectual property is used by AI models.
Matthew Prince, Cloudflare’s CEO, said, “If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone.”
The change comes as backlash grows over AI models training on creative work—like music, journalism, art, and video—without consent or compensation.
Cloudflare now lets site owners choose whether to allow AI crawlers to access their content and under what conditions.
AI firms must declare if their bots are crawling for training, inference, or search purposes so creators can decide what’s acceptable.
New Cloudflare users will be asked during setup if they want to block AI crawlers, while existing users can enable the setting manually.
Over a million users had already activated crawler blocking since Cloudflare’s initial rollout in 2024.
Roger Lynch, CEO of Condé Nast, said, “When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation.”
Kristin Heitmann of the Associated Press added, “The value of accurate, factual, nonpartisan journalism has never been more essential.”
Sharon Moshavi of ICFJ+ said, “We see journalists across the world providing vital, original reporting… yet AI bots scrape their work for free.”
Cloudflare also launched a private beta for a tool called Pay Per Crawl, designed to let creators charge AI firms for content access.
Engineers Will Allen and Simon Newton said it uses HTTP status codes, including the rarely used 402, to prompt payment or block access.
Each time an AI crawler requests content, it must either confirm payment or receive a 402 “Payment Required” response.
Cloudflare handles both the payment infrastructure and authentication, acting as the Merchant of Record.
The company hopes this helps creators stay motivated to publish quality work and develop new models for fair AI-data exchange.
They envision broader use cases in the future, such as variable pricing depending on content type or crawler purpose.
The shift could redefine how digital content is accessed and monetized in the AI-driven web.
Article updated 3 months ago. Content is written and modified by multiple authors.
