There's no easy way to put it; the internet as we know it is in trouble, and AI is playing a big part in its destruction.
Cloudflare, one of the world's largest online content delivery networks that serves massive companies like Microsoft and Apple as well as countless smaller companies, has unveiled a "pay per crawl" plan to change the internet's current trajectory.
The new default model will force AI crawlers to pay websites that have opted in for the content they scrape. As of July 1, 2025, a beta version of the Cloudflare "pay per crawl" model is now in place for website owners who choose to enroll.
"Pay per crawl" will not only give content owners some power over which data is handed out for free or sold to AI crawlers, but it will also provide feedback on how the data is being used.
It's an evolution of Cloudflare's 2024 initiative that completely blocked AI crawlers, and this time around, it's a much more nuanced approach.
In the official announcement, Matthew Prince, CEO and co-founder of Cloudflare, says:
"If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself."
Considering that current AI fortunes are largely being built on the back of free data, all the while gutting the websites that provide that free data, this seems like a massive win for publishers the world over.
Indeed, the announcement includes supporting quotes from more than 37 major companies, including The Associated Press, Condé Nast, Pinterest, Ziff Davis, ProRata AI, and TIME.
The support for Cloudflare's plan is largely unsurprising considering the host of lawsuits that have already been filed against AI firms, including one involving eight US-based news publishers and a more recent one filed by Reddit against Anthropic for alleged data theft.
Will there be a pushback against Cloudflare from AI firms?
Cloudflare's new initiative already has support from major players in the online publishing industry, but how will this new model sit with those who are powering those same AI crawlers, including the seemingly omnipotent Google?
What comes to mind immediately is a backlash from AI firms that refuse to pay for the data they crawl. Is having your content safeguarded against AI crawlers better than being completely blacklisted from search pages?
On the other hand, this initiative could be welcomed by the big AI players. After all, OpenAI and Windows Central's parent company, Future, are already partners in the content delivery world.
I'm surmising, but Prince goes on to add a bit of exposition concerning how this new model will also benefit AI firms:
"Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."
The part about helping AI companies innovate remains cloudy, but it seems to me like Prince is acknowledging the fact that Cloudflare's goal isn't to block AI firms from realizing their goals. He simply wants other websites to get their share of compensation.
At least one AI company is on board with Cloudflare's plan. CEO and founder of ProRata AI, Bill Gross, is quoted as saying:
"Our whole mission is to protect and elevate human creativity in the AI era. That’s why we’re proud to be one of the first AI companies to participate in Cloudflare’s initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve to be fairly compensated for the value they bring, and we’re thrilled to support Cloudflare in helping make that vision a reality."
Rather than countless lawsuits and the eventual demise of both AI training and online publishers, why not arrive at a compromise?
How else can the current internet model work in the age of AI?
Cloudflare's "pay per crawl" model seems like a natural reaction to the current AI scramble for free data. Looking in my own backyard, online publishing is being greatly impacted.
Traditionally, the way sites like Windows Central work is like this: content is created and published, which is then indexed by a search engine like Bing or Google. That content is then delivered to would-be readers searching for specific topics.
Readers click through to a website, where they're immersed in an ecosystem of knowledgeable writers and content creators while also generating revenue to pay those same employees.
With the rise of AI, that's increasingly no longer how the internet works. AI is keen to summarize entire webpages and deliver the results before any links are displayed.
You might have already noticed some of your favorite websites going dark, and while their issues can't be solely blamed on AI, it's certainly not helping.
