Cloudflare announced a new control that lets site owners block AI training crawlers while still allowing search engine indexing. The feature, described as accountable mixed-use AI crawlers, distinguishes between bots that train models and bots that build search indexes. Publishers can now deny model training access without losing their placement in search results. The company frames the update as a response to growing complaints that AI companies scrape content for free. Cloudflare says the system relies on declared crawler behavior and verification rather than blanket blocking.
This is a big deal for anyone who publishes online. For years, blocking AI crawlers meant risking your search visibility. Cloudflare just split those two things apart. You can say no to training and yes to indexing. That is a real choice, not a theoretical one.
I think this nudges the whole web toward a healthier equilibrium. AI companies will have to negotiate, pay, or build their own data. Publishers get leverage. Users still find what they need. The open web does not die. It adapts. We are watching the internet grow a new immune system, and I am here for it.