There’s a war brewing in the world of AI agents. After declaring a month ago that it would block AI crawlers by default on its network, Cloudflare openly accused Perplexity of deliberately bypassing internet standards to scrape websites. It published a detailed blog post, explaining how, even if its bots were blocked, Perplexity would use certain tactics—including third-party crawlers—to access those websites anyway.

Perplexity responded swiftly with its own post, pointing out that its use of third-party crawlers was actually significantly less than Cloudflare was saying. But the crux of Perplexity’s rebuttal was that Cloudflare fundamentally misunderstood its bot activity: because its agent bots act on behalf of specific user requests—and not crawling the web generally—Perplexity believe

See Full Page