The Web Says It's Blocking AI Agents for Security. It's Blocking Them Because They Don't Look at Ads.

When Amazon sued Perplexity last November over Comet, the AI browser that can log into a customer's Amazon account and place orders, Perplexity's response went straight at the money. The lawsuit, it said, was a "bald attempt" to keep Amazon shoppers from using AI agents, because agents "don't have eyeballs to see the pervasive advertising Amazon bombards its users with."

Perplexity has an obvious interest in saying that, but the point holds well beyond one lawsuit. Most of the free web is paid for by putting ads in front of people who come to read, search or shop. An agent comes to collect information for someone else. It loads the page, takes what it needs and reports back, and the person it works for never sees the banner or the sponsored listing that was supposed to pay for the visit.

There are now a lot of these visitors. On June 3, Cloudflare CEO Matthew Prince said automated traffic had passed human traffic on the web for the first time, reaching 57.5% of HTML requests about 18 months sooner than he had expected. That figure includes every kind of bot, including search crawlers and scrapers that have been around for decades. The part growing fastest is the newer kind, agents that browse, fill in forms, compare prices and check out on behalf of a specific person.

How Fast Agent Traffic Is Growing

DataDome, a bot-protection company with more than 400 customers, counted 17.7 billion AI agent requests across its network in the second quarter, compared with 12.2 billion in the first quarter.

Spread over the 91 days from April through June, 17.7 billion works out to about 194 million agent requests a day, and that is only the slice of the web DataDome sees. Quarterly growth was 45%. If the same rate continued through the end of the year, DataDome's network would handle about 37 billion agent requests in the fourth quarter, roughly three times the first-quarter count. Nobody has published that as a forecast, and growth this steep usually slows, but even at half the rate the volume would double within a year.

Other measurements show the same trend. HUMAN Security's annual benchmark report found agentic traffic grew 7,851% year over year, most of it in retail, streaming and travel. A single product produces much of it: in June, Perplexity's Comet accounted for 47% of the agentic traffic HUMAN recorded.

Every one of those requests costs the site that serves it something in bandwidth and computing. Almost none of it shows up as revenue, because ads are still sold per thousand impressions to human beings.

Most Agents Are Reading, Not Buying

Companies building agents tend to sell them on commerce, with demos of an assistant booking a flight or reordering groceries. HUMAN's monthly numbers point somewhere else. In its latest report, media sites received 43.5% of agentic traffic, slightly more than e-commerce at 42%. About three-quarters of agent activity was browsing listings, reading articles and running searches. Very little of it ended in a purchase.

Publishers are the businesses least able to absorb that. Amazon has the lawyers and the market power to fight. A mid-sized news site that depends on programmatic advertising mostly pays higher hosting bills for visits it cannot sell. Many of those sites are already losing readers to Google's AI Overviews, which answer a search before anyone clicks through. Agents go a step further, since they do click through and still deliver no reader.

The Courts Have Sided With the Agent, for Now

Amazon's first move against Comet was to block it and then sue. The company argued that Perplexity disguised its agent as an ordinary Chrome browser to avoid detection, and in March a federal judge in San Francisco granted a preliminary injunction. The judge found that Comet entered customer accounts "with the Amazon user's permission, but without authorization by Amazon."

An appeals court threw out that injunction in August. Its reasoning was that under the federal computer fraud law, Perplexity's users were the ones accessing Amazon, through the agent, and Perplexity itself was not. The case is still going. Until it is decided, an agent sent to a website by its user is treated much like the user.

Where the Walls Are Going Up

Infrastructure companies have moved faster than the courts. On July 1, Cloudflare began classifying AI traffic by purpose, separating search indexing, model training and agents acting for a person in real time. As of September 15, new publisher sites that sign up with Cloudflare block agent and training traffic on ad-supported content by default. Search crawlers are still let in. State of Brand covered what that default means for brands that depend on press coverage showing up in AI answers when it took effect last month.

The details of that default show what it is protecting. Cloudflare did not block agents everywhere. It blocked them on pages that earn money from ads, and it kept letting in search crawlers, which still send human readers back to those pages to see the ads. Amazon has framed its fight with Perplexity around security and a worse customer experience, and those concerns may well be genuine. Amazon's advertising business also brought in $19.8 billion in the second quarter, up 26% from a year earlier, and much of that comes from sponsored product listings an AI shopper has no reason to look at. Perplexity told the court that ads were Amazon's real motive. Whatever the stated reasons, the barriers built so far sit on ad-funded pages and in an ad-funded store.

Lawmakers are getting involved as well. In June, New York's State Assembly passed a bill that would require AI crawlers to identify themselves to news publishers, with fines of $15,000 a day for violations.

Efforts to work with agents instead of blocking them have had little effect so far. Many sites now publish an llms.txt file to guide AI systems, but Ahrefs' server-log research found that 97% of those files received zero requests in May.

Selling Ads to Software

A few publishers are trying to sell to agents instead. Time has started running ads aimed at the AI agents that read its pages, one of the first campaigns written with software as the intended audience. Startups have followed. OpenAds identifies agent visitors on publisher sites and serves them sponsored content related to whatever task the agent is working on, and pitches agents to investors as the largest new source of attention since smartphones.

The approach has a basic problem. Content designed to persuade an AI agent is hard to tell apart from content designed to manipulate it, and companies building agents already work to filter out hidden instructions planted on web pages. An ad written for an agent is likely to get the same treatment.

How this plays out will decide a lot about the agent market. Websites could start charging agents per visit, which would shift costs onto the companies that build them. The largest platforms could admit their own agents and block competitors. Or advertising could move inside the agent, so that whoever makes the assistant people rely on controls the main place left to show them anything.

What Companies Should Do Now

Start by measuring how much of your traffic comes from agents. HUMAN, which sells agent detection, says most analytics tools can't distinguish agents from human visitors. If yours can't, your conversion rates, acquisition costs and ad yield figures are probably counting software as customers.

Next, decide how to treat each type of AI traffic separately. Cloudflare already distinguishes search, training and real-time agents, and you can choose which to allow, which to charge and which to block. A blanket policy will treat a model-training scraper the same as an agent placing an order for one of your customers. DataDome's Segura made the same point to Digiday: "Not all AI agents are the same."

If you sell online, plan for some of your customers to send software to shop for them. Amazon CEO Andy Jassy has said agentic commerce could be good for e-commerce, though he added that agents still struggle with personalization and getting prices right. Accurate product data and prices that a machine can read easily will make it more likely an agent picks you.

If your company deploys its own agents, have them identify themselves. Much of Amazon's case rested on Comet presenting itself as Chrome, and New York's bill would put a daily fine on undisclosed crawlers. An agent that hides what it is creates legal risk for the company running it.

Finally, ask your advertising and measurement vendors how they detect and exclude agent traffic. If they don't have a clear answer, some of the impressions you are paying for may never have reached a person.