Most teams asking this question are really asking two questions at once, and the answers point in opposite directions. Separating them takes about five minutes and saves a quarter of arguing.
This is deliberately short. The long version, with verification mechanics and signal detail, is in AI agent traffic: telling assistants from attackers.
Question one: is it consuming or transacting?
An agent reading your content costs you bandwidth. An agent creating accounts, redeeming promotions, or completing checkouts can cost you money that does not come back.
These are not the same problem and should never share a rule. If your current policy uses one switch for both, that is the first thing to fix — before any vendor evaluation, before any signal work.
Question two: does the traffic carry customer intent?
When someone asks an assistant to find them a product and it lands on your site, that is a referral with an unusual user agent. Blocking it is a decision to decline the sale.
When automation is enumerating your catalogue, testing cards, or farming signups, no customer asked for it. Blocking it costs you nothing.
The uncomfortable part: both arrive automated, both often arrive from cloud infrastructure, and neither is honestly labelled. That is why the answer cannot be "detect automation" — it has to be "scale the response to what the action costs."
The decision
Run each protected action through this. Most teams find their policy is already right for reads and badly wrong for writes.
| If the action is… | And the traffic is… | Then |
|---|---|---|
| Reading public content | Anything | Serve it. Rate-limit if volume hurts. |
| Search or API query | Unverified | Rate-limit and log. Do not block. |
| Account creation | Automated, no tunnel | Challenge. |
| Account creation | Automated + VPN, proxy, or Tor | Block. |
| Promo or bonus redemption | Automated at all | Block. This is where the losses are. |
| Checkout | Automated | Challenge, and check device linking. |
| Password or email change | Automated | Block, and alert the account owner. |
Three ways this goes wrong
- Allowlisting by user agent. The header is freely settable. An allowlist keyed on a name is an instruction manual for bypassing it. Verify against published vendor IP ranges instead.
- Treating datacenter origin as a verdict. Legitimate agents run in the cloud. Blocking hosting ranges outright removes real referrals while the serious attackers, who moved to residential proxies long ago, sail through untouched.
- Deciding once at the edge. A single allow-or-block at the front door cannot express "you may read this but not buy with it," which is exactly the policy most businesses actually want.
If you only change one thing
Move enforcement from the page to the action, and make promotion redemption the strictest gate on the site. That single change captures most of the loss reduction available here, and it does not cost you a single legitimate referral.
Frequently Asked Questions
Is blocking all AI agents a safe default?
No, it is an increasingly expensive one. A meaningful share of agent traffic now carries genuine customer intent, so a blanket block declines those sales while doing little to the adversarial population, which uses residential egress and freely copies whatever user agent gets allowlisted. Blanket blocking also cannot express the policy most businesses actually want, which is that agents may read content but must not transact unchallenged.
What is the difference between an AI crawler and an AI agent?
A crawler fetches content on the vendor's own schedule to build an index or training corpus, and is generally identified by a stable user agent with published IP ranges. An agent acts on a specific user request in the moment, browsing or transacting on that person's behalf. The practical difference is intent: crawler traffic has no customer behind it right now, while agent traffic often does, which is why the two deserve different rate limits and different treatment at checkout.
Where should I put the strictest rule for agent traffic?
Promotion and bonus redemption, followed by password and email changes. Those actions concentrate the losses, are irreversible or near-irreversible, and have essentially no legitimate automated use case. Reads should stay open, queries should be rate-limited rather than blocked, and account creation and checkout should be challenged rather than refused outright so that real customers using an assistant can still complete.
Enforce by action, not by header
One call returns tunnel, datacenter, and automation signals with a verdict.
Try Sentinel free →