On October 17, The Verge reported that major websites had started turning AI agents away at the door. Users of Meta Muse, xAI Grok, OpenAI Dots, and Instinct found they could no longer complete tasks on sites that had worked days earlier. Yelp said it admits only agents that pay for its data licensing program. Delta has no partnership that lets third-party agents book flights. JPMorgan Chase and Charles Schwab confirmed they block agents from customer accounts. Amazon has been blocking Meta Muse since September 20. We mapped that news wave on October 17, but the news wave is not the measurement. This article is the measurement.
The short answer: We fetched robots.txt from 15 major domains on October 22, 2026, and checked how each treats eight named AI crawlers. Three domains block all eight at the root. Most publishers block training crawlers (GPTBot, ClaudeBot) while leaving the search-index crawlers that feed AI answers (OAI-SearchBot, Claude-SearchBot) alone. And the sites named in the October news for blocking agents, including Delta and Chase, show no crawler blocks at all in robots.txt. The lockout is enforced by anti-bot systems, terms of service, and lawsuits, not by robots.txt. That split is the whole story: declared policy says one thing, the enforcement wall is somewhere else.
How we measured the lockout
We fetched the live robots.txt from 15 domains on October 22, 2026, with a declared research user agent: nytimes.com, theverge.com, forbes.com, theguardian.com, apnews.com, wired.com, amazon.com, walmart.com, bestbuy.com, delta.com, yelp.com, cars.com, amtrak.com, chase.com, and schwab.com. For each, we parsed every User-agent block and recorded whether each of eight named AI crawlers faces a root-level Disallow (/): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, and CCBot. A crawler with no named block inherits the wildcard rule; we recorded that too. The raw 240-row dataset is saved with this article.
Two honest limits before the numbers. First, one fetch failed: schwab.com serves a bot-mitigation error page to this network, so its robots.txt policy is undetermined in our data, not confirmed open or closed. Second, robots.txt is a declared policy, not proof of enforcement. A Disallow line asks a crawler to stay out; it does not prove the site's firewall enforces it, and the absence of a line does not mean agents can complete transactions there. Treat this audit as the industry's stated position on crawlers, and the enforcement story separately.
The block matrix: 14 domains, 8 crawlers
The strictest doors belong to three domains. The New York Times, Amazon, and Yelp each block all eight named crawlers at the root, with explicit named blocks for nearly every token. At the other end, Best Buy takes a surgical approach: AI crawlers are not root-blocked, but a named block walls them off from cart, pricing, checkout, and API paths while leaving product discovery pages crawlable. Commerce browsing is welcome; agent checkout is not.
Everything in between is where the judgment lives. Forbes blocks the training crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) at the root but explicitly allows OAI-SearchBot, Claude-SearchBot, and Claude-User. The Guardian and Wired show the same pattern: ClaudeBot and the search crawlers blocked, but GPTBot's training crawler not named at all. The Verge is the oddest case in the set: it blocks ClaudeBot, PerplexityBot, and CCBot at the root, yet GPTBot gets an explicit Allow: /. Nobody in this industry has a single coherent policy; everyone is negotiating with each crawler vendor separately.
The most important finding is the gap. Walmart, Delta, Cars.com, Amtrak, and Chase show no named crawler blocks and no root-level wildcard block. Per The Verge's October 17 reporting, Delta prevents agents from booking tickets and Chase blocks agents from customer accounts, yet their robots.txt files say nothing about AI crawlers. The declaration layer and the enforcement layer are different machines.
The five layers of the lockout
Once you see the gap, the lockout stops looking like one wall and starts looking like five, each with a different cost to build and a different weakness. Here is the taxonomy, with a real example for each layer.
| Layer | What it does | Real example | How strong it is |
|---|---|---|---|
| 1. robots.txt declaration | Asks named crawlers to stay out. Voluntary; honest bots comply, bad ones ignore it. | nytimes.com: named Disallow: / for all eight crawlers we tested. | Weak. A request, not a lock. |
| 2. Terms-of-service prohibition | Makes agent access a policy violation, which platforms enforce selectively. | Yelp prohibits non-human traffic unless the agent pays for data licensing; eBay restricts unauthorized agents. | Medium. Enables enforcement, needs tech behind it. |
| 3. Anti-bot and IP enforcement | Rate limits, CAPTCHAs, behavioral analysis, and bot-mitigation vendors stop agents at runtime. | Delta rejects third-party agent bookings; users report agents failing on Cars.com and Amtrak. Security reviews have catalogued how agents get stopped at the account layer. | Strong, but blocks real users' tools too. |
| 4. Litigation | Sues the agent maker under computer-access laws. | Amazon sued Perplexity in November 2025 over its shopping agent; a district court granted a preliminary injunction on March 9, 2026, and the Ninth Circuit vacated it on August 4, 2026, ruling Amazon was unlikely to win on its computer-access claims. Amazon has amended its complaint and the case continues. | Uncertain. The courts just told Amazon its computer-crime theory was a misfit for user-directed agents. |
| 5. Paid licensing | Charges agents for structured access instead of blocking them. | Yelp's data licensing program: agents pay, and the company reportedly earns tens of millions of dollars a year from AI data deals. | Strongest business model, but only works if your data is worth buying. |
The banking layer deserves its own note. When JPMorgan Chase CEO Jamie Dimon was asked about agents during the bank's earnings call in mid-October, he said: "We're not going to allow just any outside agent to come in inside the company, because it's just too risky for clients and for JPMorgan." A spokesperson later confirmed to CNBC that the bank blocks outside AI agents from performing tasks for customers. That is a rational security call, and it sits in a different category from a publisher blocking a training crawler. An agent with account access is a fraud surface; our 2026 AI security review catalogued why account-layer automation is where the damage concentrates.
What blocking costs, and what licensing earns
This section is ByteHerald analysis, not reported fact. The numbers we measured are above; what follows is the economic reasoning they point to.
Blocking has two different price tags. Blocking a training crawler costs a publisher almost nothing: GPTBot and ClaudeBot collect data for future model training, so turning them away loses no traffic today and preserves the licensing value of the content. Blocking a search-index crawler is the expensive one. OAI-SearchBot builds the index ChatGPT cites; Claude-SearchBot feeds Claude's answers. Turn those away and your pages stop appearing inside AI answers, which is the fastest-growing discovery surface on the web. Our audit suggests publishers understand this: the most common posture in the data is blocking training bots at the root while leaving the search-index bots alone. Forbes states it outright with explicit Allow lines for OAI-SearchBot and Claude-SearchBot.
The commerce side of the ledger is where the real money sits. The upside of staying open to transaction agents is the agentic-commerce wave we have been tracking since last year: shoppers delegating purchases to software. Meta has chosen partnership over prohibition, with Muse shopping integrations at Walmart, Gap, Sephora, and Best Buy, and Shopify has opened its storefront network to agents. Every site that blocks agent checkout while Walmart leaves the door open is donating that delegation flow to a competitor. The Ninth Circuit's August ruling makes the legal bet shakier still: if user-directed agents are legally the user's own browser, the litigation layer may not hold, and sites that spent 2026 building walls could find the walls declared void.
Yelp's licensing model is the case study for the middle path. Rather than blocking agents or letting them in free, Yelp sells structured access. Reported earnings in the tens of millions of dollars a year from AI data deals suggest the data is worth more metered than walled. The catch is that licensing only works for data nobody can get elsewhere. A review corpus is licensable; a commodity product listing is not, because the agent can read it on a competitor's site.
The verdict
So is the lockout a rational defense of the business model, or a strategic error that hands agentic commerce to whoever stays open? The audit says it is both, depending on which layer you mean. Blocking training crawlers in robots.txt is rational, cheap, and clearly understood; that is why nearly every publisher in our set does it. Blocking transaction agents is the strategic error. The enforcement tools are expensive, the courts are skeptical of the legal theory, and the alternative is proven: Walmart partners, Shopify opens its doors, Yelp meters its data. The sites that wall off agent checkout are not protecting a business model; they are protecting a checkout page while the purchase itself moves to whoever lets the agent in.
Methodology and sources
Audit method: direct HTTP fetch of each domain's robots.txt on October 22, 2026, parsed for User-agent blocks; treatment recorded as explicitly blocked (named Disallow: /), explicitly allowed (named Allow), blocked by wildcard, or not blocked; schwab.com undetermined (bot-mitigation error page). Raw data: 240 rows (15 domains x 16 crawlers, expanded crawler list), saved as pillar-lockout-audit.csv.
First-party and primary sources: The Verge's October 17, 2026 reporting as originator of the block wave (Muse/Grok/Dots/Instinct blocks; Yelp licensing; Delta booking block; Chase and Schwab account blocks; Amazon blocking Muse since September 20; Meta retail partnerships; Shopify agent storefronts), corroborated via TechCrunch's October 6 report on Yelp's licensing program and CNBC's confirmation of Chase's account-access block; Ninth Circuit decision in Amazon v. Perplexity (opinion filed August 4, 2026, vacating the March 9, 2026 preliminary injunction); vendor crawler documentation for bot token functions. Economics and taxonomy are ByteHerald analysis, labeled as such.
Related ByteHerald coverage: our October 17 news timeline of the block wave; our 13-month tracker of agentic commerce, which follows the partnership side of this split; and our 2026 AI security report card on why account-layer agent access is where the damage concentrates.
Comments
Post a Comment