A Guide to Policy-Making for GPTBot & Web Scraping by Generative LLMs

In 2023, the question was simple enough to answer in a hallway conversation: "Should we block GPTBot?" You added one line to robots.txt, felt vaguely responsible, and moved on.

That question is now obsolete — not because the answer changed, but because the question was wrong. There is no single bot. There is no single decision. And the line in robots.txt you added two years ago is almost certainly costing you visibility in the exact channel that is replacing the search traffic you used to depend on.

As of 2026, your website is being crawled by more than fifteen distinct AI agents, each with a different purpose, a different legal status, and a different effect on your business. Some are harvesting your content to train models you'll never see a dollar from. Others are fetching your pages in real time to answer a prospect's question — and citing you as the source. Treating those two things the same way is the most expensive mistake we see companies make right now.

This is a guide to building an actual policy: one that distinguishes between the bots, maps to your business model, accounts for the legal weather, and treats your AI-crawling rules as the strategic lever they've quietly become.

The Problem Changed While You Weren't Looking

The scale of the shift is hard to overstate. According to Cloudflare's 2025 analysis (From Googlebot to GPTBot, July 2025), GPTBot's crawl traffic grew 305% year over year from May 2024 to May 2025, and GPTBot's share of crawler activity surged from roughly 5% to 30% — while Meta's crawler, Meta-ExternalAgent, entered the picture at 19% of crawler share. This is no longer a fringe consideration on the edge of your infrastructure. For some site categories it's the dominant traffic pattern.

If you run anything resembling a storefront, the numbers are starker. a 2026 report from Retail Economics, AWS, Botify, and DataDome found that AI bots crawl retail sites 198 times more than Google does, and that retail saw a 5.4x increase in AI bot traffic over the course of 2025. That is real bandwidth, real compute, and a real line item — traffic that delivers no human to your funnel unless you've deliberately set up the rules to make it count.

Meanwhile, the standard everyone reaches for to control this — robots.txt — was designed in 1994 for cooperative search engines. It is a polite request, not a fence. A Duke University study (May 2025, via arXiv) found that fewer than 40% of AI bots re-check robots.txt within 168 hours, and that compliance drops as the rules get more restrictive. The tool most companies rely on as their entire AI-crawling strategy is leaky by design, and leakier the harder you lean on it.

So the modern policy has to do three things at once: understand the bots, respect the legal and infrastructure realities, and make a deliberate business choice. Let's take them in order.

Know Your Bots: Training vs. Retrieval

The single most useful distinction — the one that collapses an intimidating list of fifteen-plus user agents into a decision you can actually reason about — is training versus retrieval.

Training crawlers harvest your content to build or fine-tune large language models. The model absorbs your words; your specific page does not get cited back to a user, and you generally get nothing in return. These are the bots most companies have a legitimate, defensible interest in blocking. The major ones:

Retrieval and search crawlers fetch your pages in real time (or near-real-time) to answer a specific user's question, and they typically cite or link back to you. When someone asks ChatGPT or Perplexity a question and your page shows up as the source, that's a retrieval crawler doing its job. Blocking these is how you make yourself invisible in AI search. The major ones:

Note the deliberate separation inside single vendors. Anthropic now splits its activity across three distinct bots — ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live fetch) — precisely so that site owners can say "no" to training while keeping the door open to citation. Google did the equivalent with Google-Extended, decoupling its AI-training crawl from Googlebot so blocking one doesn't tank your normal search ranking. The vendors built the granularity for you. Most companies haven't used it.

This is the crux: a blanket Disallow aimed at "AI" usually blocks both columns. You wanted to stop giving away training data. What you actually did was also delete yourself from the answer engines your prospects are increasingly starting with. Always verify current user-agent strings against the operators' own documentation — OpenAI publishes its bot list, and Search Engine Journal maintains a regularly updated master reference — because this roster changes faster than any blog post can keep up with.

The Legal Weather You Can't Ignore

You don't need to become a copyright lawyer. You do need to know which way the wind is blowing, because the legal landscape is now a genuine input to the policy decision — not background noise.

A few markers, presented as reported and subject to change:

• The New York Times v. OpenAI litigation has been working through discovery, with a summary-judgment decision reportedly expected around mid-2026. The outcome is not settled, and we'd flag any confident prediction as speculation — but the case is the closest thing the industry has to a bellwether on whether training on copyrighted web content is fair use.
• Anthropic reached a reported $1.5 billion settlement in copyright litigation (Bartz v. Anthropic), which press coverage (NPR / Copyright Alliance, September 2025) framed as roughly $3,000 per book across approximately 500,000 titles. Whatever the final accounting, it establishes that training-data disputes now carry ten-figure financial stakes.
• Perplexity has reportedly faced suits from multiple parties including Reddit, Amazon, and the New York Times over its crawling and content-use practices.
• On the regulatory side, the EU AI Act's general-purpose AI obligations entered force on August 2, 2025, with full high-risk enforcement reportedly arriving August 2, 2026 (per IAPP/EU AI Act guidance). For companies touching EU users, the Act's transparency requirements around training data are a compliance timeline you can plan against.

There's also a quieter signal worth more than any single headline. The Data Provenance Initiative reported that 45% of the C4 dataset — one of the most widely used web-text training corpora — is now restricted by terms-of-service crawling restrictions. The open data commons that early models were built on is closing fast, as thousands of site owners reassert control. You're not contemplating an exotic move. You're contemplating joining a migration already well underway.

The practical takeaway for a CEO or general counsel: a documented, intentional AI-crawling policy is itself a risk-management artifact. If your content is sensitive — a law firm's published analyses, a medspa's patient-education material, proprietary research — "we never decided" is a worse position than either "we allow it" or "we block it." Decide on purpose, write it down, and treat this section as a flag for your own legal review rather than legal advice.

The Three Policy Archetypes

Across the web infrastructure we manage for clients in law, healthcare, professional services, SaaS, and non-profits, nearly every workable AI-crawling policy collapses into one of three archetypes. Find yours.

1. The Open Door — Allow training and retrieval

You want maximum reach and your content's value is in distribution, not exclusivity. A service business whose blog exists to generate inbound leads usually lives here: being quoted by ChatGPT is free top-of-funnel. The risk is real but acceptable — you're trading training data for visibility, and visibility is the point.

Fits: most service businesses, agencies, consultancies, and SMBs whose content is marketing, not product.

2. Selective Access — Block training, allow retrieval

The default we recommend for most companies that have anything to protect but still want to be found. You stop feeding model training while remaining fully citable in AI answers. This is the archetype the vendors' bot-splitting was built to enable, and it's where the training-vs-retrieval distinction earns its keep.

Fits: publishers with original IP who still want traffic, B2B companies with proprietary frameworks, and most ICP-4 organizations weighing protection against GEO.

3. The Walled Garden — Block everything

Your content is the product, or it's legally sensitive enough that the safest posture is no AI access at all. Subscription publishers, research firms selling reports, and businesses under strict confidentiality obligations belong here. Accept the cost clearly: you will be largely absent from AI-generated answers, and given robots.txt's weak compliance, enforcement requires infrastructure-level controls, not just a text file.

Fits: paywalled publishers, premium research, and confidentiality-bound practices.

The point of the archetypes isn't to pick the "best" one. It's to force the conversation that most companies skip: what is our content actually for, and what are we willing to trade?

Technical Implementation

Once you've chosen an archetype, implementation is concrete. Here's the spine of a robots.txt for Selective Access — block training, keep retrieval — as an illustrative pattern (always validate current user-agent strings against operator docs before shipping):

# Block training crawlers

User-agent: GPTBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: CCBot

Disallow: /

User-agent: Meta-ExternalAgent

Disallow: /

# Allow retrieval / search crawlers (omit or explicitly Allow)

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: PerplexityBot

Allow: /

But remember the Duke finding: robots.txt is a request, not enforcement. If your archetype is Selective Access or Walled Garden and the stakes are real, you need a layer that can actually say no. This is where edge infrastructure matters. Cloudflare's AI Crawl Control lets you block or challenge AI bots at the network edge, where a Disallow becomes an actual block. Cloudflare reported in 2025 that 3.8 million domains were using its managed robots.txt to block AI training — automated, maintained, and enforced rather than hand-edited and hoped over.

Two emerging standards are worth putting on your roadmap now, even if they're not load-bearing yet:

• The IETF AIPREF working group is developing a standardized Content-Usage HTTP header — a successor to robots.txt purpose-built for AI preferences. Adopting it early is cheap insurance.
• The W3C TDM Reservation Protocol (TDMRep) gives you a machine-readable way to reserve text-and-data-mining rights, which maps directly to the EU's opt-out framework. If EU compliance is on your horizon, this is the technical hook.

And llms.txt — a proposed standard, distinct from robots.txt — lets you publish a curated, AI-friendly map of your most important content. It's less about blocking and more about shaping what AI systems understand about you. For Open Door and Selective Access companies pursuing visibility, it's a lever in the other direction.

The GEO Imperative: Why Blocking Everything Is the Riskiest Move

Here's the strategic point most security-minded checklists miss entirely, and it's the one we'd put in front of any CTO about to reflexively block all AI traffic.

The channel you're protecting your content from is the channel replacing the one you've optimized for two decades. Press Gazette reported (November 2025) that Google search traffic to publishers dropped 33% globally — 38% in the US — in 2025. At the same time, the Previsible AI Traffic Report found AI-referred sessions jumped 527% year over year in the first five months of 2025. Traditional organic search is shrinking; AI-referred discovery is exploding from a small base.

Block the retrieval crawlers and you optimize yourself for a declining channel while disappearing from the growing one. That's the trap. The retrieval bots are how you show up when a prospect asks an answer engine the question your content was written to answer. Your robots.txt policy is now, directly, a GEO (Generative Engine Optimization) decision — the single setting that determines whether you're citable in the places people increasingly look first.

This is also why the SMB anxiety is well-founded but often misdirected. A Clutch.co survey (2025) found 43% of SMBs report AI threatens their content ownership and IP. The fear is real. But the answer for most of them isn't the Walled Garden — it's Selective Access. Stop the training harvest you get nothing for; keep the citation traffic that brings you customers. The companies that conflate the two and slam every door are protecting an asset by destroying its distribution.

Building Your Policy: A Decision Framework

Pull it together into something you can run through in a meeting:

1. Classify your content. Is it marketing (its value is reach) or product (its value is exclusivity)? This is the single biggest input and it splits service companies from publishers almost cleanly.
2. Pick your archetype. Open Door for distribution-driven content. Selective Access as the sensible default for most. Walled Garden only when content is the product or confidentiality demands it.
3. Weigh your legal exposure. Sensitive verticals — law, healthcare, regulated research — should bias toward more restriction and should route this past counsel. Tie it to your EU AI Act timeline if you touch EU users.
4. Account for GEO. Before you block any retrieval crawler, ask whether you're willing to be absent from AI answers in your market. For most service businesses the honest answer is no.
5. Implement in layers. robots.txt for the cooperative bots, edge enforcement (Cloudflare AI Crawl Control or equivalent) for the rest, and llms.txt if you're leaning into visibility. Adopt AIPREF and TDMRep as they mature.
6. Write it down and revisit quarterly. This landscape moved twice while you read this article. A policy you set and forget is a policy that's already stale.

The cross-industry pattern we keep seeing is this: the companies in real trouble aren't the ones who chose wrong. They're the ones who never chose — who blocked GPTBot in 2023, forgot about it, and quietly fell out of the answer engines their competitors now own. The decision is no longer binary, and it's no longer optional. It's a configuration of your most strategic web infrastructure, and it deserves the same deliberation you'd give any other lever on growth.

If you'd rather not navigate fifteen user agents, three emerging standards, and a moving legal target on your own, that translation from technical configuration to business policy is exactly the kind of work we do — turning a robots.txt file into a decision your leadership team can actually stand behind.

Building or revisiting your AI-crawling policy? Facet Interactive manages web infrastructure across law, healthcare, SaaS, and professional services — and we help companies turn the GPTBot question into a strategy. Let's talk.