• AI Work
  • Should You Block AI Crawlers? A Robots.txt Guide for Australia

Should You Block AI Crawlers? A Robots.txt Guide for Australia

Ammar Saleem profile picture
Robots.txt rules for blocking AI crawlers like GPTBot and Google-Extended

Table of Contents

Yes, you can block AI crawlers, mostly with a few lines in your robots.txt file. But for most Australian businesses, blocking all of them is a mistake.

The smart move is to split them up. Block the bots that only collect content to train AI models, if you want to. Keep the bots that let AI tools find and cite you in answers.

This guide is for business owners and marketers deciding what to do about ChatGPT, Gemini, Claude and other AI bots. You’ll get a plain-English list of bots, copy-ready robots.txt rules, and the new controls Google and Bing announced on 30 October 2026.

Can you block AI crawlers?

Yes. You block AI crawlers by adding rules for each bot’s name to your robots.txt file. Most major AI companies say they follow those rules.

Robots.txt is a small text file at yourwebsite.com.au/robots.txt. It tells bots which parts of your site they may visit. Think of it as a sign on your front gate.

Here’s the catch. A sign only works on people who read signs. Well-known bots from OpenAI, Google, Anthropic and Microsoft read it. Shady scrapers often ignore it.

If you want stronger control, you need a firewall too. More on that below.

Not sure what your robots.txt currently says? It’s one of the first things we check in our technical SEO audit checklist. If you’d rather hand it over, our SEO services team can set up AI bot rules for you.

What do AI crawlers do?

AI crawlers visit your website and copy its text. They do it for one of three jobs: training AI models, powering AI search answers, or fetching a page a user asked for.

Type of bot What it does Example bots Block it?
Training bot Collects content to teach future AI models GPTBot, ClaudeBot, CCBot Your choice
AI search bot Finds pages to show and cite in AI answers OAI-SearchBot, PerplexityBot, Claude-SearchBot Usually keep
User-triggered fetcher Opens a page when someone asks the AI to ChatGPT-User, Perplexity-User Hard to block with robots.txt

The difference matters. Blocking a training bot keeps your content out of future models. Blocking a search bot can keep you out of today’s AI answers, and the leads that come with them.

Watch out: Many “block all AI” plugins and copy-paste lists block every bot at once. That can quietly remove your business from ChatGPT search. Check what a plugin does before you switch it on.

Is ChatGPT a web crawler?

ChatGPT isn’t one crawler. OpenAI runs several bots, and each one has its own name and job.

According to OpenAI’s crawler documentation:

  • GPTBot collects content that may be used to train OpenAI’s models.
  • OAI-SearchBot finds pages for ChatGPT search. OpenAI says sites that block it won’t be shown in ChatGPT search answers.
  • ChatGPT-User visits a page when a user asks ChatGPT to. Because a person started the action, OpenAI says robots.txt rules may not apply.
  • OAI-AdsBot checks landing pages for ChatGPT ads.

OpenAI also says changes to robots.txt can take about 24 hours to affect search results.

So “block ChatGPT” can mean four different things. Decide which job you’re blocking first.

If you want ChatGPT to recommend you, not your competitor, you need to stay visible. Our guide on why AI recommends your competitor instead of you covers the other half of that work.

Is there a robots.txt for AI?

There’s no separate file. You use the same robots.txt and add a group for each AI bot by name.

Here are three ready-made setups. Pick the one that fits your goal.

Option 1: Block AI training, stay in AI search (what we recommend for most businesses)

# Block AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow AI search and citations
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Option 2: Allow everything (best for visibility)

Do nothing. If your robots.txt has no AI rules, AI bots can crawl your site.

This suits most service businesses. Your pages are marketing. You want them read, quoted and linked.

Option 3: Block all AI bots (rarely the right call)

Add Disallow: / groups for every AI bot, including the search bots. Only do this if your content is your product, like paid research or a members’ course.

Pro tip: Google-Extended only controls whether Google uses your content to train and ground its Gemini models. It doesn’t affect your Google Search rankings. Google handles its AI features in Search separately.

What’s changing in 2027?

On 30 October 2026, Google announced plans to support a shared AI training control in robots.txt. It uses one rule, User-agent: __usage_ai-training, that many AI companies can read. You can read Google’s announcement on Search Central.

Microsoft, OpenAI and Anthropic also backed it. Reports say Microsoft starts honouring it in February 2027, and Google plans to in early 2027.

Bing went further. It now has separate settings for AI training and AI grounding (using your page to build an answer in Copilot or Bing). Neither setting affects normal Bing rankings, as Search Engine Land reported. If you used Bing’s old NoArchive or NoCache tags, review them before 1 March 2027.

Not sure which AI bots are hitting your site? We’ll check your robots.txt and server logs and tell you what to block and what to keep. Book a free strategy call.

Does robots.txt actually work?

Robots.txt works on well-behaved bots, which includes the big AI companies’ named crawlers. It doesn’t work on bots that choose to ignore it.

It also only controls crawling, not what happens to content a bot already has. If an AI company crawled your site last year, a new rule today won’t pull your pages out of an existing model.

Robots.txt works best when:

  • You use the exact bot name from the company’s own docs
  • Each bot has its own User-agent group
  • The file sits at the root of your domain
  • You test it in Search Console’s robots.txt report

Can robots.txt be ignored?

Yes. Robots.txt has no power to force anything. Any bot can ignore it.

Google’s Gary Illyes said in October 2026 that AI has made bot impersonation worse. That’s when a scraper pretends to be Googlebot to slip past your rules. Search Engine Roundtable covered the interview on 30 October 2026.

If you need real blocking, add a second layer:

  1. Use a firewall. Tools like Cloudflare can block known AI bots and suspicious traffic at the server level.
  2. Verify “Googlebot” visits. Real Googlebot comes from Google’s published IP addresses. Fakes don’t.
  3. Rate-limit heavy bots. Slow down any bot that hits hundreds of pages a minute.
  4. Watch your server logs. Look for sudden spikes from unknown user agents.

Watch out: Don’t block real Googlebot or Bingbot by mistake. A bad firewall rule can drop your whole site out of search.

Is AI scraping illegal?

In Australia, AI scraping isn’t automatically illegal, but copying and reusing your content can still breach copyright. The law here is still catching up.

Here’s where things stand as of November 2026:

  • No AI training exception. In October 2025, Attorney-General Michelle Rowland ruled out a text and data mining exception to the Copyright Act. That means there’s no special free pass for AI training in Australian law.
  • Licensing talks are ongoing. The government is consulting on how AI companies should license content.
  • Robots.txt isn’t a contract. Ignoring it isn’t a crime in itself. But your website terms of use can add weight.

We’re marketers, not lawyers. If you think a company has copied your content, get advice from an Australian IP lawyer.

Key Takeaways

  • Robots.txt is a polite request. Big AI companies follow it. Scrapers often don’t.
  • Training bots (like GPTBot) and search bots (like OAI-SearchBot) are different. Block one without the other.
  • Blocking AI search bots can remove you from ChatGPT and other AI answers.
  • A new shared AI training opt-out is coming in 2027, backed by Google, Microsoft, OpenAI and Anthropic.

Final Remarks

You can block AI crawlers with robots.txt, but it’s a request, not a lock. For most businesses, the safest setup is to block training bots if you want to and keep AI search bots, so AI tools can still find and cite you. Don’t confuse this with llms.txt, which can’t block anything. Our guide to llms.txt SEO explains why.

Adcept Marketing can review your robots.txt, check which bots visit your site and set rules that protect your content without hurting your leads. Book a free strategy call to get started.

FAQs

Will blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI’s training bot. Blocking it keeps your content out of future model training. ChatGPT search uses OAI-SearchBot instead. As long as you allow OAI-SearchBot, your pages can still appear and be cited in ChatGPT search answers. Check your robots.txt after any plugin update.

Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a separate control from Googlebot. Blocking it doesn’t change how Google crawls, indexes or ranks your site in normal search. It only tells Google not to use your content for training and grounding its Gemini models. Google handles AI features in Search separately.

How do I check if AI bots are crawling my site?
Look at your server logs or hosting analytics for user agents like GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot. Cloudflare and some hosts show AI bot traffic in a dashboard. GA4 won’t show it, because most bots don’t run the tracking code.

Should a small business block AI crawlers?
Usually not all of them. Most small businesses want to be found and recommended, and AI answers are a growing source of enquiries. Blocking training bots is a fair choice if it matters to you. Blocking search bots usually costs more than it saves.

Can AI crawlers slow down my website?
Yes, heavy crawling can slow a small or shared server. If bots cause slowdowns, rate-limit them or block the worst offenders at your firewall. That’s often better than blocking everything in robots.txt, because it keeps your useful AI visibility. Your host can usually help set this up.

Can an agency set up AI bot controls for me?
Yes. An agency can review your robots.txt, firewall rules and server logs, then set rules that match your goals. At Adcept Marketing, a Melbourne digital marketing and AI automation agency, we do this as part of our SEO work. We review them again when bot names change.

Picture of Ammar Saleem
Ammar Saleem

Ammar Saleem is a Copywriter at Adcept Marketing who’s spent the last five years helping brands turn smart automation into real results. From search engine optimization to sales funnels and landing pages, he creates content that connects with audiences and drives action. Ammar Saleem has a talent for breaking down complex ideas into clear, practical messaging and he loves helping businesses simplify their marketing so growth feels effortless.

Table of Contents

AI-powered marketing. Eight channels. One customer journey.

Contact Now ›

Related Insights

Book a 15-min demo

Pick a time that suits you. We'll show you a setup with your own hours and fees, and answer your questions.


$599/month + GST · No setup fee · No lock-in

Ask about direct booking

Tell us what software you use and we'll come back with what's possible and a quote.

See how Riley fixes it

We'll walk you through it with your numbers in a 15-minute demo.

Your estimate
$5,200
/month

20 missed calls a week, $400 average first visit.

See how Riley fixes it

We'll walk you through it with your numbers in a 15-minute demo.

Get a call back

We'll call to set up a 15-minute demo at a time that suits you.