Meta Commercial Terms October 30: Website Crawling and What to Do

Updated 16 October 2026. Meta's updated Commercial Terms take effect on 30 October 2026, and this article covers what changes and what website owners can do before then.

From 30 October 2026, Meta's updated Commercial Terms let Meta crawl websites tied to the businesses that use its commercial tools. In the new text, you authorize Meta to use scrapers, bots and crawlers on the domains, pages and URLs associated with your products or services, and to use that content to develop, improve and train Meta's products. The preview text names no opt-out and does not mention robots.txt.

What you can do: decide crawler by crawler, add robots.txt rules for Meta's AI crawlers, block at the server if you need a hard stop, check your logs, and ask a lawyer whether blocking fits the terms you accept. Nobody we read has answered that last question. This article is information, not legal advice.

Related: Forward Domain Email to Gmail: 4 Ways, Catch-All and Send As

What changes on 30 October 2026

Meta's Commercial Terms page carries a banner saying the update takes effect on 30 October 2026, with the new text on a preview page. The key part is Section 1, Licenses and Rights You Provide Meta:

• Crawling. You authorize Meta, or anyone acting for it, to use automated tools such as scrapers, bots, crawlers and user agents to access, index, cache, analyze, collect, retrieve or crawl the domains, pages or URLs "associated with your products or services". Meta can also pull updates from them from time to time.
• Use, including training. Meta may use content from those domains to provide, develop, improve, research, train and maintain the integrity of the Meta products, services and features offered under the Commercial Terms.
• Wide reach. The Commercial Terms pull in other documents, including the Self-Serve Ad Terms, Business Tools Terms, Seller Agreement, Meta Business Agents and Platform Terms of Service, and the Meta AI Business Assistant Supplemental Terms.

Advertisers have seen similar language before. Section 6 of the current Self-Serve Ad Terms, effective 22 August 2025, already lets Meta access, index, cache, analyze or crawl content on an advertiser's domains for ads and business products, research and training. Those terms are also being updated on 30 October 2026. Our reading of the two documents, not a statement from Meta: the crawl grant now reaches business users of Meta's commercial tools, not only advertisers.

What is confirmed and what is not

On enforcement, Nieman Lab reported, as summarized by AI Weekly, that media lawyer Matt Topic said it is unclear whether a court would enforce the clause, but publishers should assume it would. James Grimmelmann, also quoted in that report, called the terms very broad.

Meta's crawlers and what each one does

Meta's crawler documentation lists five crawlers. In your logs, each user agent string starts with the token shown here, followed by a version number such as 1.1.

Meta says robots.txt changes can take up to 24 hours to apply. It suggests allowlisting by user agent or IP address, but the crawler page publishes no IP list.

What a business website owner can do

1. Add robots.txt rules

To stop the AI training and AI search crawlers but keep ads crawling and link previews, add this to the robots.txt file at your domain root:

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-WebIndexer
Disallow: /

To keep a crawler out of some folders only, list the paths instead:

User-agent: Meta-ExternalAgent
Disallow: /members/
Disallow: /downloads/

Meta's own example writes the token in lowercase, as meta-externalagent, which is how it appears in your logs. The robots.txt standard, RFC 9309, says crawlers must match the user agent line without regard to case, so either form works. Under the same standard, a crawler with its own group follows that group instead of the User-agent: * group, so repeat any shared Disallow lines in each Meta group. And robots.txt is a request, not access control.

2. Block at the server for a hard stop

A server rule refuses the request instead of asking politely. On nginx, inside the server block for your site:

if ($http_user_agent ~* "(meta-externalagent|meta-webindexer)") {
    return 403;
}

The ~* operator makes the match case-insensitive. On Apache, a CDN or a web application firewall, write the same rule on the user agent field. Two limits apply: anyone can fake a user agent, so this stops only honest requests, and a hard block raises the same contract question as robots.txt.

3. Watch your server logs

Count visits from each Meta crawler before and after 30 October, and again a day after any robots.txt change. On a Linux server running nginx:

for ua in meta-externalagent meta-webindexer meta-externalads meta-externalfetcher facebookexternalhit; do
  printf '%s: ' "$ua"
  grep -ci "$ua" /var/log/nginx/access.log
done

Change the path to match your access log. If a crawler using a Meta name still fetches blocked pages a day after a change, check its IP addresses: the name may be fake.

4. Move sensitive content behind a login

Crawlers read what is public. Paid articles, dealer price lists and member files belong behind a login, whatever you decide about robots.txt. See our guide on how to secure a web application, and our SEO checklist for a new website for the wider setup.

5. Get a legal review

Before 30 October, ask a lawyer who knows your market these questions:

• Which of our domains count as associated with our products or services?
• Do we accept the new terms just by using Meta's business tools after 30 October?
• Does blocking Meta's crawlers conflict with the license we grant?
• Does content we license from others, or data about our customers, limit what we can grant?

For background on the legal side of crawling, see is web scraping legal.

Trade-offs: visibility in Meta's AI vs reuse of your content

Blocking has a cost. Each crawler supports something you may want:

The bigger trade-off is AI search. Meta says Meta-WebIndexer helps Meta AI cite and link to sites, so blocking it may cost you mentions when customers ask Meta AI about your field. Our generative engine optimization guide explains how answer engines choose sources. Publishers and course creators, whose content is the product, may block both the training and AI search crawlers. A local service business that wants to be found may block only the training crawler, or nothing.

Checklist before 30 October

1. List the Meta business accounts and tools you use (Facebook, Instagram, WhatsApp, ads) and the domains linked to each.
2. Read Section 1 of the new Commercial Terms, and Section 6 of the Self-Serve Ad Terms if you advertise.
3. Check your logs for each Meta crawler, then decide per crawler: allow, limit to some folders, or block.
4. Update robots.txt, wait 24 hours, then check your logs again.
5. Add server or firewall rules if you need a hard stop.
6. Move paid and sensitive content behind a login.
7. Get legal advice on blocking, consent and which domains are covered.
8. Record the decision in your company AI policy; see our AI policy template for business.
9. Re-check Meta's terms and crawler page after 30 October.

We can handle the technical side as part of our web development work: the log check, robots.txt and server rules, and moving sensitive pages behind a login. The contract question stays with your lawyer. Get in touch if you want help before the deadline.

Meta commercial terms FAQ

What changes in Meta's Commercial Terms on October 30, 2026?

The updated terms include a license in Section 1. Businesses that use Meta's commercial tools authorize Meta, or anyone acting for it, to crawl the domains, pages and URLs associated with their products or services, and to use that content to provide, develop, improve, research and train Meta's products. The new text is on Meta's preview page, and the update takes effect on 30 October 2026.

Can I opt out of Meta crawling my website?

The preview text describes no opt-out and does not mention robots.txt. Nieman Lab reported, as summarized by AI Weekly, that there is no opt-out and that continuing to use Meta's business tools after 30 October counts as consent. Meta's crawler docs still explain how to block each crawler with robots.txt, but whether doing so conflicts with the new terms is unclear.

Does blocking Meta's crawlers break the new terms?

Nobody we read has answered this, including Meta. The new terms grant Meta a right to crawl associated domains, and blocking its crawlers may or may not conflict with that grant. Ask a lawyer who knows your market before 30 October, and keep a dated record of every robots.txt or server change you make.

Which user agent blocks Meta's AI training crawler?

Meta-ExternalAgent is the crawler Meta uses for purposes such as training foundation AI models. Add a group with User-agent: Meta-ExternalAgent and Disallow: / to your robots.txt to block it. Meta's own example writes the token in lowercase, and the robots.txt standard says matching ignores case. To also stay out of Meta AI search results, block Meta-WebIndexer.

Will blocking Meta's crawlers break Facebook link previews?

Only if you block FacebookExternalHit. That crawler fetches the title, description and image that Meta's apps show when someone shares your link, so most sites should keep it. Based on Meta's descriptions, blocking Meta-ExternalAgent or Meta-WebIndexer does not affect previews. Note that FacebookExternalHit may bypass robots.txt for security or integrity checks, and Meta-ExternalFetcher may bypass it for fetches a user requests.

Does this affect me if I do not run Facebook ads?

Possibly. The Self-Serve Ad Terms already gave Meta a similar crawl right over advertisers' domains. In our reading, the new Commercial Terms reach business users of Meta's commercial tools more broadly, and Nieman Lab reported, as summarized by AI Weekly, that they cover business accounts on Facebook, Instagram and WhatsApp. If your business uses any of those tools, read the new text and get legal advice.

Sources

• Meta Commercial Terms (current version, with the update notice)
• Meta Commercial Terms preview (effective 30 October 2026)
• Meta Self-Serve Ad Terms
• Meta for Developers: web crawlers
• RFC 9309: Robots Exclusion Protocol
• nginx documentation: ngx_http_rewrite_module