Meta has revised its commercial terms after concerns that the previous wording could be interpreted as allowing it to collect content from news websites to train AI, so the language has been removed entirely.
Meta confirmed that this change does not alter how it uses web crawlers, and content owners can still use robots.txt to express their preferences as before.
News website owners became concerned after finding terms stating that Meta might use content from domains, web pages, or URLs to develop, improve, research, and train AI products.
Neiman Lab and James Grimmelmann, a professor of digital law at Cornell, viewed the wording as potentially allowing Meta to collect content from websites for free, possibly conflicting with instructions in robots.txt.
How the Previous Wording Disappeared from Meta’s Terms
| Factor | Before the change | After the update |
|---|---|---|
| Wording in the terms | Included language allowing content from domains, web pages, or URLs to be used to develop and train products | The wording was removed from the commercial terms page |
| Crawler clarification | Experts questioned Meta’s right to collect content | Meta stated that it had not changed how it uses web crawlers and had not granted new rights that conflict with robots.txt |
The key point is that Meta removed the disputed wording from its commercial terms, while the company insists that the update did not change how it uses web crawlers.
Where Meta Stands on the Use of Web Crawlers
Meta is a company with AI products, as well as other products, services, and features, so its commercial terms regarding the use of content from domains, web pages, or URLs to develop and train systems have drawn scrutiny.
The key issue is that Meta says the included provisions apply only to ads and business products. They do not mean that Meta has new rights to collect content from news websites. However, some experts interpreted the previous wording as potentially allowing scraped data to be used for free and conflicting with signals in robots.txt.
From the Initial Clarification to the Latest Terms Revision
After facing backlash, Andy Stone and Daniel Roberts denied that Meta had changed how it uses web crawlers, saying that this interpretation was inaccurate.
Meta later removed the wording from its commercial terms that was viewed as potentially allowing it to scrape website content to train AI. Roberts also clarified to The Verge that the terms applied only to ads and business products, while reaffirming that content owners could still use robots.txt to express their preferences as before.
What robots.txt Means After the Update
For website administrators, robots.txt remains a way to signal whether Meta is permitted to use web crawlers to access content. According to Meta’s clarification, this update did not change how its existing crawler operates.
In practice, website owners can therefore continue to control their preferences through robots.txt, and the new terms do not give Meta rights that override those signals.
How Experts Interpreted the Previous Wording
Neiman Lab and James Grimmelmann viewed the previous wording as potentially allowing Meta to use content collected from websites, including news sites, to train AI products without payment.
The concern was that the wording could be interpreted as giving Meta the right to use content even when a website had issued a prohibition through its robots.txt file.
How Fully Does Meta’s Clarification Address the Concerns?
Meta insists that it has not changed how it uses web crawlers and that content owners can still use robots.txt to communicate their preferences. This clarification helps ease concerns about new rights that conflict with website owners’ wishes.
However, removing all of the disputed wording still raises the question of why the terms needed to be revised if Meta truly did not change how it uses web crawlers.
Pros
- +Confirms that how Meta uses web crawlers has not changed
- +Reaffirms that content owners can still use robots.txt
Cons
- −Removing the previous wording still raises questions
- −The clarification does not explain the reason for the removal in detail
The wording of terms of use can immediately affect interpretation because the previous text stated that Meta might use content from domains, web pages, or URLs to develop and train AI, leading many parties to believe that Meta might use content from news websites.
This issue shows that websites and AI companies need to communicate clearly about content access. Ambiguous terms may cause content owners to misunderstand their rights relative to the company, even though Meta insists that website owners can still use robots.txt to express their preferences and that the wording change did not actually alter how it uses web crawlers.
What to Watch Going Forward
Going forward, it will be important to monitor how Meta uses web crawlers and whether it continues to respect website owners’ robots.txt signals as the company has stated.
Another point is how clearly Meta defines the terms for its advertising and business products, since the company says it has revised them to clarify this scope. Any further changes in wording should therefore be reviewed carefully.