X updates its terms to ban crawling and scraping

Elon Musk-owned X, formerly Twitter, has updated its terms of service to prohibit scraping and crawling — likely to fend off any AI models training on its data.

The new terms, which are effective from September 29, ban any kind of scraping or crawling without “prior written consent.”

NOTE: crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited.

The previous version of the terms allowed crawling in accordance with robots.txt.

“NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without our prior consent is expressly prohibited,” it read.

In the last few months, Twitter has also altered its robots.txt file — a file that gives instructions to robot crawlers about what parts of the site they are permitted to visit — to remove instructions for all crawler bots apart from Google.

In 2015, Twitter confirmed that it had a firehose deal in place with Google to surface tweets in search results. It is not clear if the nature or terms of that deal have changed under the new management.

We have reached out to Google to understand if the search giant has an agreement in place. We will update the story if we hear back.

X has also altered the robots.txt file to disallow crawlers to get information such as likes and retweets related to specific posts. It also bars robots from looking at an account’s likes, media and photos.

In June, the social network briefly stopped logged-out users from looking at posts. Days later, it removed the login requirement to view tweets. At that time, Musk had said that it was a temporary measure because the site was getting “data pillaged so much that it was degrading service for normal users.”

Musk has been against companies scraping Twitter/X data to train AI models. In April, he threatened to sue Microsoft for illegally using the social network’s data to train AI models. In July, he filed a scraping lawsuit against unknown companies.

Earlier this month, X changed its privacy policy to state it might use public data to train AI models. Musk has previously noted during a Twitter space that xAI, a company founded in July, would use public data such as tweets to train its models. X’s new privacy policy also has provisions for the collection of biometric data of users along with education and job history.