While everyone was busy trying out New Twitter or tweeting about how they want New Twitter, Twitter itself was doing something secret behind the scenes. The startup quietly flipped the switch on an entirely new backend for their search, they reveal in a blog post today.
“One of our main goals, but also biggest challenges, was a smooth switch from the old architecture to the new one, without any downtime or inconsistencies in search results,” they write in the post. Mission: accomplished, it seems, as no one outside of Twitter even seemed to be aware that they switched anything.
Twitter notes that they had to build this new backend because they were still using the search technology that they acquired in the Summize deal. Obviously, that tech was great at the time, but Twitter was much smaller at the time of that deal; they’ve grown massively since then. “Scaling the old MySQL-based system had become increasingly challenging,” they note.
So what is this new search? “Since we love Open Source here at Twitter we chose Lucene, a search engine library written in Java, as a starting point,” Twitter notes in the post. But they say that they had to modify it give their demands for real-time search. What type of demands? These types of demands:
Our demands on the new system are immense: With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines. As we want the new system to last for several years, the goal was to support at least an order of magnitude more load.
They also have a goal of indexing a tweet after it’s tweeted in less than 10 seconds.
Twitter says that any custom work they did/do on Lucene will be put back into the open source.
Join 10k+ tech and VC leaders for growth and connections at Disrupt 2025
Netflix, Box, a16z, ElevenLabs, Wayve, Hugging Face, Elad Gil, Vinod Khosla — just some of the 250+ heavy hitters leading 200+ sessions designed to deliver the insights that fuel startup growth and sharpen your edge. Don’t miss the 20th anniversary of TechCrunch, and a chance to learn from the top voices in tech. Grab your ticket before doors open to save up to $444.
Join 10k+ tech and VC leaders for growth and connections at Disrupt 2025
Netflix, Box, a16z, ElevenLabs, Wayve, Hugging Face, Elad Gil, Vinod Khosla — just some of the 250+ heavy hitters leading 200+ sessions designed to deliver the insights that fuel startup growth and sharpen your edge. Don’t miss a chance to learn from the top voices in tech. Grab your ticket before doors open to save up to $444.
And the new system, which again, has been rolling out for a few weeks now, seems to be working. Twitter estimates that they’re only using about 5 percent of the available backend resources. They say the new indexer could also run 50 times more tweets per second than they currently get. In other words, it will scale.
More: A Billion Queries A Day Later, Twitter Finally Working On Search Again
