Outage often happens in Roblox due to several reasons. The outages interferes the players while playing the games, which is kind of thing that is hated by everyone. Actually, most of them are short outages. However, outage is still outage.
Talking about the outages, do you know the longest outage in the Roblox history? If you have no idea and are curious about it, for your information, it happened in 2021. If you are interested about this longest outage in the Roblox history, you can keep reading the post until the end.
On October 28, 2021 at 4:45 PM PST or at 12:45 AM UCT, the servers of Roblox started to experience system disruptions across the website and APIs. It resulted in the website services such as the Avatar Editor to unable to load. Not only that, it also caused huge chat issues. Besides, it turned the in experience performance to very slow. What made it worse is that a lot of users and then in the end all were kicked out of the experiences. Due to a number of issues, Roblox Studio also started to not function well.
It was such a mess. Everyone was everywhere on the social media and news sites expressing their frustration about the possibly losing their login streaks that experiences often to reward the visitors. Most of them were either sad, angry, or both as their efforts that they had done for months or even years turned into dust.
Knowing that there was something wrong, Roblox first action was to put the site under maintenance. The decision was made after 18 hours of the outage. It was the first time ever since 2019. Some developers and designers said that they lost Robux to ads placed in the 24 hours period preceding the outage, due to the decreased site traffic. The good news is that Roblox started to get better and reopened access to the website, allowing the first users to join experiences 63 hours after the start of the outage. Aside from these users, the others had to wait longer to gain access. On the last day of October 2021 at 5:45 PM PST, the website was fully reopened worldwide. In total, the outage lasted for 72 hours and 45 minutes or more than 3 days. This outage topped the record of the longest outage in the Roblox history.
People were so angry and disappointed at the engineers who failed to identify the issue for about 48 hours. It was such a long outage and caused so much people to lost money and effort. When the issue was everywhere, a number of influencers made mock videos about the posts posted on the official Twitter account of Roblox. Most people did not really know what to do so they just expressed their feeling through social media and so on through different ways.
At the time, a lot of people were believed that the outage was caused by the event known as Chipotle Borrito Maze event where visitors could get vouchers for free burritos. Some other also believed that it was due to the other traffic congestions. Besides, there were also some rumors spread through popular platforms like Reddit and Twitter saying that Roblox was shutting down due to a few reasons, including inappropriate content on their website and bankruptcy. However, all the rumors were quickly denied by the Roblox through its official Twitter account and later by the postmortem blog post. The outage was such a huge internet issue. Everyone was talking about it everywhere on Twitter, Youtube, Discord, and the other social media and news platforms.
Several hours after everything went back to normal, the CEO of Roblox named David Baszucki posted an apology on the Roblox Blog about the events that happened during the past 73 hours. At the end of the apology, he stated that Roblox would take action on preventing another major site wide failure like this in the future. Apart from that, he promised that Roblox will publish a post mortem with more details about the cause of the outage once they were done with the analysis. Besides, it was also guaranteed that no user persistence data was lost and would issue refunds to developers for ads that were not able to run due to the outage, as stated on the Developer Forum. Actually, the outage did not only harm the players and the developers but the Roblox itself. According to the report, Roblox lost more than $1.5 billion in the market cap on November 1, 2021, a period in which would take a minimum of $15 million in revenue.
While the outage was considered as finished, in fact, there were still some Catalog API errors 20 hours after the end of the outage. At the time, no one was found in the newly released accessories in the avatar shop. Actually, it only happened to a very small portion of users and the development or mechanics of experiences did not affected.
On November 3, 2021, Roblox started to give the refunds to the developers who were affected by the outage. As mentioned before, their ads could not run on the platform due to the outage. Five days after that, Roblox announced that they would compensate the developers for lost revenue due to the outage by estimating Robux earned from the marketplace and in experience transactions and also Premium Payouts.
Earlier this year (on January 20, 2022), a Roblox staff shared a post that explains that the outage was caused by a misconfiguration for server discovery and platform configuration. It is also stated that they need more time to diagnose the issue, that’s why the time required to do it was prolonged due to the Roblox’s heavy reliance on the impacted systems. In the same announcement, they explained that they have taken the important steps to prevent this kind of long outage form happening again and they are eager to expand to multiple datacenters in order to reduce the outages.
On my daily job, I am a software engineer, programmer & computer technician. My passion is assembling PC hardware, studying Operating System and all things related to computers technology. I also love to make short films for YouTube as a producer. More at about me…