Note: Keep your data available with cloud hosting, which has more redundancy than any other hosting solution on the market today.
One data center gives the Internet a rare glimpse at the behind-the-scenes scramble to keep a data center operational during Hurricane Sandy.
Squarespace was founded in 2003 in New York City. The company wanted to provide web publishing services to its customers, including blogging or content management tools. Additionally, the service also provides a file server service. Despite the company’s growth, they’ve retained their Manhattan address, and you can guess where this story goes next. When Sandy struck, Squarespace threw up a quick blog to update its customers on the data center status throughout the storm. What resulted was an interesting inside look into what keeps a data center ticking and how to navigate through the increasing stages of what was Hurricane Sandy.
First, on Tuesday of this week, an unidentified Squarespace employee posted a blog update letting users know that their data center named Peer 1, in Lower Manhattan, lost power at 4:30pm EDT on the 28th. Thankfully, Peer 1 had generators in place in case of emergency which began powering the facility when grid power went down. The company knew they had enough generator fuel to power the servers for three or four days.
On the 29th, at 8:30pm EDT, the lobby of the building the data center is housed in began to flood. Two hours later, the majority of the Peer 1 basement was underwater. By 5am EDT on the 30th, the fuel pumps and tanks were underwater and whatever was currently in the generators was all that would be getting to the generators. At 8am EDT, it was determined that the generators would go for another three or four hours max. After this update, the blog post apologized for the downtime everyone would experience starting about noon on the 30th.
At 12:45 pm EDT, another blog post was published explaining that fuel readings have been erratic, since the fuel system is underwater and not functioning entirely up to par (hence a limited amount of fuel getting to the generator, and giving Squarespace a looming downtime deadline). The company had decided that at the last possible moment, their engineers would shut off their systems entirely, which would be better for the servers instead of suffering an unexplained shutoff due to a lack of power. Their last prediction was downtime was imminent within the hour. The sliver of hope was that a fuel truck with technicians willing to help them figure out their situation was en route.
The next update came about an hour later, as the unidentified blogger explained that there was an operational fuel pump in the building they could potentially use with their now delivered fuel, but the building was damaged so it was a risk no one wanted to take. Five hours after that, another blog was posted reporting that Squarespace data center engineers had been carrying fuel to their generators (a difference of seventeen floors) to keep the servers running. Although these guys are our competitors, we have to give credit where credit is due; that is incredible dedication.
A blog dated as 1 am EDT on Halloween stated that they were prepared to keep fueling the generators through the night, as a maintenance crew was pumping water out of the basement. The hope was that with the fuel pump exposed, they would be able to fix it, and automate the fueling process so the generator could run without human interaction, for days if necessary. In the meantime, a 6 am EDT update specified that the three companies that had vested interests in the building, Peer 1, Fog Creek, and Squarespace, had employees arrive at the building to help with the fueling efforts, including making an assembly line spanning seventeen floors to keep the generators running.
By 1 pm EDT, the workers got a break in light of a full generator fuel tank, but that would only be full for about an hour and a half before it was back to work. Additionally, the water wasn’t pumping out of the basement and the first attempt at rigging up the building’s fuel pump to get to the 18th floor didn’t work. A manpowered fuel system was unsustainable since employees on hand were limited in numbers. Things didn’t look great, after all of their hard work.
Four hours later, more help was hired to come in from Brooklyn and Queens. Hundreds of gallons of fuel were available to put into the tank when needed, but at the moment it was full. Water began to be pumped out of the basement. But the anonymous blogger and Squarespace employee reminded all that it wasn’t a sure thing that there wouldn’t still be downtime at some point during the saga. After a midnight fuel refill via runners going upstairs and a human assembly line, the team was ready to catch some much needed rest until a new, more efficient fuel schedule would start the next morning, the first day of November.
The next morning, the basement was cleared of water and it was discovered that a broken line had been adding water to the basement, hence the appearance that the pumps weren’t working. After that was shut off, the water was pumped out, and the fuel pumps needed to be fixed. Later that morning, due to a gas shortage, the building was out of fuel. Users were put on a three hour notice until potential downtime.
But as seemed to be the case with the whole fiasco, more good fortune sprung up when a fuel pump was made operational. The generator’s fuel filter needed to be cleaned, which would mean inevitable downtime, but only an hour, instead of the feared multi-day outage. There hasn’t been more word from Squarespace about the filter cleaning or the potential downtime, but I’m sure their faithful blogger will be sure to update the site if something happens.
For some of you, that was probably a boring recount of what happened to one random data center during the wrath of Sandy. However, we thought it was fun to share if for no other reason than the incredible resourcefulness and dedication that the Squarespace employees showed to their customers and their reputation. Other than that, it’s also interesting to get to see the underpinnings of something we rarely get to see – a data center in emergency. Thankfully, the NetHosting data center is landlocked and in a mountainous region so hurricanes, tornadoes, and most other natural disasters, aren’t a threat.