We shut down our offices on 12/22 with the plan that we would have minimal physical need to visit our data center. We have an aging data center, with air conditioning units that are over 20 years old (actual age unknown). We just received replacement funding last October, but seasonal weather postponed the actual replacement project until spring. As we started planning for replacement, we learned about zinc whiskers and the need to plan to minimize dust contamination, so the extra time for planning has been valuable.
During the holiday break, our lead UNIX sys admin, Andrew, noticed that email was not available about 5 PM on Christmas Eve. He went to the data center to check, and found the temperature over 102 degrees. Both AC units were in a failed state. Andrew called, and our facilities manager Gail and I went in. Some of the servers had gone into automated shut-down due to the heat. Andrew proceeded to shut down everything except the most essential systems. We keep fans around because we've experienced failures before, so we opened the doors, turned on the fans, and called in the contracted AC technicians.
It was Christmas Eve, so it took a while for a response. Gerard the AC tech came in to find that the main electrical feed to the roof chillers was not working. We then had to call in the university electricians (another drive wait), who found the circuit breaker tripped. The breaker was reset, cooling was restored and we went home around 2 AM.
