The electrical power is out at most Stanford buildings as of 5:20 this morning (Tuesday, Jan 19). SHC Engineering and Maintenance reports that the Stanford Cogen power plant (the power plan that powers Stanford) is currently off line. Emergency generators in most of our server rooms are operating, but one room (associated with VSPG* servers) is currently without power.
We do not currently have an estimate of when power will be restored. Moreover, while much of FAH is still up at the moment, we may have to take servers down if the temperature in the server rooms gets too high (the cooling is down as well).
We'll update the blog as we get news. You can also see which servers are up or down on our serverstat page.
UPDATE: 3pm Pacific Time
It took a long time for the server room in the Computer Science Building to come back on line, but it's back now. We are checking out the servers and restarting binaries to serve FAH WUs. Looks like this is basically over. Due to redundancy in our use of server rooms on campus, most of FAH was up even while the whole University was in the dark, so while this outage was non-ideal, luckily most of FAH was working during the outage.
UPDATE: 5pm Pacific Time
All of our servers are back up. They are getting hit hard due to the backlog, but this will settle down in time. If you're having trouble getting or returning work, this is not unexpected and should settle down in a few hours.
