Here's our (I think) last update on this recent outage. This was a major disaster at Stanford affecting the whole campus and I'm grateful for our team coming in on Sunday to get things back up. The workservers have been up since then and work and stats have been saved. The stats updating was put on hold until we can make sure everything looked ok. We've turned it back on. Please note that there is no stats loss while we turn off updates. People should see a big bump in their stats shortly. Thanks for bearing with us through this.
The whole Stanford campus is having a major issue due to the lack of chilled water on campus. This is causing issues for many of our servers in different server rooms. As of this moment, this is still not resolved. We cannot bring servers back on line until this is resolved. It's getting late (~10:30pm PST), so we'll check back tomorrow morning to see hopefully that this is resolved. Until then, several of FAH's servers are down unfortunately.
UPDATE 4:30am Pacific Time: Chilled water came back on line at 11pm, but several of our servers are still down. Our sysadmins will work to get them back up, but it may not be until Monday, depending on their availability on Sunday.
UPDATE 11:30am Pacific time: Our sysadmins have been in the office getting machines back on line. We're almost there, although it looks like there are a few machines which have issues resulting from the outage.