For some time, we've been keeping an updates thread with information for every change that we do to the FAH backend (new WUs, servers on- and off-line, etc). However, many donors likely don't know about it, so here's a reminder:
Looks like the server room updates have been completed and our team is starting to bring the servers back up. It will likely take a few hours for some, since this is an opportunistic time for some routine server maintenance (since they machines had to go down anyway). Some of the down servers are already back up and serving, more on the way in the next few hours.
We've started dealing with all of the outage. It looks like we should have the basics online (stats, fah-web, main AS, most of the key servers), but most of the redundant systems will be down, so there could be outages even with all we've done to try to have the basics up. Hopefully this will be straightened out by the end of the day pacific time.
It looks like we will have to take a few machines down early in anticipation of the server room outage tomorrow. Right now, it looks to be just vspg11 and vspg12 (and their related VM's). We're working to keep as much up as we can though during the outage.
We've been working to try to minimize the effect of the server room maintenance coming up tomorrow. Since this is our main room, FAH will be stretched pretty thin during the outage and we expect there will be WU shortages. Also, clients will not be able to send WUs back to servers that will be unavailable during the outage.
However, some good news is that we have been able to get power to a few key machines so the stats and web page should be up as well as most of the key servers. If you have trouble getting or returning WUs tomorrow, please just wait it out until we get a chance to get everything back on line.
During Stanford's Winter Closure (December 18 through January 2), IT Services plans to schedule a network backbone maintenance window every morning from 4:00 a.m.- 8:00 a.m. to implement improvements in the network, as we did last year. In most cases, the changes should not affect the connectivity of departmental or home networks. In cases where they might, any interruption in service should be under 5 minutes.
While Folding@home will be up for this period, there may be brief (~5 min) network interruptions for network traffic from/to off-campus during the daily maintenance windows of 4:00 to 8:00 am pacific time. The upshot is that the campus should get improved network performance after the upgrades.
One of our main server rooms will be undergoing maintenance on December 16, 2010. This will mean some of the FAH servers will be off line on that day. It looks like most of the key FAH infrastructure will be up, but there will likely be WU shortages on that day since a large fraction of machines will be down.
We are working to optimize what we can by distributing jobs to servers in other server rooms, but we wanted to give donors a heads up in advance so they know this is coming.