Folding@home

Planned server room maintenance: Saturday, November 10th, from 5AM to 8AM

One of our key server rooms will undergo network maintenance on Saturday, November 10th, from 5AM to 8AM pacific time.  During the maintenance window, we expect that the servers in that room will be unreachable, hopefully for only 30 minutes each, but potentially for the full time range.  We note that this is only one of our server rooms, so the FAH backend should still be primarily operational, but some donors will see some issues with returning work during this time.  We also stress that the FAH server backend is architected such that even when servers are down, the points for donors will not be lost, and just the accounting for those points will be delayed until the servers are back up. 

Network issue at Stanford

We have a network issue at Stanford in one of our data centers.  We are working on it now.  

Update:  Everything is back up now.

Reports of issue with the Stanford medical school network: Updated

We just received notice that the School of Medicine has scheduled an emergency network downtime Tonight between 5-6 pm (pacific time).  Network engineers expect end users to experience a single, 30-60 second network disruption as part of their work, but I'm posting this in case there's something more significant. 

 

Reports of issue with the Stanford medical school network

Several of the FAH servers are located at Stanford Medical School.  The Stanford School of Medicine Networks have reported outages last night (Thursday, Nov 17, 7:50-9:18 pm, pacific time) and this morning (Friday, Nov 18, 7:47 am, pacific time).  Stanford IT is working with the vendor to diagnose the source of the problems.  We have been warned that the medical school network may possibly be unstable today.

However, note that many of the servers at Stanford are not on this network and moreover we have redundant systems (eg redundant assignments servers) not on this network, so work should go smoothly.  There may be delays in updating stats however.  

Hopefully this is a resolved issue, but we are monitoring the situation in case it isn't.

 

One physical server down (vsp09), bringing down CS's

One physical machine (vsp09) is down, bringing down all of its related VM's (vsp09*).  We're working on it and hopefully will be able to get it back up today, unless there's a more serious issue.

UPDATE:  The machine is now back up and the main CS (vsp09a) is running.  We are keeping the other CS's down, since the other CS's are now obsolete and have been integrated into vsp09a.

update 4: server room maintenance: December 16, 2010

Looks like the server room updates have been completed and our team is starting to bring the servers back up.  It will likely take a few hours for some, since this is an opportunistic time for some routine server maintenance (since they machines had to go down anyway).  Some of the down servers are already back up and serving, more on the way in the next few hours.

update 3: server room maintenance: now (December 16, 2010)

We've started dealing with all of the outage.  It looks like we should have the basics online (stats, fah-web, main AS, most of the key servers), but most of the redundant systems will be down, so there could be outages even with all we've done to try to have the basics up.  Hopefully this will be straightened out by the end of the day pacific time.

update 2: server room maintenance: December 16, 2010

It looks like we will have to take a few machines down early in anticipation of the server room outage tomorrow.  Right now, it looks to be just vspg11 and vspg12 (and their related VM's).  We're working to keep as much up as we can though during the outage.

reminder & update: server room maintenance: December 16, 2010

We've been working to try to minimize the effect of the server room maintenance coming up tomorrow.  Since this is our main room, FAH will be stretched pretty thin during the outage and we expect there will be WU shortages.  Also, clients will not be able to send WUs back to servers that will be unavailable during the outage.

However, some good news is that we have been able to get power to a few key machines so the stats and web page should be up as well as most of the key servers.  If you have trouble getting or returning WUs tomorrow, please just wait it out until we get a chance to get everything back on line.

Stanford network upgrades – Dec 18 through Jan 2

During Stanford's Winter Closure (December 18 through January 2), IT Services plans to schedule a network backbone maintenance window every morning from 4:00 a.m.- 8:00 a.m. to implement improvements in the network, as we did last year.  In most cases, the changes should not affect the connectivity of departmental or home networks.  In cases where they might, any interruption in service should be under 5 minutes.

While Folding@home will be up for this period, there may be brief (~5 min) network interruptions for network traffic from/to off-campus during the daily maintenance windows of 4:00 to 8:00 am pacific time.  The upshot is that the campus should get improved network performance after the upgrades.