Folding@home

Machine went down last night, affecting others. Reset this morning.

A machine went down last night, which brought down vspg1, vspg1v, and vspg1v2 (these are all different virtual interfaces on the same physical server).  Our admins (working during the holidays -- thanks to them for their hard work) reset the machine this morning and we got the machines going again.

While vspg1* were down, this lead to a much greater load on other machines, causing issues for vspg2v2, vspg4v2, and vsp07v.  These reset themselves some time around 7:30am pacific time (with the exception of vsp07v, which I reset).

So, it looks like the server farm is settling down.  I can't wait until we get the new, really powerful machines loaded up with jobs to help take over some of this load.  Some of the jobs being tested right now are on those machines, so that should be very soon.

Note that during the holidays (especially the next week or so), we have a limited staff in the office, but those that are working are diligently looking for problems.  Happy Holidays!

main web page issues

We're seeing some issues with the main http://folding.stanford.edu web page.  We have some ideas, but it's late here (11pm pacific time), so we'll have to wait for IT to get in to work in the morning to get a fix implemented.

For now, if you see "Access Denied", please try to reload the page: that should yield a working page.

UPDATE:  9am Dec 9 
Stanford ITS has sent me an email saying that this is now fixed, but it doesn't look like that to me (although it is better).  I have informed them that it isn't and am waiting on them.  For now, you can hit retry until you get the page you want, but clearly this is still broken badly.

UPDATE:  Looks like this has now been resolved.

stats hiccup -- we're on it

There was a problem with the stats server over the night.  We got to it first thing in the morning (around 6:30am pacific time), but it looks like a few stats updates were missed.  The missed data is backed up on another machine and we can re-enter it.  However, we typically take a couple of days to re-enter it, to make sure it is done right.

So, the bottom line is that the stats system is back up, will we recredit some missing WUs not credited over the night, but this will take a couple of days to complete.  Sorry for the delay on the recredit, but it's good to make sure we do it right, rather than in a rush.

Power outage

I just got news that our main power feed has been damaged by the SIM1 construction crew.  We may be out of power for over 6 hours.  The good news is that most of FAH is immune from this right now.  The bad news is that we may have to stop the stats update and turn off the web server which serves the stats.

Note that the Assignment servers and all of the work servers are separate from this now, so FAH will be running, even if the stats update is put on hold.  Stats would continue to accumulate, just not update on the web site until we turn that back on (if we are forced to bring down those servers).

Brief unplanned outage

There was a brief, unplanned network outage. 

There was an unintentional network outage caused by maintenance this morning from about 7:30 to 7:50 AM. Ironically, the work we were performing this morning is part of our redesign to prevent things like this from happening.

The good news is that this sort of thing should become much more rare in the future.

Two servers down: vspg2 and vspg2v

Maintenance is under way on these machines.  Looks like a PDU went down during a equipment move today and the redundant power did not kick in right.  Hopefully this will be something as simple as powering up the RAID again and restarting the server.

It's 7pm pacific time, so the admins are likely not going to get to this until they get back in the morning, but we'll see.  We've filled out a ticket and mentioned this is a very high priority item.

We are also monitoring the collection servers for these two servers.  They appear to be up and running well, so hopefully this will not impact donors' return of WUs.

UPDATE:  The servers went back up about an hour ago and have been running well since then.  Thanks to our admins for their work on this one.

GPU server issues

We've had a rough night with GPU servers.  One has been down hard over the day yesterday (it crashed hard and now can't find its / partition -- the admins are attempting a rescue disk fsck this morning).  Two more went down last night (PST) due to the heavy load, but those were easy to get back up (they are up now).

We are stretched a bit thin as we are implementing the new server infrastructure in parallel with the old one.  The upshot is that once the new one has been deployed, we will have much more functional collection servers (CS's) and also get work servers (WS's) that should not need to be restarted nearly as frequently when under heavy load.

We are beginning the roll out of the new WS (v5) code this week onto GPU servers, although these issues have slowed us down a bit. 

Main AS and stats down, ETA for fix 10am pacific time

The machine which handles the main assignment server (AS) and the stats update is currently down.  We expect that this machine will be back up by 10am pacific time.  Since this has happened previously somewhat recently, we are looking into transferring some duties on this machine to both keep it less loaded and to lessen the severity if it goes down again.

This means that the stats update will be down until we get the machine back up, but the back up AS should handle all classic clients, and the PS3 and GPU clients are unaffected (as their AS is on a different machine).

Updates to the stats system

We have made some updates to the stats code to make it faster and more useful.  The recent changes speed web page creation in general and also cache team pages such that they can be read during stats updates.  This is the first stage of further improvements of the stats to come in 2009.

Stanford Winter Holiday Schedule

Stanford will be closed for the next two weeks for the Winter Holidays (ending January 5, 2009).  FAH will still be up (FAH is always up), but we will be on a reduced staff.  FAH team members have broken up their vacations so that there will always be someone around, but the main issue is that responses to problems will likely be slower than normal.  However, we have been working to add lots of jobs, clear out lots of HD space on servers, get all the servers up, such that we should hopefully be in good shape even if there are some problems. 

We'd like to wish all the FAH donors a happy holiday and thank all of you for all your great help with our project.