Folding@home

server room maintenance: December 16, 2010

One of our main server rooms will be undergoing maintenance on December 16, 2010.  This will mean some of the FAH servers will be off line on that day.  It looks like most of the key FAH infrastructure will be up, but there will likely be WU shortages on that day since a large fraction of machines will be down.

We are working to optimize what we can by distributing jobs to servers in other server rooms, but we wanted to give donors a heads up in advance so they know this is coming.

VSPG network down, sysadmins on it

I'll post more info as we get it.

UPDATE 10:46pm PT:  Looks like the issue has been fixed, at least for now.  Apparently this outage was somewhat widespread throughout the Stanford campus.  It was just a network outage (the servers were all up the whole time).

Update on NV GPU servers

We have been working to track down the nasty bug on the NVIDIA GPU WS's that is causing problems for donors sending back WUs.  We have been trying different fixes over the last week, but this has been very tricky to figure out.

After another brainstorming session this afternoon, I think we have a good plan for the short term and long term.  I hope that new WUs being assigned won't see this problem due to rerouting of assignments.  Joe is also going to pound out the bugs on his new WS on vspg11a to get that going.

I'm very sorry for this major issue.  This has been called the worst outage we've had and I think we agree.  I've had a long chat with the development team about this and we've talked about how to fix issues in the WS code release cycle.  I think the plan we have in place will stop this from happening in the future, but the main issue right now is to solve the problems at hand.

UPDATE 6pm 2/19/2010 -- after a week of working on this, trying lots of stuff, and nothing working, I think we've found something promising.  I'm nervous typing this as everything looked promising before, but at least I think Joe's found the reason for the problem, which is the hard part.

UPDATE 11pm -- so far so good.  It looks like this fix may be sticking.

UPDATE 7:30am 2/20/2010 -- looks like the fix is indeed working.  We will continue to monitor the servers closely over the weekend.

Stats db migration in progress

We've talked about this for some time, but now's the time to start the migration to the new stats db hardware.  We are doing it now and everything looks ok so far.  We are keeping several safeguards in place in case there is a problem.

IF there is a problem with the stats, please bear with us.  There are several links we need to update and it's possible that a link is still pointing to the old db.  Also, in case of emergency, we are keeping track of all the new stats from this point in a special place, so even in the worst case scenario, we can just go back to the old db and input all the new stats into it.

So, the stats will be down for a bit and there may be some inconsistencies for a day or so while we get all the links updated.  The good news is that we'll have much faster stats soon, which will be great for all of us.

UPDATE 1
The migration is now done and it looks like everything is working.  We've tested out the stats pages and done a small manual stats update.  All looks good.  However, since stats are so important, before diving in and just putting everything back to automatic updates, I wanted to see if donors see any problems.  If you do, please report them in our forum (http://foldingforum.org).

UPDATE 2
It looks like everything has migrated well.  We have the stats back on normal updates and those updates are going fast (under 10 minutes).  With the new hardware, I bet we can make it even faster, but that's for later.  We have turned back on certain features we previously turned off (eg CPU counts).  We have more ambitious plans for the future, especially ideally getting to the point where the stats are never off line (even during updates), which is now possible with the new hardware.

Irregular stats update schedule

Over the weekend, we've had trouble with cron on the server which initiates stats updates.  This doesn't affect that stats data itself (past, present, or future), just the initiation of various tasks.  This machine was backedup and then restarted this morning and it looks to be happy now.  We are keeping an eye on things to make sure that all is back to normal.

GPU2 core auto-upgrade planned for Monday Jan 4, 2010

Happy New Year!  We're starting out the new year with a planned auto-core upgrade for GPU core 11.  This is planned to happen Monday morning Stanford time.  This will bring the GPU core to version 1.31, which has many improvements.  In particular, this core is very helpful with the new P10101 WUs.  If you'd like to upgrade sooner, you can always do a manual core upgrade at any time by deleting your GPU core (or perhaps safer -- renaming the file so you can revert back if you want).

However, such manual core upgrades are just for FAH experts.  The FAH client software will do all of this for you if you like, by the auto-upgrade mechanism built-in on Monday.

It's also worth mentioning that we have made good progress with the GPU3 core and that should be rolling out of beta to production early in 2010.  

Machine went down last night, affecting others. Reset this morning.

A machine went down last night, which brought down vspg1, vspg1v, and vspg1v2 (these are all different virtual interfaces on the same physical server).  Our admins (working during the holidays -- thanks to them for their hard work) reset the machine this morning and we got the machines going again.

While vspg1* were down, this lead to a much greater load on other machines, causing issues for vspg2v2, vspg4v2, and vsp07v.  These reset themselves some time around 7:30am pacific time (with the exception of vsp07v, which I reset).

So, it looks like the server farm is settling down.  I can't wait until we get the new, really powerful machines loaded up with jobs to help take over some of this load.  Some of the jobs being tested right now are on those machines, so that should be very soon.

Note that during the holidays (especially the next week or so), we have a limited staff in the office, but those that are working are diligently looking for problems.  Happy Holidays!

main web page issues

We're seeing some issues with the main http://folding.stanford.edu web page.  We have some ideas, but it's late here (11pm pacific time), so we'll have to wait for IT to get in to work in the morning to get a fix implemented.

For now, if you see "Access Denied", please try to reload the page: that should yield a working page.

UPDATE:  9am Dec 9 
Stanford ITS has sent me an email saying that this is now fixed, but it doesn't look like that to me (although it is better).  I have informed them that it isn't and am waiting on them.  For now, you can hit retry until you get the page you want, but clearly this is still broken badly.

UPDATE:  Looks like this has now been resolved.

stats hiccup -- we're on it

There was a problem with the stats server over the night.  We got to it first thing in the morning (around 6:30am pacific time), but it looks like a few stats updates were missed.  The missed data is backed up on another machine and we can re-enter it.  However, we typically take a couple of days to re-enter it, to make sure it is done right.

So, the bottom line is that the stats system is back up, will we recredit some missing WUs not credited over the night, but this will take a couple of days to complete.  Sorry for the delay on the recredit, but it's good to make sure we do it right, rather than in a rush.

Power outage

I just got news that our main power feed has been damaged by the SIM1 construction crew.  We may be out of power for over 6 hours.  The good news is that most of FAH is immune from this right now.  The bad news is that we may have to stop the stats update and turn off the web server which serves the stats.

Note that the Assignment servers and all of the work servers are separate from this now, so FAH will be running, even if the stats update is put on hold.  Stats would continue to accumulate, just not update on the web site until we turn that back on (if we are forced to bring down those servers).