Folding@home

Updated GPU2 server code in place

We've updated the server code on 171.64.122.74 to make it more responsive.  We're keeping an eye on it to see for sure.  One of the major differences is how long the server waits for a connection (it was waiting way, way too long before and that filled up all of its threads).  We've cut that down as well as significantly increased the number of threads.  In combination with no new assigns for this server for a while, this should drain it and get the server I/O back to normal.

GPU2 server performance: update

We've been looking over the server code to see what's going on and I think we have some news. What's happening is that the server is being a bit too generous in terms of when it times out connections, filling up all of the available server threads. For now, we've bumped up the # of threads (not unlike adding more cashiers at the grocery isle, so that a few slow people don't slow down everyone) from 200 to 500. That should help a lot in the short term.

On Monday, we'll make some code modifications to make a more complete fix to this.  It may take a little while to get this tested and implemented, but I expect that this should be in by Monday afternoon.  That should greatly ease what we're seeing.  (PS Note Monday = Monday pacific time)

In some ways, this has crept up on us, as nothing has changed (all our servers are up and running) -- it's just that the extended down time built up a large # of WU's to return and the server is still trying to catch up.  We've turned off new assigns from this machine to help it catch up (no new WU's will mean it spends all it's time receiving WU's).

With this change, the threads tweak, and the code update coming later today we should have some more news (hopefully good news) later on Monday.

Collection server code update

For a while, the collection servers (CS's) haven't been working well.  We've been looking into why.  A few weeks ago we overhauled server-CS interaction and added some more CS's to help the load.  We have continued to look into CS issues and have some new ideas.

With these new ideas, we've made an update to how the collection server works.  We think this will help clients send back results significantly.  This update code has been placed on 2 CS's right now (171.64.122.76 and 171.64.122.86) and we're going to watch to see if this improves the situation.  We have a few other tweaks that might help as well, but we want to try this one first.

Issues with maintenance on GPU2 server

One of our servers (171.64.122.74) had some issues during maintenance which delayed it's coming back on line.  Due to this, 171.64.122.74 was back on line on Saturday (PST) instead of Friday evening, due to an issue found during maintenance.  Luckily this maintenance is a once a year thing, so it won't be needed for a while.  It's back up now.

Due to this server being off line, the backup server (171.64.65.20) took quite a hit.  To avoid this, we're moving to longer WU's (8-24 hours) and looking to add an additional server so the servers take less of a hit.  Longer WU's alone will make a HUGE difference, since the servers would get hit much less frequently and thus the load would be much smaller.  That should greatly improve these issues.

The current situation is that all servers are up and running, but there's a lot of clients trying to access them.  We'll keep an eye on them throughout the weekend to make sure they're going well.

Planned server downtime over the next week

Starting today and lasting for about a week, we will be taking down servers every day for planned maintenance.  Today, we have taken down servers with the following IP addresses

171.64.122.124
171.64.122.125
171.64.122.131
171.64.122.132
171.64.122.133
171.64.122.136
171.64.122.137
171.64.122.138
171.64.122.139
171.64.122.141
171.64.122.142
171.64.122.70

and they will be back up today (if all goes according to plan).  These servers above are not heavily used, so their downtime should not highly affect FAH.

In subsequent days, we will be taking down other machines (see below).  The downtimes for these machines should be much less (~2-5 hours).

Update date     IP     Machine Name  Contact
24-Jul    171.64.65.56     vspg4     kasson
24-Jul    171.64.65.73     vspg4v    vvoelz
24-Jul    171.64.65.106    vspg4v2   relly1

24-Jul    171.64.65.58     vspg5     densign
24-Jul    171.64.65.83     vspg5v    luttmann
24-Jul    171.64.65.111    vspg5v2   vvoelz

25-Jul    171.64.122.73     VSP06    densign
25-Jul    171.64.122.83     VSP16    kasson
25-Jul    171.64.122.88     VSP21    kasson

25-Jul    171.64.122.74     vsp07    densign
25-Jul    171.64.122.84     vsp17    nick
25-Jul    171.64.122.42     vsp22    karsson

28-Jul    171.64.122.75     vsp08     kim.branson
28-Jul    171.64.122.76     vsp09     luttmann
28-Jul    171.64.122.86     VSP19     luttmann

29-Jul    171.64.122.77     VSP10    
29-Jul    171.64.65.121     vspg6-vz7 pande

30-Jul    171.64.65.20     vspg1     densign
30-Jul    171.64.65.63     vspg1v    densign
30-Jul    171.64.65.96     vspg1v2   densign

30-Jul    171.64.65.42     vspg2     luttmann
30-Jul    171.64.65.64     vspg2v    kasson
30-Jul    171.64.65.102    vspg2v2   vvoelz

31-Jul    171.64.65.55     vspg3     jchodera
31-Jul    171.64.65.65     vspg3v    kasson
31-Jul    171.64.65.103    vspg3v2   densign

31-Jul    171.64.122.72    VSP05     gbowman
31-Jul    171.64.122.78    VSP11     gbowman
31-Jul    171.64.122.82    VSP15     gbowman

Stats under maintenance -- update

Looks like the worst is over.  We've rebooted a server, pulled a db from a backup, checked the consistency and it's looking ok so far.   We're running a manual stats update right now.

Stats under maintenance

We're working on the stats system right now.  The estimated ETA for completion is ~5pm pacific time.   We'll give updates as there's more to say.

Sharing FAH results -- raw data now available

We've been working behind the scenes for a while to find the right way to share the raw data results from Folding@home.  We've partnered with the Simbios National Biomedical Computing Center to provide data for download.  If you're curious, check out our first data set project page

https://simtk.org/home/foldvillin

We'll be releasing more data as time goes on.  Our hope is that by making the raw data openly available, this will greatly supplement the published results.

Modification to Assignment Server code

We've made a modification to how the Assignment Server (AS) code works.  We've done some initial testing and are now releasing it to the backup AS (assign2.stanford.edu).  If that checks out, we'll release it to the main AS.  The change involves how we assign SMP WUs.  If you're seeing something strange there (eg SMP WUs' to non-SMP clients or vice versa), please let us know in the forum (http://foldingforum.org).

FAH/SMP Q & A

There was a good question in the forum that I thought others would be curious to hear:

From Vijay's blog entries it would seem that the SMP client has some fundamental advantages over running multiple singlecore client, but I can't really think of how that might be. Do you know of some architectural overview of how the MPI stuff is being used in this context?

We could just run multiple independent clients, but this would be throwing away a lot of power.  What makes an SMP machine special is that it is more than just the sum of the individual parts (CPU cores), since those cores can talk to each other very fast.  In FAH, machines talk to each other when they return WUs to a central server, say once a day.  On SMP this happens once a millisecond or so (or faster).   That 86,000,000x speed up in communication can be very useful, even if there isn't 100% utilization in the cores themselves. 

The easy route would have been to run multiple single-CPU FAH-cores (this is what other projects do), but that would be a big loss for the science, as this throws away a very, very powerful resource (fast interconnects between CPUs).  Indeed, it is this sort of fast interconnect which makes a supercomputer "super", since the CPUs in supercomputers (eg BlueGene) are pretty slow, but the communication between cores is very, very fast. 

We've done a lot to develop algorithms for FAH-style internet connections between CPUs, but there are some calculations which require fast interconnects, and that's where the FAH/SMP client is particularly important.  By allowing us to do calculations that we couldn't do otherwise, the science is pushed forward significantly (and we thus reward SMP donors with a points bonus due to this extra science done and the extra hassle involved in running the SMP client).

I guess it remains to be seen if we can pull off MPI on FAH to the point where it works effortlessly, but so far Lin and OSX look pretty good, so we're close.  The A2 core should hopefully seal the deal.  Now, the main task is getting Windows/SMP behaving well ...