Folding@home

Donor Question: Why do certain servers go down more easily?

I'll start posting replies that I make in the forum (http://foldingforum.org) when I think it would be of broad interest.  Here's one regarding server downtime.

Is there a reason, that this particular server seems to have an exceptional amount of down time?

Physically, the server is fine. However, when it gets overloaded, it just fails to continue and the binary must be killed.  This is a software issue, likely in the low-level HTTP code/library.

And what can be done, on a long term basis ...

We have been looking into solutions for the past two years. Two years ago, we refactored the server code to clean up elements and try to isolate the problem (creating the v4 branch of the server code from v3). That did help, but not enough. One feature of the v4 code is that it can detect when it has this problem and restart itself, but we don't want it falsely restarting itself (that can lead to other problems), so the timeout is somewhat long.

About a year ago, we took a radically different approach and worked with an outside programming company (Joseph Coffland from Cauldron) to rewrite the server code from scratch. That new server code (v5) is now under beta testing. Project 3798 are run on the new server platform. We are rolling it out gradually to avoid any sort of disaster, but so far so good; beta testing has found bugs in the v5 code which Joe has fixed and it looks like the code is ready for a bigger rollout (which is underway). I expect that it will see its first duty in production projects (i.e. not server test projects) in early 2009 (likely January 2009) and a more complete roll out throughout the year. 

The SMP and GPU servers will get it last, since they need additional code to bring the v5 code path up to spec with what the GPU and SMP needs. However, we expect this won't be too onerous to get done.

Collection Server (CS) overhaul

We've made some changes to the Collection Servers (CS) which seems to be making a big difference.  We've migrated them to a new network (the old net had many issues, as discussed in previous posts) and recently made some configuration tweaks.  The two new CS's (171.67.108.17 and 171.67.108.25) seem to be doing reasonably well right now.

Hopefully, this will help donors get WUs in faster and allow the FAH team some leeway when a server goes down, knowing the CS is there as a backup.  We'll continue to monitor the CS issue, since this is such a critical part of FAH.

Overuse of fah-web.stanford.edu web pages

Please note that one should not access the cgi pages from the fah-web.stanford.edu web pages too frequently, since that means that other people are blocked out of their stats.  We have seen numerous people accessing these pages hundreds to thousands of times per day (most likely by running a script).  

We have blocked the worst offenders for now.  If you find yourself blocked, please see what was causing this (eg turn off your script) and post a message in our forum (http://foldingforum.org) to get unblocked.

One can access our "fast" team pages by scripts.  For example, for the Pande Lab team (team 1), the cgi-bin url is this

http://fah-web.stanford.edu/cgi-bin/main.py?qtype=teampage&teamnum=1

this would be an example of a bad url to access frequently.

However, if you look at that page, you will see this

Pande Lab

Date of last work unit	2008-11-28 06:53:43
Active CPUs within 50 days	3682
Team Id	1
Grand Score	141820948 (certificate)
Work Unit Count	472377 (certificate)
Team Ranking (incl. aggregate)	30 of 148437
Home Page	http://www.stanford.edu/group/pandegroup/index.html
Fast Teampage URL	http://fah-web.stanford.edu/teamstats/team1.html

Note the "Fast Teampage URL" line at the end.  This url (http://fah-web.stanford.edu/teamstats/team1.html) is designed to be loadable quickly.  The top 2500 teams have a fast teampage URL, so it's likely that your team is one of them.

Another alternative is the 3rd party stats.  They are very slightly less up to date (since they parse a file we send out less frequently), but they help balance the load.  See http://fahwiki.net/index.php/Third_Party_Contributions for a list of 3rd party stats systems.

Please do not use scripts or only use scripts on the fast teampages or 3rd party stats.  By overloading the stats by using automatic scripts, it blocks out others and makes the stats slow or unreachable for everyone.

UPDATE: Main AS back up, but stats, etc will take some time

We got the Main AS back up and all looks good on that box.  The main systems will take some time to get completely back in order (say a couple of hours), but it's looking good so far.

Main AS down. We're looking into it.

The main AS is down.  The two consequences of this are that the secondary AS is handling all of the load and the main stats update have been suspended until this is resolved.  The points are being kept locally on the servers, so WU's are being credited, just not showing up on our web site.

Culprit found in NV core v 1.15 issues for certain hardware?

Engineers at NVIDIA (notably Scott LeGrand) have come up with a theory for the EUE's seen in core 1.15 (and a few others in the 1.15 to 1.18 range) on certain hardware. They found that this core had code optimizations that drove the GPU so hard that it would draw a lot more electricity (one sign of this was running hotter). In some boxes, this was too much electricity and this lead to numerical instabilities. When the same machine was given a beefier power supply, the problem went away.

We've been told that 8800's require 600W power supplies, but we're finding that even a little bigger (eg at least 650W) is important to leave some room for error. We are working to see if there is some way to detect this issue in software, but for now, if you're getting EUE's on the NV GPU client, this is something to consider.

By the way, this will be very important for us to consider future code optimizations. NV core v1.19 removed some optimizations to solve this problem, but there are many cards which would run fine w/this more optimized code. If we can find a way to detect whether the card can draw enough power, we may be able to choose different code paths to allow for greater optimization for cards which can handle it.

We're still looking into this. For now, if you're seeing issues with your card, please consider trying out a bigger power supply. We will continue to look to see if this is indeed the problem and what we can do to help the situation such that the code runs stably on all machines.

PS Please post replies here:

http://foldingforum.org/viewtopic.php?f=52&t=6761

Forum down -- we are talking with the hosting company

The forum at foldingforum.org is down.  We are in contact with the hosting company and they are working on it.  We don't have an ETA from them, but I hope this will be resolved later today.

Folding@home servers are up and running and is not affected in any way from the forum hosting company.

Folding@home/PS3 wins "gold" award at the Good Design Award competition

We're happy to announce that the PS3 client for Folding@home has won a prestigious design award:

The PS3's Folding@home service has nabbed a prestigious Good Design Award, AKA "G-Mark." Since 1957, the Good Design Award has been instituted by the Japanese Ministry of International Trade and Industry to award outstanding design in industrial and consumer products.

Here are some more links.  First a news site discussing it

http://kotaku.com/5078119/foldinghome-wins-japanese-design-award

and the Sony Press Release

http://www.scei.co.jp/corporate/release/081106de.html

PS3 server 171.64.65.83 reset, PS3 clients looking better

We reset 171.64.65.83 and now the PS3 clients are looking better.  We'll keep an eye on this, but it is the weekend in California right now, so we are on a limited staff (and thus response will be a slower than on week days).