Folding@home

Updates thread

For some time, we've been keeping an updates thread with information for every change that we do to the FAH backend (new WUs, servers on- and off-line, etc).  However, many donors likely don't know about it, so here's a reminder:

http://foldingforum.org/viewtopic.php?f=24&t=14714&start=60

How does FAH code development and sysadmin get done?

One of the more common question I get asked is how we do our client/server/core programming and backend system administration.  Also, others were curious about updates on various core projects.  So, I thought it made sense to answer both in one post, since the answers are related.  This will be a bit of a long answer to several short questions, but hopefully it will help give some insight into how we do what we do.

First, some history.  When we started in 2001, I personally wrote most of the code (client, server, scientific code integration, etc), with some help from a summer student (Dr. Jarod Chapman) and some help from Adam Beberg on general distributed computing issues and the use of his Cosm networking library.  I was just starting out as a professor then, with a relatively small group (4 people at the time), so it was common for the leader of the lab to do a lot of hands on work.

As time went on, the group matured and grew, with increasing funding from NIH and NSF.  This allowed the group to grow to about 10 people in 2005.  At this point, much of the duties were given to different people in the lab: the server code development was performed by (now Prof.) Young Min Rhee and then later by Dr. Guha Jayachandran.  Client development was done by Siraj Khaliq, then Guha, then help from several people (including Adam Beberg as well as volunteers, such as Uncle Fungus).  Core development was done by Dr. Rhee, (now Prof.) Michael Shirts, and others.

This model worked reasonably well, with each team member giving some significant, but not overly onerous amount of his/her time (eg 10% to 20%) to FAH development.  These key developers were able to add a lot of functionality, both to aid the science and the donor experience.  

 However, in time, this model became unscalable and unsustainable.  As time went on, the individual developers graduated (in academic research, the research is done by graduate students or postdoctoral scholars, both of whom do not stay longer than say 3-5 years).  While the original team was able to build a powerful and complex system, maintaining that system by new generations of students/postdocs became unsustainable.  The code was complex and well known by the original authors, but maintenance by new developers was complex and easy to make errors, due to the complexity of the software.  

In parallel with these efforts in code development, we also were maturing in terms of our server backend.  We went from having a few small (10GB hard drives!) servers, to a very large, enterprise style backend, with hundreds of terabytes of storage.  This too became a major challenge to manage by the scientific group.

A new plan. Therefore, in 2007, I started a new plan to migrate these duties (code development and system administration) to professional programmers and system administrators.  Today, most of FAH code development is done by professional programmers, and in time I expect all of it will be done that way. The desire to start with a clean code base lead to new projects, such as the v5 server code, second generation GPU code (GPU2), second generation SMP code (SMP2), new client (v7 client in the works), which have been developed with a clean slate.  

There are some differences in how donors will see the fruits of these efforts.  I have found that while the programmers write much cleaner code (much more modular and systematic and maintainable), the code development is typically slower.  While the scientific group can often make certain changes say in a month, the professional programmers may take 2 or 3.  What we get for that extra time is more cleanly written code, no hacks, and a plan for long term sustainability (clean code, well documented code, high level programming practices, etc).  Some projects are still done by the scientific staff (eg Dr. Peter Kasson continues to do great things with the SMP client as well as work towards SMP2), I expect that in time this will all be done by programmers.  

Analogously, sysadmin has been pushed to a professional group at Stanford.  Similarly, they are more careful and methodical, but slower to respond due to this.  My hope is that as we migrate away from our older legacy hardware and they set up clean installs with the v5 server code, the issues of servers needing restarts should be greatly improved.  This infrastructure changeover has been much slower than I expected, in part due to the practices used by the sysadmin team to avoid hackish tricks and to keep a well-organized, uniform framework amongst all of the servers (eg scripting and automating common tasks).

One important piece good news is that the people we've got are very good.  I'm very happy to be working with some very strong programmers, including Peter Eastman, Mark Friedrichs, and Chris Bruns (GPU2/OpenMM code), Scott Legrand and Mike Houston (contacts at NVIDIA and ATI, respectively, for GPU2 issues), Joe Coffland and his coworkers (v5 server, Protomol Core, Desmond/SMP2 core, v7 client).  System admin is also now done professionally, via Miles Davis' admin group at Stanford Computer science.  Also, since she has help desk experience, Terri Fedelin (who does University admin duties for me personally) has also been working on the forum helping triage issues.

Where are we now?   Much of their work is behind the scenes and we generally only talk about big news when we're ready to release, but if you're curious, you can see some of it publicly, such as tracking GPU2 development via the OpenMM project (http://simtk.org/home/openmm) and the Gromacs/SMP2 core via the http://gromacs.org cvs (look for updates involving threads, since what is new about SMP2 is the use of threads instead of MPI).  You can also follow some more of the nitty gritty details on my Twitter feed (http://twitter.com/vijaypande), where I plan to try to give more day-to-day updates, albeit in a simpler (and less gramatically correct) form; the hope here is to try to have more frequent updates, even if they are smaller and simpler.

As the GPU2 code base matured in functionality, GPU2 core development has been mainly bug fixes, which is a good thing.  SMP2 has been testing in house for a while and I expect it will still take a few weeks.  The main issue is trying to make sure we get good scalability with threads based solutions, removing bottlenecks, etc.  The SMP2 initiative lead to two different cores, one for the Desmond code from DE Shaw Research and another for a Gromacs variant (a variant of the A4 core).  We having been testing both in single cpu-core format (the A4 Gromacs core is a single core version of what will become SMP2) and we hope to release in a week or two a set of single core Desmond jobs.  If those look good, multiple-core versions via threads (not MPI) will follow thereafter.

The v5 system roll out is continuing, with the plan to have a parallel v5 infrastructure (set up by the new sysadmins) with our current one, and have the science team migrate new projects to the new infrastructure.  The v5 code has been running for a while in a few tests and we expect one of the GPU servers to migrate this week, with one or two servers migrating every week as time goes on.  The new code does not crash/hang the way the v3/v4 code does (it hung under high load and needed the process to be killed) and so we expect much more robust behavior from it.  Also, Joe Coffland has been great regarding responding to code changes and bug fixes.

So, the upshot of this new scheme is that donors will likely see more mature software, which also means slower revs between cores, both since fewer revs are needed in the new model (a lot of issues are simplified by the cleaner code base) and because the revs now involve a lot of internal QA and testing and more careful methodical programming.  

The long term upshot for FAH is better software and more sustainable software.  It's taking time to get it done, but based on the results so far (eg GPU2 vs GPU), I think it has been worth the wait (but we still have a fair ways to go before we can see all of the fruits of this work).

Update of the EULA

We have added one word to the End User License Agreement (EULA), but it is an important word.  The EULA can be found at

http://folding.stanford.edu/English/License

We added the word "written" to this sentence:

"You may use this software on a computer system only if you own the system or have the written permission of the owner."

We felt that this was an important addition in order to avoid any confusion.  There have been a few situations where donors felt that they had permission, but the owners of the computers did not.  Having written permission is the best way to make sure that there is no doubt.  It also gives protection to the donor in that he/she would then have proof of permission, avoiding problems involving oral agreements.

What does the SMP core do?

Our SMP core is very different than the way other distributed computing projects handle multi-core CPU's, and I thought it might be interesting for the FAH community to hear about the differences, pro and con.  As I think most people interested in computers know, Moore's law stating that the transistor count in CPUs will double every 1.5 years has continued for decades.  Most people think of Moore's law in terms of the speed of CPU's, but this isn't what Moore originally had in mind.  In the past, more transistors have lead to greater CPU speeds, but that has essentially ended (at least for traditional CPU's) a few years ago. 

But if Moore's law is still marching along (as it is), what do all those transistors do?  Over the last few years, more transistors have translated into more CPU cores, i.e. more CPUs on a chip.  While this is not what we wanted, this is perhaps not necessarily a disaster, if one can use these multiple CPUs to get faster calculations.  If we simply do more calculations (i.e. multiple Work Units, or WU's, simultaneously) not faster calculations (a WU completed in less time), distributed computers will run into the same problems that face supercomputers: how to scale to lots and lots of processors -- i.e. how can we use all these processors to do a calculation faster over all.

In FAH, we've taken a different approach to multi-core CPUs.  Instead of just doing more WU's (eg doing 8 WU's simultaneously), we are applying methods to do a single WU faster.  This is typically much more valuable to a scientific project and it's important to us.  However, it comes with new challenges.  Getting a calculation to scale to lots of cores can be a challenge, as well as running complex multi-core calculations originally meant for supercomputers on operating systems not meant for this (eg Windows).

Right now, our SMP client seems to be running fairly well under Linux and OSX -- operating systems based on UNIX, as is found on supercomputers.  We use a standard supercomputing library (MPI) to run these WU's and MPI behaves well on Unix-based machine.  MPI does not run well on Windows and we've been running into problems there.  However, as Windows MPI implementations mature, our SMP/Windows app will behave better.  Along the way, we also have a few tricks up our sleeve which may help as well.  However, if we can't get it to run as well as we'd like on Windows, we may choose to overhaul the whole code, as we did with the GPU1 client (which was really hard to run).

We're very excited about what the SMP client has been able to do so far.  One of our recent papers (#53 in our papers web site http://folding.stanford.edu/English/Papers) would have been impossible without the SMP client and represents a landmark calculation in the simulation of protein folding.  We're looking forward to more exciting results like that in the years to come!

More info about the GPU1 to GPU2 transition

There's been several questions regarding the GPU1 client and why we decided to shut it down.  I hope I can shed some light here at least on why we're doing what we're doing, such that even if people disagree with our decisions, they can at least see where we're coming from. 

Some people have asked "why shutdown the client if it's working?" The bottom line here is that the GPU1 results are no longer scientifically useful. It's pretty clear now that DirectX (DX) is not sufficiently reliable for scientific calculations. This was not known before (and some people wouldn't believe this until we proved it). With the GPU1 results, we can now show what the limitations are pretty decisively.

GPU1 also did help us a lot in terms of developing its successor and what's needed to run GPU's in a distributed computing fashion.  The good news here is that GPU2 is behaving very well, on both ATI and NVIDIA hardware, and this is a direct result of what we've learned with GPU1 WU's.  In the end, however, GPU1 will not be able to help us understand protein misfolding, Alzheimer's Disease, etc due to these unresolvable limitations. We could keep GPU1 live just crunching away in its current form, but that would be wasting people's electricity at this point, as we've learned everything we can learn from those cards, based on what they can do.

In the past, we had a somewhat similar shutdown situation, i.e. when QMD core projects stopped. In that case, donors were left hanging since we didn't give any warning for stopping QMD projects. We did try (perhaps unsuccessfully) to handle the GPU1 situation better than QMD. In QMD, we stopped needing that core and so we stopped the calculation without warning, not realizing the impact that would cause. With GPU1, we gave a several month warning (indeed, note that GPU1 is still actively running), so this is information in advance to shutting down GPU1). We tried to avoid the QMD situation by giving advance warning, but it looks like donors would like even more advance warning. However, there's limits to how much in advance we know the situation ourselves.

Indeed, the knowledge that it made sense to end GPU1 came reasonably recently to us.  We have been working on CAL for a while and it seemed that CAL might be a solution, but we only knew until we got some testing "in the wild."  DirectX (DX -- what GPU1 is based on) works much better in the lab than in the wild, and it was possible that CAL behaved that way too.  After seeing that CAL behaved well in the wild, it became clear that the GPU1 path was obsolete.  However, this is a relatively recent finding and we made the announcement about the situation relatively shortly thereafter.

It was a tough decision. Some suggested we just leave GPU1 running, even though people's electricity really would be going to waste, other than generating points. I didn't think that was a good idea. We did know it would be a tough PR hit, but when people talk about the history of FAH, I want to make it clear that we're here to address AD and other diseases, not just running calculations for the sake of points and nothing more (which has been the critique of some other distributed computing projects).

So, what's the right thing to do?  I guess it comes to this: would GPU1 donors be happier if we just keep GPU1 servers running, doing with no scientific value, but just for points? We could do that, at a cost of taking away personnel from improving existing clients, keeping existing servers going, etc for the sake of keeping GPU1 running. However, that's not what FAH is for and I think it's important that FAH not devolve into a big points game, losing sight of why we're doing what we're doing.

PS Some further discussion can be found here

How the GPU2 core works

There have been some misunderstandings on how the GPU2 core works.  In particular, for small proteins like villin on GPU's with large number of stream processors (SP's) like the 3850 or 3870, the protein is too small to use a larger number of SP's unless the CPU is very fast.  Some people have guessed that there is some internal SP limit.  This is incorrect; the problem is that small proteins can't be parallelized amongst a large number of SP's. 

We are working to release larger proteins (about 2x the number of atoms) as they are more interesting scientifically and use the GPU's (even the high end ones) much closer to 100%.  The exciting part for us is that the larger proteins run at almost the same speed as the slower ones on GPU's (whereas on CPU's, they're 4x slower); this is where the GPU2 code should shine.  In parallel, Mike Houston at AMD is working to optimize CAL such that it has lower CPU overhead. 

For now, we're pushing out villin WU's as a test (good to know that the code is working well), but we expect the larger WU's to be going out soon (say a week or two, pending internal testing).

How FAH works: Thermodynamic alchemy

One of the remarkable strengths of thermodynamics is that it is in many ways more general than many of the laws of physics.  One of the key concepts in thermodynamics is that of a state function.  The key aspect of a state function is that if you're interested in the differences in a state function between some initial and final state, it doesn't matter what you do to get from the initial to the final -- whatever path you take would get you the same value for the difference of the state function.

How is this useful?  Well, often we can take paths in simulations which are not possible in reality, but since the values of state functions only depend on the initial and final state, as long as those as the same as in reality, we get the same result!  One key way we use to take advantage of this is thermodynamic alchemy.  For example, often one is interested in the difference between an amino acid could be a single methyl group.  In simulation, we can turn a proton alchemically into a methyl group and calculate everything we need thermodynamically, even if this isn't possible to do experimentally in a direct way!

How FAH works: Folding on streaming processors (GPU's and PS3)

One of the critical issues in computer science right now is the limits to how fast a single CPU can calculate.  While Moore's law is still going strong in a literal sense -- i.e. the number of transistors which one can put on a chip is doubling every 2 years or so -- this doubling of transistors is not leading to a doubling in CPU speed as it used to over the last few decades.  Well, at least not for typical programs (eg Microsoft Word).  In order to get big speed increases, there'sa major change in the programming paradigm.   One key change is the existence of "streaming processors."  GPU's and the Cell processor in the PS3 are both examples of streaming processors.

What makes streaming processors potentially much faster than regular CPU's is how they handle computation vs memory access.  Normal CPU's use lots and lots and lots of transistors on cache (local memory on the CPU chip to help keep the CPU fed with data and instructions).  Streaming processors use the additional transistors on additional computing elements (eg floating point units).  By doing so, they can do lots of FLoating point OPerations per second (FLOPS) in an optimal situation, although getting one's code to behave optimally is not easy.  Typically this means balancing FLOPS with memory access to make sure that there's data available for calculation.  This has been the primary challenge in our GPU and PS3 codes, and is something which we have, for the most part, figured out for a significant subset of the calculations we run on FAH.

These advances have lead to our GPU and PS3 clients.  The family history of all of this starts with the GPU core.  This GPU code was then brought over to the PS3 and enhanced.  We are working to bring back some of those scientific enhancements back to the GPU code.  This is all pretty bleeding edge, but so was distributed computing in 2000 when we started.  Our expectation is that given how modern processors are developing, in 8-10 years streaming processors will be much more standard and will be a major way in which FAH works. 

Finally, it's also interesting to think of how CPU's may themselves turn into streaming processors.  As CPU's add more cores, they start to have more functionality similar to streaming processors in a limited way.  Perhaps more interesting are some of the new chips rumored to be developed at Intel and AMD/ATI.  Intel's 80-core chip is very interesting and something which our code would likely run well on.  Also, the fusion of AMD's CPU's with ATI's GPU's could be very exciting, potentially bringing the best of both worlds.  We're looking forward to these and lots of other emerging technology.  FAH is running very fast now (over a petaflop, i.e. 1,000,000,000,000,000 floating point operations per second !) and we look forward to continuing to push the frontiers.

How FAH works: Markov State Models

This post may get a little technical, but I wanted to start a new set of posts describing the inner workings of Folding@home.  To say FAH is complex is in many ways an understatement.  On the surface, FAH does seem a lot like other distributed computing projects: there's lots of WU's which go out to client machines and they get calculated and then come back.  However, there are a lot of differences going on in FAH.

One of the principal challenges in FAH is that we're trying to use lots of processors to in a sense speed up a calculation which many would have thought was intrinsically serial, i.e. that could only be done by a single very, very, very fast processor.  The reason why is that we are studying how proteins change in time, and it's hard to parallelize the 23rd step if you haven't completed the 22nd step, etc.

However, through the years we have been developing ways to solve this issue.  In the last few years, we have made significant progress in a method called "Markov State Models" or MSMs for short.  MSMs in a sense allow us to parallelize these seemingly intrinsically serial tasks.  The way this works is that we build a kinetic model for the process, dividing up the possible dynamics into a series of states (related protein conformations) and rates between these states.  The rates are what's calculated in FAH WU's.  Once we have all this data, we need to run some fairly sophisticated Bayesian Machine Learning methods to identify what are reasonable states and then to calculate the rates between them.

We have had a several recent advances in MSM methodology and those papers are on our papers website.  We have also had several MSM applications including studying protein folding, lipid vesicle fusion, and abeta aggregation (Alzhiemer's Disease simulations).  While we will continue to improve our MSM methodology, we are very much excited about the potential applications and have great thrusts going in both areas.