PS3 issues, update #6

We have been working aggressively to find the root of this problem, investigating all possibilities (server, client, network, etc).  We have been debugging the entire FAH system on the PS3 over the last few days (as there could be several causes for what we're seeing), examining especially how the server is interacting with the clients, what the clients are doing, and how the Stanford network is handling the situation.  Several Stanford network engineers have looked into the problem to see if this is a network issue, but that does not appear to be the case.  The servers are running extremely well right now.  Our investigation so far has found that this issue is due to a problem in the client, and we have identified a specific issue that's causing this problem. 

With the completion of this investigation Saturday night, we have given the info of the results of our debugging and our plan on how to fix this to Sony last night, and we are hoping that they can come up with an updated client soon.  Unfortunately with the PS3, we cannot update the client ourselves, otherwise we would release a client update ourselves, as we have done in the past as needed.  Therefore, over the last few days, we have worked on server side tweaks (the only part we can work on) while Sony can work on the client.  Before the client gets updated, we expect that the situation will continue to improve gradually but with the issues we have been seeing.

It looks like there are some misconceptions about the situation, based on the comments posted here.  This is not an issue of the FAH servers not being powerful enough -- any server network would be showing this same issue with the client issue that's going on (in fact, the FAH servers serving the PS3 are more powerful than in other parts of FAH, which is operating just fine with far more clients); indeed, the PS3 backend has been spec'd out to handle 1M PS3 clients and we are way under that.  Also, this is also not an issue that the PS3's are too fast.  The server load depends on the number of clients and the amount of bytes they send back; in FAH, the PS3's compute faster, and that is used to do more complex calculations, not to send more bytes, and thus does not create a greater server load.

The client update would fix how the client interacts with the servers to stop the problem we're seeing right now, including the issues with assigns (getting new WU's), accepts (returning WU's), and points.  Sony is a large company and the development team likely cannot publicly give out ETA's on when this will be fixed, etc, but it's important to stress that they are working on this and know this is a very, very critically important update to make, and they are working aggressively to fix it.

Comments

[Hank Page]

How can you say with certainty that this is a client issue? The v1.3 client had been running successfully on my two PS3's for several weeks prior to this issue surfacing.

[ploeg]

From reading previous status reports, I gather that the issue is that the client is not backing off on transmissions during high-traffic periods like it is supposed to do. It's like a chain-reaction auto pileup in a fog bank. If cars slow down when approaching the accident, it allows the mess to be cleaned up with minimal disruption to traffic. If cars continue at the same pace when approaching the accident, it doesn't matter how wide the highway is, the pileup is only going to get worse.

The issue did not manifest itself for a month because the specific condition that caused this issue did not occur until a week ago.

[Stiffy]

Thanks for the updates. I am sure that you are as annoyed with this as we are. The people that are moaning about servers etc surely have no idea on what they are talking about.

Looking forward to Sony putting something on their site about this in the next day or so.

[sofong]

Hopefully deadlines can be extended until the Work Units can be uploaded?

[Hazzzard]

So i dont understand, shall we continue folding or stop complete ? i dont wanna fold days after days and get 0 credit.

[smASHer88]

thanks for keepin us posted.. no-one minds dealing with a couple of issues.. but its great when u get feedback on it so thanks again.

[The Server Surfer]

Stanford stated that they are extending deadlines, so yes, keep folding.

[dylan]

make all cameras able 2 download pic on the ps3 thanks

[Matt Johnson]

Hi,
wonder what the latest update to this one is. It's a longtime ago that someone wrote a respond or gave feedback if it is fixed or not.

--Matt

[PS3 Game Online]

Nice post, I enjoy a lot playing games actually, especially the latest console tech like the PS3. IMHO, PS3 is way better than Wii or XBox 360.