A machine went down last night, which brought down vspg1, vspg1v, and vspg1v2 (these are all different virtual interfaces on the same physical server). Our admins (working during the holidays -- thanks to them for their hard work) reset the machine this morning and we got the machines going again.
While vspg1* were down, this lead to a much greater load on other machines, causing issues for vspg2v2, vspg4v2, and vsp07v. These reset themselves some time around 7:30am pacific time (with the exception of vsp07v, which I reset).
So, it looks like the server farm is settling down. I can't wait until we get the new, really powerful machines loaded up with jobs to help take over some of this load. Some of the jobs being tested right now are on those machines, so that should be very soon.
Note that during the holidays (especially the next week or so), we have a limited staff in the office, but those that are working are diligently looking for problems. Happy Holidays!
