The caucho server went down last night. Argh. Couldn't access it remotely, so went to the colo. Still couldn't get onto it, and rebooting didn't help. Looked like the drives had failed or the SCSI controller was bad. Took the system back to my office to work on it.
After lots of testing of parts and swapping with another system configured exactly the same, I couldn't get anything to boot. Called SuperMicro for tech support and after further testing, they confirmed the problem was with the drives, not the SCSI controller, cable, or backplane.
Called Caucho and told them about the problem. Since both drives were bad, my assumption is that one drive failed a while ago and the other one continued to work since the server had mirrored drives. Then the 2nd drive failed last night. They didn't have any backup service on their server, and it sounded like they didn't do backups themselves. So I took the drives out of the system and shipped them for Monday delivery.