A few of you might have noticed some recent problems with the classiccmp server ;)
The machine has six 300gb hard drives. Two are PATA, and set up as a mirror that contains
just the OS and mailing list. The remaining four are sata and set up as a zfs raidz pool.
One of the two drives in the OS mirror failed. That did not cause downtime of course, but
I noticed it when the other failure occured.
One of the drives in the raidz set (websites, user data, etc.) also failed. Because
it's raidz, that shouldn't have caused a problem. However, it marked the raidz set
as failed and wouldn't mount it, saying there was not enough drives left in the set. I
can only guess this was a bug in zfs on that system.
Thanks to hard work from Ryan, the machine is back up. Basically he pulled dd images of
the remaining drives in the zfs set, imported them into zfs on one of his machines and the
data re-appeared. A two drive mirror was set up on the remaining working drives and then
all the data copied back. This took a few days due to the volume of data and network
links. I thought that was the end of the problems. But no......
Once the machine was back up, someone said websites were up but no mailing list (I had
verified that it was back up after the data re-import). Checking again, now the last drive
in the OS mirror is having read errors and is probably going to fail soon. No other
systems in our datacenter are having issues like this, so I suspect it's not power
related.
Long story short, the machine is back up, all website/user data is restored, and
there's no loss of data. However, I can't be sure it will stay up with an OS hard
drive mirror set where one drive is dead and the remaining drive is having lots of read
errors.
My plan is to go buy six new sata drives for the server, probably 1tb's, plus a 4 port
sata controller. I'd prefer to buy the drives new rather than take drives as donations
just due to the above issues and time constraints. I found some seagate sata3 1tb 32mb
cache drives for $60 each plus tax. Adaptec 1420sa controller is about $90, so the total
is about $450. If anyone cares to donate to help cover the purchase, paypal jwest at
classiccmp.org
Now I just have to find another machine to move the list & web content off to, rebuild
the classiccmp server, and move everything back :)
Jay