21 Feb
2007
21 Feb
'07
1:45 p.m.
Google has a very vast herd of machines with a
large number of hard drives. Very fruitful
that they analyze failures and publish
the results to the web!
While the drives they are studying are definitely
not classic (all dating from 2001 or later),
those of us who host large quantities of classic
material may find the results of interest:
http://labs.google.com/papers/disk_failures.pdf
Side note: at one point I found it unbelievable
that Google was using consumer-grade hardware
to host their stuff. Since then, I've developed a
lot of respect for this approach!
Tim.
21 Feb
21 Feb
2:02 p.m.
There's an even *more* interesting paper on this from a CMU post-doc
(math warning...lots of statistics) that seems to blow away many (if not
all) assumptions about drive failures/quality/raid/etc.
It was presented at the same conference that the Google paper was. I'll
see if I can't dig up a URL to it.
Tim Shoppa wrote:
Google has a very vast herd of machines with a
large number of hard drives. Very fruitful
that they analyze failures and publish
the results to the web!
While the drives they are studying are definitely
not classic (all dating from 2001 or later),
those of us who host large quantities of classic
material may find the results of interest:
http://labs.google.com/papers/disk_failures.pdf
Side note: at one point I found it unbelievable
that Google was using consumer-grade hardware
to host their stuff. Since then, I've developed a
lot of respect for this approach!
Tim.
--
TTFN - Guy
2:14 p.m.
At 01:02 PM 2/21/2007, you wrote:
There's an even *more* interesting paper on this
from a CMU post-doc (math warning...lots of statistics) that seems to blow away many (if
not all) assumptions about drive failures/quality/raid/etc.
I saw this a few days ago but thought I shouldn't post because
it would be off-topic.
Here's the CMU paper:
http://www.usenix.org/events/fast07/tech/schroeder/schroeder_html/index.html
And here's a summary of Google's results:
http://storagemojo.com/?p=378
Both examinations dethrone many rules of thumb. MBTF isn't that meaningful,
there's little difference in longevity between consumer and "enterprise"
class drives, beliefs about infant mortality, heavily used drives
don't fail much more often than little-used drives, SMART isn't, etc.
- John
23 Feb
23 Feb
2:42 a.m.
On 2/21/07, John Foust <jfoust at threedee.com> wrote:
At 01:02 PM 2/21/2007, you wrote:
Indeed...and I was criticized for posting a OT thread on this list before
and here comes a OT thread that is openly discussed...what gives??? Why was
I flamed while this guy is not? I just don't get it...:(
There's an even *more* interesting paper on
this from a CMU post-doc
(math warning...lots of statistics) that seems to blow
away many (if not
all) assumptions about drive failures/quality/raid/etc.
I saw this a few days ago but thought I shouldn't post because
it would be off-topic.
- John
10:34 a.m.
Hexxie wrote....
Indeed...and I was criticized for posting a OT thread
on this list before
and here comes a OT thread that is openly discussed...what gives??? Why
was
I flamed while this guy is not? I just don't get it...:(
One might view a discussion of modern hard drives in and of itself as being
off-topic. However in this case I believe that this thread came about due to
a discussion of preservation of historical data, which would necessitate
discussion of modern hard drives. Many of us are preserving old software and
of course are interested in the best ways to do so using whatever means
necessary and need to know how long into the future that method may be
considered reliable. Thus a brief discussion of modern hard drive failure
rates in that light is on-topic IMHO.
Jay
10:48 a.m.
On Friday 23 February 2007 02:42, Hex Star wrote:
Indeed...and I was criticized for posting a OT thread
on this list
before and here comes a OT thread that is openly discussed...what
gives??? Why was I flamed while this guy is not? I just don't get
it...:(
Whining about the inequities of life is off-topic.
Pat
--
Purdue University ITAP/RCAC --- http://www.rcac.purdue.edu/
The Computer Refuge --- http://computer-refuge.org
21 Feb
21 Feb
11:26 p.m.
Tim Shoppa wrote:
Side note: at one point I found it unbelievable
that Google was using consumer-grade hardware
to host their stuff. Since then, I've developed a
lot of respect for this approach!
Quantity over quality. If you have enough money to get that off the
ground, you win in the long run.
Related note: All throughout the early to mid 1990s, Pixar standarized
solely on SparcStation 20s for their renderfarm. Why not a more
powerful system, or a cheaper system? Because they claimed, due to form
factor, it was the most number of CPUs they could fit into a room.
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Help our electronic games project: http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
A child borne of the home computer wars: http://trixter.wordpress.com/
22 Feb
22 Feb
12:25 a.m.
On 21 Feb 2007 at 22:26, Jim Leonard wrote:
Related note: All throughout the early to mid 1990s,
Pixar standarized
solely on SparcStation 20s for their renderfarm. Why not a more
powerful system, or a cheaper system? Because they claimed, due to form
factor, it was the most number of CPUs they could fit into a room.
I seem to recall an article that mentioned that the effects on
"Titanic" were rendered using a room/closet full of Peecees (Dell, I
think)... Something about the hardware (running Linux, IIRC) cost
less than $50K.
I may have all or some of this wrong, but I think the article was
published in IEEE Computer some years back.
Cheers,
Chuck
12:55 a.m.
Chuck Guzis wrote:
I seem to recall an article that mentioned that the
effects on
"Titanic" were rendered using a room/closet full of Peecees (Dell, I
think)... Something about the hardware (running Linux, IIRC) cost
less than $50K.
Not sure about that, but the first Matrix film "bullet time" effects was
done on FreeBSD and a PC renderfarm.
The third Matrix film used ridiculous amounts of cash with third-party
effects houses, some of which were put out of business by the endeavor
(not sure how that works).
--
Jim Leonard (trixter at oldskool.org) http://www.oldskool.org/
Help our electronic games project: http://www.mobygames.com/
Or check out some trippy MindCandy at http://www.mindcandydvd.com/
A child borne of the home computer wars: http://trixter.wordpress.com/
1:15 a.m.
I seem to recall an article that mentioned that the
effects on
"Titanic" were rendered using a room/closet full of Peecees (Dell, I
think)... Something about the hardware (running Linux, IIRC) cost
less than $50K.
160 x 433Mhz Alpha's for the rendering farm.
See:
http://www.linuxjournal.com/article/2494
Martin
10:01 a.m.
Martin Marshall wrote:
There used to be another good write-up about this on the web, but
unfortunately I don't seem to have the URL any more to even plug into the
wayback site.
I recall using it as evidence of Linux being used to solve real-world problems
back in the 90s, when trying to win some people over to increased Linux usage
at the place where I worked then.
I seem to recall an article that mentioned that
the effects on
"Titanic" were rendered using a room/closet full of Peecees (Dell, I
think)... Something about the hardware (running Linux, IIRC) cost
less than $50K.
160 x 433Mhz Alpha's for the rendering farm.
See:
http://www.linuxjournal.com/article/2494 
7:06 p.m.
I worked in the disk industry for several years (Fujitsu, Maxtor, and on
the silicon side at Cirrus Logic.) and Google's findings are not
unexpected.
One of the major disk drive manufacturers (Quantum?) discovered a problem
with their drives on Unix systems. Since Unix doesn't tend to make
spurious reads and writes while sitting idle (Windows does it a LOT), the
heads would sit in the same place for long periods of time, eventually
pushing the lubricant aside, resulting in a head crash. The manufacturer
countered this problem by running a butterfly seek every few minutes to
level out the lubricant.
In any event, spinup/spindown events are hard on the head/disk interface,
and temperature cycling causes expansion/contraction which stresses
marginal solder joints.
I would expect very long lifetimes from hard drives.
Most modern drives use fluid bearings for the spindle so, aside from
degradation of the oil, they will last forever. Major IC manufacturers
have a good handle on electromigration issues, so the IC's are unlikely to
fail.
Drive firmware has improved as well. Most drives will detect a block going
bad (high correctable error rate) and move the data to a spare block.
Unless you issue special commands to the drive you don't even notice the
error and relocation occurred. It is a good idea to read everything on the
drive occasionally so bad spots can be fixed before they corrupt the data.
Enough rambling,
Clint
On Wed, 21 Feb 2007, Tim Shoppa wrote:
Google has a very vast herd of machines with a
large number of hard drives. Very fruitful
that they analyze failures and publish
the results to the web!
While the drives they are studying are definitely
not classic (all dating from 2001 or later),
those of us who host large quantities of classic
material may find the results of interest:
http://labs.google.com/papers/disk_failures.pdf
Side note: at one point I found it unbelievable
that Google was using consumer-grade hardware
to host their stuff. Since then, I've developed a
lot of respect for this approach!
Tim.
6359
days inactive
6361
days old
11 comments
11 participants
participants (11)
-
cclist＠sydex.com -
cctech＠porky.vax-11.org -
ggs＠shiresoft.com -
hexstar＠gmail.com -
jfoust＠threedee.com -
julesrichardsonuk＠yahoo.co.uk -
jwest＠classiccmp.org -
martinm＠allwest.net -
pat＠computer-refuge.org -
trixter＠oldskool.org -
tshoppa＠wmata.com