On Sat, 29 Jan 2005, Jules Richardson wrote: Wow. My 500MHz laptop renders postscript just fine. There's a "noticable" render time (eg. it's not instantaneous) but it's hardly an impediment to reading. (I've got a Sony VX89, 512M RAM, 500/900MHz CPU (acpi) nice screen etc, I put up with it for its 6 - 9 hr battery life.) freeBSD now, SuSE linux then. Haven't tried a DOS/Win port of it in a long time though. It's still useful, but it's being perverted as a "read only" no-fair-use medium for DRM purposes, on Windows anyways. It's misused a lot too. Tom Jennings <tomj(a)wps.com>

From: "Scott Stevens" &lt;chenmel(a)earthlink.net&gt; <snip> I don't agree, Adobe needs to make some improvements. One I really want is compressed files. Other formats have great compression ratios when compared to PDF's. I post PDF's because that is the defacto standard but I want something that lets me post more on my site without costing me more. Randy www.s100-manuals.com

From: "Eric Smith" &lt;eric(a)brouhaha.com&gt; but more. DjVu has a hugely more efficient scheme for their files, DjVu files of the same document are a fraction of the size of PDF's. I don't want to switch to DjVu, partly of the legal tug of war going on with it, partly because I already bought Adobes Acrobat, partly over the fact it would confuse too many users unfamiliar with it. Randy www.s100-manuals.com

Eric Smith wrote: To be fair, DjVu lossy-encodes the graphics, not the text. One of the "selling points" of the format is that B&W text is kept on its own lossless layer. Agreed. Heh, also agreed :-) -- Jim Leonard (trixter(a)oldskool.org) http://www.oldskool.org/ Want to help an ambitious games project? http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/

From: "Jim Battle" &lt;frustum(a)pacbell.net&gt; Sent: Monday, January 31, 2005 8:14 AM <snip> For me I looked into DjVu but like Jim I decided it is not practical to only post DjVu. I started with a simple goal to post the manuals of greatest interest to me. I found a site that had similar goals (Howard Harte's manual site) and decided to send my manuals to him and mirror his site. I also mirror other sites I consider important such as Jim's SOL-20 Archive and post other files not part of my mirrors. After some thought I signed up for a 3gb site with a 25gb/mo band width. Before I knew it Howard's site exploded to over 3.5gb. I faced with the choice of going to DjVu or putting out more cash to keep the files as PDF's. I ended up purchasing another 4gb site with 75gb/mo bandwidth that I hyperlink to transparently giving me 7gb, I just hope it is good enough to last a year. I use PDF's simply because it works and it is the defacto-standard. It is not a perfect answer but it is the "best" answer, it gives access to the majority. Anyone unable to handle either the large file sizes or PDF formats have other options: They can talk a friend into downloading it and converting to PS or printing, pay a office supply/cyber-cafe to do it, check with Herb Johnson. I provide the PDF's for free but if someone has to pay someone to print them they are at least available. Randy www.s100-manuals.com

Alternatively the existing hobby sites might become the latest collateral damage in the war being actively fought by various DCMA-enforcing bodies. Antonio -- --------------- Antonio Carlini arcarlini(a)iee.org

Eric Smith wrote: Eric, the level of differences it allows aren't anything so gross: gee, these are both serif fonts, so they are close enough. No. When you scan to bilevel, exactly where an edge crosses the threshold is subject to exact placement of the page, what the scanner's threshold is, and probably what phase the 60 Hz AC is since it to some degree couples to the lamp brightness (hopefully not much at all, but if you are splitting hairs...). Thus there is no "perfect" scan. If you are scanning as such a low resolution that two "e"s from different fonts might get confused with each other, your OCR attempts will be hopeless as well. lossless from what? scan your perfectly clean page with the most expensive scanner you can afford, then reseat the page and do it again, 1000 times; no two are going to be exactly the same. So lossless in this context means perfectly preserving a known imperfect image. Realizing that there is an error floor, allowing other substitutions that are within that error bound to get a 3x compression improvement sounds like a great algorithm to me. If you scan at a good enough resolution (300 dpi for any normal sized text, I doubt it makes one whit of difference. As an aside, I worked four for years at a company that ended up merging with what was Caere and is now softscan. If Bob Stek is still subscribed, he will remember the company. He used the Calera Recognition Systems scanners to scan all of the Sherlock Holmes books and made the first electronic archive of all of the stories. What I knew about OCR is now 10 years past, so perhaps things have changed radically, but I doubt it.

Eric Smith wrote: I'm here completely with Eric. However, probably we should distinguish how we actually scan the stuff, and how we distribute the scans. As the most work is anyway in setting up the scanner, name the files, check if all pages are there, etc. I don't like to do it twice, so I scan at least 300-400 dpi, most of the time with two bits per dot/pixel. And put it on DLT as an original, then play with what I got. And, every few years, I even check if the OCR is good enough, or still not, ... Cheers

Randy wrote: As I stated in another post, my comments are only intended to influence ongoing archival efforts, and are not in any way intended to criticize the efforts of people who are scanning documents and making them available. I have found several of the documents on your web site to be very helpful. Thanks for providing them! Even though I don't like lossy compression applied to text and line art, and try to discourage it, a lossy-compression DjVu file of a document is MUCH more useful than not having the document at all. I might try to draw the line at JPEGs of text and line art though. They literally hurt my eyes (by causing eye strain) due to the bluriness of the edges. Still, if it's a document I really need I suppose I'd rather be able to get the JPEG than nothing at all. None at all. Anyone that thinks otherwise is a fool. Eric

Eric Smith wrote: "potentially" is the key word. if the encoding software is crappy, then they such a substitution could turn all "e"s into "x"s. sure. but the djvu encoder doesn't make gross substititutions like that. Contrary to what you say, skew has a much larger effect on the sampling than djvu's encoders have. Which scanner you use has a much larger effect on the sampling too. ... Eric, in picking a case where the djvu algorithm *might* cause problems, you must also confess that in this case scanning in bilevel, even lossless, is going to be a bad choice too. If the page is that poor, you should be using grayscale. Why be religious about lossiness and claim anything less is going to "f*&#" up your efforts when you've just tossed away the bulk of the information? "A little" can be 3x. For distribution, it is a big deal. Until recently, it made a signficant difference on disk price too, but now that you can get 120 GB hard drives in a box of cereal, that isn't so much of a concern. Of course you can use whatever format you want for your archiving. Making it available in a more accessible format means that more people are likely to take advantage of it. For most documents, it is the information that I care about preserving, not the pixels. I would be overjoyed if Adobe would buy out lizardtech and adopt some of their technology, even the lossy bits.

Antonio Carlini wrote: yes, it is information loss. scanning bilevel is a much worse information loss. scanning at 300 dpi, or 600 dpi, or 1000 dpi is information loss. viewing the document on a CRT is information loss. yes, you are right, *if*. And that is where you are wrong to assume it is likely to make a difference. Antonio -- yes, if you assume that the encoder is going to make gross errors, then it is a bad program and it shouldn't be used. but have you ever used it? it doesn't do anything of the sort. imagine a page with 2000 characters, all of one font and one point size, and that 150 of them are the letter "e". In a tiff image, there will be 150 copies of that e, all very slightly different. In the djvu version, the number of unique 'e's will depend on the scanned image, but it isn't going to replace them all with a single 'e' -- there might be 50 'e's instead of 150. Thank about that -- to the naked eye, all 150 look identical unless yo blow up the image with a magnifying tool. djvu is still being selective enough about what matches and what doesn't that it still has 50 copies of the 'e' after it has collapsed ones that are similar enough. It isn't very agressive at all about coalescing glyphs. As far as I know there is a bound on how small of a size it will try to group so that for really small point sizes, nothing bad happens at all. The differences it allows are truly inconsequential. It is like complaining that mp3 (or insert your favorite encoder here) sucks because in theory it can do a poor job of it. In fact, ones that do a poor job get left behind and the ones that do a good job get used.

I wrote: Jim wrote: If so, that's apparently a recent change. When I experimented with the demo of the commercial product about 18 months ago, it definitely was using lossy coding on the entire image. When I used it to encode a bitmap of a page of text (from a TIFF encoded with lossless G4 coding) into a DjVu file, and extracted it again, it was not able to reconstruct pixel-identical output. I xor'd the original with the extracted file, and there were differences at all of the character edges. It is my belief that this is why it was able to store the file in only about 70% of the space occupied by the G4 lossless version. I'm not disputing that the DjVu format is *capable* of doing what you say, but in my experience that's not what the available software actually did. Jim wrote in another message: I'm not sure I fully understand, but it doesn't sound like anything that the PDF format can't support. I would rather invest effort into improving the capabilities of free PDF viewer software such as xpdf rather than pushing a different standard. I've been doing it myself with an experimental version of my "tumble" program. With that, it's an entirely manual process. I have to split the continuous-tone images into a separate layer or file using a separate editor (such as Gimp). Then I use tumble to compose a page with the background as G4 and the images as JPEG. I also am experimenting with using this for pages that are only text and line art, but that have a few colors. For instance, DEC manuals that had user input printed in blue. I separate the blue text into another layer, and G4 encode it separately from the black text. PDF has an imagemask operator that can be used to draw a blue rectangle the size of the whole page, but clipped to the blue G4 image. Acrobat reader handles this correctly, but some other PDF processing programs do not. I hope to automate the multi-color text problem in tumble using code derived from Tim Shoppa's "timify.c". Automating the detection and processing of continuous tone images is in my plans as well, but further out. There don't seem to be any good published algorithms for image detection, so I'll have to experiment with it. As a first step I plan to do 2D FFTs on areas of the page; text and line art should predominantly have DC and high frequencies, while continuous-tone images should have a more even frequency distribution. So far I'm doing this work by scanning a page twice, once in bilevel and once in greyscale or color. I do that because the published algorithms for converting greyscale text and line art to bilevel (thresholding) are nowhere near as good as what's done in a good scanner. Picture Elements makes a PCI card that can be used to do this (and even works with Linux), but it's very expensive so I really want a software-only solution. But the experimental version of tumble is still very buggy and not yet publicly released, so it won't help you any. Acrobat can do the manual approach as well, if you use a separate editor to split the page into layers, and import each layer separately. It appears that there are some expen$ive programs that do such things in a more automated fashion listed on pdfzone. Also I'm told that some OCR packages can do this, but I haven't verified it. Eric

Personally, I'm not among the "I hate PDF's" crowd. I first purchased a copy of Acrobat to create my own PDF's 9.5 years ago, and have upgraded a couple times since that time. I'm now running Acrobat 6.0 Professional, and while I'm not fond of DRM in all of it's aspects, I do like the fact that I lock down a PDF. I recently wanted to send something to someone relating to a non-computer related project I'm working on, and due to what it is, I don't want anyone being able to modify the file, but I don't mind them printing it out, I was able to do this with PDF. Another big advantage for me is that the file was created with an application that is 10+ years old, and very few people can read the format, while almost anyone can read the PDF. One thing I find very handy is that applications such as Adobe Illustrator and Photoshop can now open PDF's. If the PDF contains vector graphics, you can edit them with Illustrator. This is the first good method I've found for moving a lot of my data to something modern. It's interesting to note that a site that sells DRM'd PDF's of old RPG books for a lot of major companies has started offering "watermarked" PDF's that can be copied anywhere rather than the DRM'd ones that can only be used on one computer. Zane -- -- | Zane H. Healy | UNIX Systems Administrator | | healyzh(a)aracnet.com (primary) | OpenVMS Enthusiast | | | Classic Computer Collector | +----------------------------------+----------------------------+ | Empire of the Petal Throne and Traveller Role Playing, | | PDP-10 Emulation and Zane's Computer Museum. | | http://www.aracnet.com/~healyzh/ |

der Mouse wrote: Well, where do you stop? DRM in video cards so that you can't take a screen capture? DRM in monitors so that the picture blanks out when you turn your head? DRM in the human optic nerve? I mean, come on. PDF is great for it's intended purpose: Creating portable documents that can be viewed and printed regardless of the resolution of the output device. A PDF I create with vector line art looks great on my 10-yr-old 300 DPI postscript printer, but on a 1200 DPI printer of today it is flawless. I have seen a lot of misuse of PDF (scanning something and NOT OCR'ing it, turning the PDF into a 100MB container for JPG files) but, overall, PDF is pretty darn good. -- Jim Leonard (trixter(a)oldskool.org) http://www.oldskool.org/ Want to help an ambitious games project? http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/

I don't view PDF's as limiting my audience, I would view TIFF's as limiting my audience. People here need to stop and remember that they're not typical computer users. Shoot, some computer users have enough trouble reading PDF's, do you really think they're capable of handling other formats? Zane -- -- | Zane H. Healy | UNIX Systems Administrator | | healyzh(a)aracnet.com (primary) | OpenVMS Enthusiast | | | Classic Computer Collector | +----------------------------------+----------------------------+ | Empire of the Petal Throne and Traveller Role Playing, | | PDP-10 Emulation and Zane's Computer Museum. | | http://www.aracnet.com/~healyzh/ |

Jules Richardson wrote: You must be talking from the position of a *nix user. In the Windows world -- where, like it or not, much of our newbie target audience is -- it is nearly impossible to view a .ps but is almost universally possible to view a .pdf. That is one of the MISuses of PDF. PDF should not be used as a container for bitmap images. PDF *should* be used for text-only documents, or documents lightly mixed with graphics (preferably resolution-independant vector/line-art). In case it wasn't obvious, PDF *is* Postscript! It's *portable* postscript. If the scanned pages cannot be OCR'd then I completely agree. And it is at this point that I believe I am arguing for PDF for a poster that is not yourself :-) I should know better than to jump into threads where I can't see the top post ;-) The best format for mixed text and graphics that do NOT need to be indexed (ie converted to text) is DjVu ("DejaVu"). DjVu is a mixed-type format that keeps a B&W image of the text and a color or grayscale image of everything else in the same file, and each layer is compressed with a method appropriate to their content type. Unfortunately, the best DjVu tools cost significant money, so it hasn't taken off. I haven't used it beyond a few experiments myself. -- Jim Leonard (trixter(a)oldskool.org) http://www.oldskool.org/ Want to help an ambitious games project? http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/

der Mouse wrote: That is the fault of the person who created the PDF, not a fault of the format itself. I'm sorry, do you normally post about things you haven't researched? -- Jim Leonard (trixter(a)oldskool.org) http://www.oldskool.org/ Want to help an ambitious games project? http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/

Eric Smith wrote: I maintain that PDF should not be used merely as a container for existing graphics files because there is normally no easy free way to extract the image data and use it in another program. I know that it *can* do it, but the majority of users who do this screw it up massively (I'm thinking 150 DPI JPGs of scanned text). I am familiar with the internals of PDF as well, which is why I wrote portable. Portable does not imply complete. Perhaps I shouldn't have used the "*is*" emphasis... DjVu has other advantages, such as local/window/viewport decoding of images with ludicrously high dimensions/resolutions but I understand your point. Where are the tools to create DjVu-like PDF files? The best Acrobat can do is OCR text but still leave the source bitmap in place... If I scan in a page with a background color image with B&W text foreground, where are the PDF tools to properly handle layer seperation? (Not CMYK seperation, you know what I mean :-) -- Jim Leonard (trixter(a)oldskool.org) http://www.oldskool.org/ Want to help an ambitious games project? http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/

But not really any different from PostScript in that regard. At least not if you think of the Document Structuring Conventions as mandatory. And a PostScript file ditto, ditto. /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse(a)rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

Personally? Only ones with Adobe Acrobat Distiller, that way they can be turned into PDF's that can then be viewed and printed :^) Zane -- -- | Zane H. Healy | UNIX Systems Administrator | | healyzh(a)aracnet.com (primary) | OpenVMS Enthusiast | | | Classic Computer Collector | +----------------------------------+----------------------------+ | Empire of the Petal Throne and Traveller Role Playing, | | PDP-10 Emulation and Zane's Computer Museum. | | http://www.aracnet.com/~healyzh/ |

As long as the interface is documented enough for open-source support, and the bugs aren't in the scanner itself, I'm fine with that. If I caught my printer doing that, I would tell them to fix it. If they tried to claim it was a feature, or otherwise refused, I'd return it and find a printer that printed what - and nothing but what - it was given to print. /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse(a)rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

On Sun, Jan 30, 2005 at 11:11:02PM -0800, Zane H. Healy wrote: Maybe someone could write custom firmware for an existing printer? -- Joshua D. Boyd jdboyd at jdboyd.net http://www.jdboyd.net/ http://www.joshuaboyd.org/

Hi, Seems like going about things the wrong way to me. Here in the UK when high quality laser printers came out they added a security metal strip to the paper in the bills and changed the pattern and colours of ink used for ones which are pretty much impossible to copy accurately. All new printer designs are tested by the Treasury to see if the technology has advanced enough to make good copies. In our firm with a weekly cash turnover of over $1 million we used to get a few forgeries, but they were easy to spot. (Are your bills still all the same size, or have you copied everyone else and made the larger dominations bigger to stop people bleaching small bills to print big ones?) -- Cheers, Stan Barr stanb(a)dial.pipex.com The future was never like this!

Not quite everyone else. Canadian paper money is still all the same size independent of denomination. We have, however, recently started issuing new $20 notes (and presumably higher denominations) with metallic inks and raised printing and watermark pictures and such that are approximately impossible to do with consumer-grade printers. /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse(a)rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

On Sun, 30 Jan 2005, Zane H. Healy wrote: But that isn't true! I can trivially take apart PDFs. I cut'n'paste for fair-useish purposes routinely. "Don't want" people to take files apart, I agree with you, a well-crafted document becomes a mess. But if someone is determined, etc. It makes for a nice disclaimer for you to not support it post-mess, but it doesn't and can't actually stop anyone. It's just (viewable as) a byte stream, after all. Tom Jennings &lt;tomj(a)wps.com&gt;