What I'm leaning towards now is making two sets: one for archiving and one for distribution. The archival images will be 600dpi greyscale TIFFs. They will not be converted to pdf, but just stored as TIFFs. The images intended for download and distribution will be 200dpi greyscale JPEGs. Using these, I expect a 128-page download to be about 20 MB. While perhaps not suitable for ocr, these images are very comfortable to read on-screen, and can later be replaced with improved versions made from the original 600dpi TIFFs. I know a lot of you expressed concerns about JPEGs, but I haven't been able to get anywhere near the compression using other methods, for greyscale images. Am I overlooking any options? Tom Applefritter www.applefritter.com

On Sun, 2004-06-27 at 17:59, Tom Owad wrote: I tend to do that too. I'd much rather use my favourite image viewer to flick through images than deal with a pdf file, plus I can just put everything in a tar or zip archive if I do need to distribute as a single file. I've not reached a conclusion for what the best format is for OCRed data which may contain text and images. RTF is maybe the most portable, but no doubt it doesn't handle embedded imagery. MS Word is just a nasty format and none too portable either. I never thought I'd say it, but maybe wrapping the data up in *simple* HTML markup is the best way - at least then it is readable in a plain-text editor, and finding a machine with a web browser is probably easier than finding a machine with Word installed. Actually, I suppose seperate images can help here too as people can navigate straight away to what they want, plus they don't need to download the whole of a huge pdf file before they can start reading. Probably not. JPEG is lossy after all, so I expect it'll always do better than a non-lossy format. It's a tradeoff between size and quality - personally I'd always go for the quality for historical data. Actually, the BBC Documentation Project for the old Acorn 8 bit machines at http://www.bbcdocs.com is worth a look. They seem to do quite well in terms of getting both scanning and OCR volunteers. Only problem is that you need to be *really* sure that your OCR versions are good before you can risk taking the raw scans offline, which means having a lot of people doing a lot of proofreading. But OCR data will typically be a fraction of the size of raw image scans, regardless of what resolution / format you hold images in. Just keep raw image scans in an offline archive then just in case. cheers, Jules

Ugh! Please, no! For pure text, use - pure text! HTML is maybe appropriate for serving it up on the Web, but even there, I'd much prefer text/plain for something that is indeed plain text. Well, _I_ can't. Or at least if I can I have no idea how. GhostScript (the only open-source PDF reader I know of) doesn't seem to have a plain-text output option. /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse(a)rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

HTML is just text and, if push comes to shove, is easy to discard with a simple matter of programming. For anything else, (TIFF, PDF, whatever) more programming effort is required to come up with searchable text. The main advantage of html (as I see it) is that you can keep the flow of text and pictures roughly as it was meant to be (rather than having text full of "see figure 1"). Well the British Library are spending a good deal on digitising all their old stuff (Magna Carta, that kind of thing). They are (apparently) taking high resolution photographs (at resolutions sufficiently high that you would not dare say "600 dpi" in the same room) and making them available on the net (eventually). I don't know if they are using digital cameras (not your average high street jobbies here!) or using high quality film cameras and then digitising. Either way, if they are going to all that trouble for their stuff, surely we can do the same for stuff that really matters :-) Antonio -- --------------- Antonio Carlini arcarlini(a)iee.org

Woo. Cool. I wouldn't've tried it directly, because "ps" doesn't say "PDF" to me. (I also didn't know it existed; when I went looking for PDF converters, I looked for "*pdf*", which of course doesn't find ps2ascii.) Thank you very much. I suspect this will come in handy for more than just what I originally wanted it for. /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse(a)rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

Hi all, Since I will be in the U.S. during VCF/E, I might as well try to actually make it there. Is anyone driving there from the Silicon Valley area? Cheers, Fred

On Mon, 2004-06-28 at 14:56, Paul Koning wrote: Except, as said, not readable on a platform that doesn't have a PDF viewer (or has a broken/slow/old one). IIRC, older versions of the Acrobat viewer wouldn't even let you selectively copy and paste ASCII text - I don't even know if the current versions do that. Plus I don't know if I can run a PDF file through my favourite search tool - certainly the few I've just looked at appear to be unreadable garbage. If this wasn't a classic computer list *and* we were all expected to have PCs running MS software then I'd agree that PDF might be a valid choice. I'm just not so sure it's that simple in this case... cheers Jules

Personally? Because it has all of the disadvantages of PostScript (not easily usable with plain-text tools, mostly) and more (usually the whole thing is comparessed so it's not at all usable with plain-text tools). Because (when the PDF is just a bunch of images, as with most scanned manuals) it's often relatively difficult to extract individual pages' images to do anything useful with them. Because it's annoying to have to print it out in order to get something readable without strain (at typical font sizes, characters are only ten or so pixels high, and reading text at that size when the font has not been bitmap-tuned for it, is a strain). Now that I've been pointed at ps2ascii (which contrary to its name does work on PDFs), some of the disadvantages of PDFs are alleviated - for PDFs that contain text as opposed to image-per-page scans.... /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse(a)rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

I have a 2.8GHz Intel box running Debian at work and that works quite nicely (firefox and xpdf IIRC). I also have a 450MHz Intel box running Windows and Acrobat 6 on that runs like something that died quite a while ago :-) For anything current, I expect all three to be available. For anything classic, you cannot expect more than a text editor to be around. But I don't expect to read schematics on a WT78! The Adobe stuff you pay for (although the reader is free on a few platforms). The specifications are freely available and I've not seen anything that says you cannot implement your own reader. Perhaps they want you to pay if you implement a writer but I expect not since Eric Smith's tumble is free. You can edit a PDF but not quite in the same way as you would edit a word processed document. You can replace characters within a line but there seems to be no "wrap" to the next line and so on. It's OK (if somewhat painful) for the sort of OCR-error-correction that you have in mind. Obviously you cannot edit images (without extracting them, mangling in an image editor and re-inserting). I've scanned bucket loads of docs (small buckets compared to others, but enough to fill 10CDs or so - probably 100 docs). That was not too bad with an automated sheet-feeder and a decent scanner that would drop a file onto a share. A bit of post-processing (convert to G4, check for missing pages) and you are done. Even that probably took a day per doc on average. Throw in OCR and you would magnify that by maybe a factor of three. Throw in proof-redaing and you'd go mad. It's trivially easy to miss errors so you would really have to have each doc verified by multiple people. And then you need to measure their error rate, perhaps by injecting deliberate misatkes (did you see it, or have you had to go back to look) and verifying that they spot them. On top of that, if you go to all the trouble of scanning stuff and verifying it to a certain extent (and putting together a checksum in case something goes wrong), surely you need to have some kind of backup for when the disk dies? (I've spent the last 24 hours recovering a dead 40GB drive and the previous 24 hours - while waiting for the replacement to arrive - wondering just how dead it was and how much I'd not yet backed up ...). I keep everything I've put any real effort into on two CDs and I try to get it hosted on at least one site somewhere. Antonio -- --------------- Antonio Carlini arcarlini(a)iee.org

The place I worked in previously upgraded from 3 to 4 (because of the support for a new PDF spec I think) and there did not seem to be much difference in speed. I forget why I went to Acrobat 5, but that seems to be not too much slower than 4 at starting (at least for the reader, I've never compared full products against each other). Acrobat 6 is just such a stunningly slow starter, it's amazing that noone at Adobe noticed. Yes. This has always been true I think. It is (apparently) possible to prepare PDFs in some way that avoids this (i.e. displays first page as it arrives and downloads the rest in the background) but since I almost always want to keep a copy around I mostly go with the right-click approach. Antonio -- --------------- Antonio Carlini arcarlini(a)iee.org