On Wed, 1 Jun 2005, Eric Smith wrote: D'OH!! :-) I recently saw the most impressive PDF I've ever seen -- on www.mscdirect.com, a catalog page -- presented as PDF -- had hyperlinks on product catalog numbers to the "shopping cart" mechanism. QUite nice. While I do believe PDFs are often abused and partially deserve their bloated reputation (Adobe pushes them too much) they can be used properly as containers for multiple-component documents. It's unfortunate that it's so much work to hand-type or "OCR" old documents to produce PDFs as lovely as MSC's. I've offered to help out with automation to produce PDFs or other efficient containers for a thousand or so scanned pages (TIFFs) to be bundled into chapter-sized chunks. (It's an American Motors factory parts catalog -- a year's automobile product line broken down into parts down to the solitary screw.) Does anyone have any references to cookbooks/how-to's on how to go about doing this sort of thing?

The systems I saw about ten years ago were damn good. I assume they have become better, just like OCR. William Donzelli aw288 at osfn.org

On Thu, 02 Jun 2005 04:18:29 +0200, Eric Smith <eric at brouhaha.com> wrote: You do not get my point: Instead of adding an extra step to make PDF files, I think it is better to keep the raw scans. Preferably as PNGs, but if the scanner software does not provide that, TIFF is workable. Thus, those who scan things (a task I really appreciate a lot) do not spend valuable time making the results less accessible, and it will be easy to OCR the images later on. My point is that a pure bitmap is both easier to make and more useable than an Acrobat document, both for archival purposes and for further work. The problem with TIFF is that it is bulky and not a single well-defined format, so it happens that software A will generate a TIFF that cannot be displayed by software B. PNG is a nice, open and flexible format which is designed to be the best cross-platform lossless image format. -- Bj?rn

On Thu, 2005-06-02 15:09:35 -0400, Paul Koning <pkoning at equallogic.com> wrote: Do we actually *have* the tools? We've got tumble to assemble a PDF file, but do we have proper tools to disassemble one? ...and I really mean exporting the initial TIFF, not something that looks like it. MfG, JBG -- Jan-Benedict Glaw jbglaw at lug-owl.de . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

On Thu, 2005-06-02 16:39:58 -0400, Paul Koning <pkoning at equallogic.com> wrote: You didn't answer my question:-) Consider I prepare a TIFF file that contains (with additional tags) eg. some raw OCRed text, not read-checked. Now I preapre a PDF from this and use gs to get the image back. Is my text still there? Or do I get an image that "looks" almost the original, but doesn't contain my extra-data? MfG, JBG -- Jan-Benedict Glaw jbglaw at lug-owl.de . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

On Thu, 2005-06-02 at 16:58 -0400, Paul Koning wrote: It's a pretty cool format for that kind of thing; I suppose like HTML there are a bare minimum of tags which any decoder should support, and should be able to skip over anything it can't handle and still output an image. It tends to be let down by bad decoder code though - in particular decoders typically either can't handle multi-page images (or do so badly), or they don't support all the common compression schemes out there. cheers Jules

Jules> Looking at the PDF file, I'm not convinced there's any TIFF Jules> data in there to be honest. It looks more like the image is Jules> re-encoded from the input TIFF to PDFs own way of storing Jules> bitmap data - in other words it's not simply a wrapper for a Jules> bunch of TIFF images, but merely a wrapper for bitmap data in Jules> PDF's own format. That's something of a disappointment; I Jules> always thought PDF just encapsulated the input images rather Jules> than re-encoding in any way... Makes sense. The purpose of PDF is to distribute final form documents, for viewing and for printing, including commercial printing. So it's like PostScript, but more rigorously defined, and compressed for space efficiency. A PostScript file produced from an image has the pixels, so they can be printed (in the original colorspace, unless your document manipulation program converted it, as some do or can be told to). Given the purpose of these file formats, there's no reason for it to preserve the TIFF metadata, nor the TIFF format, since the intent is to accurately put pixels on screen or paper. The encoding will be lossless, though (at least normally -- I think you can tell Distiller to use JPG style compression but sensible people know better than that). So when you export the image from the PDF, you get the original pixels back. For purposes of scanned document archives, that is what you need, which is why PDF is fine there. paul

Worse, it is probably re-rendering the pixels, so unless it's quite careful, it's introduced blur due to mismatches between the original's pixel boundaries and the output's pixel boundaries. Did you compare the pixel contents bit-for-bit? It's not *quite* that simple. I've seen PDFs containing JPEGs which I could pick the JPEGs out of simply by looking for the \xff\xd8\xff\xe0..JFIF marker. Unless of course the "PDF's own format" *is* JPEG, which would be both surprising and disappointing. /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse at rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

On Thu, 2005-06-02 at 23:25 -0700, Eric Smith wrote: Heh, I'm with you there. Personally I like those who scan schematics and save as GIF - not that GIF is necessarily bad, but they seem to be the same people who apparently move whatever it is they're scanning around the scanner bed during the process, or run some pixel randomiser on the resulting data or something :-) Oh, that or they get the threshold settings totally out so that each line on the schematic ends up a jagged mess as wide as your arm. Urgh. I keep on having to redraw downloaded schematics because the source image is such a mess, and it's starting to be a pain :-( cheers Jules

der> Worse, it is probably re-rendering the pixels, so unless it's der> quite careful, it's introduced blur due to mismatches between der> the original's pixel boundaries and the output's pixel der> boundaries. Is that speculation or have you seen something that supports this notion? Given Adobe's background, I would assume your speculation is false -- Adobe knows this stuff far better than most of us. paul

From: "Jules Richardson" <julesrichardsonuk at yahoo.co.uk> Sent: Friday, June 03, 2005 9:34 AM What gets me is the simple fact that non-Adobe utilities changed a comment field in a tiff -> pdf -> tiff and from that it is taken as gospel that is the fault of the PDF structure. I may look into trying it with other utilities including Adobe's Acrobat. Even if it does destroy the comment field the info can be put in the PDF meta data. Randy www.s100-manuals.com

On Fri, 2005-06-03 at 15:48 +0000, Jules Richardson wrote: fount it at: http://partners.adobe.com/public/developer/pdf/index_reference.html ... via google; any variation on "PDF specification" through Adobe's own website found nothing! Image XObject streams (section 4.8.4) seem to support an optional "Metadata stream" that appeared in version 1.4 of the PDF specification; prior to that metadata could only be attached to the doc as a whole, not to individual parts. It looks like what appears in the metadata stream is fairly tightly controlled though. What's missing is any defined way of saying what the original image was, and therefore how to either convert image metadata to PDF metadata format, or do the reverse when extracting image data back out of the PDF file. In both cases that'd be dependant on what the tools decided to do; image metadata preservation is totally up to the tools and nothing in the spec says where or how it should be preserved. I suppose that makes sense as far as Adobe are concerned; they consider PDF as an end format, and that nobody would ever want to actually deconstruct one into some other format. Likewise it's really beyond their scope to say how to convert metadata when pixel images are recoded to PDF's internal representation; as a group we'd likely have to come up with a code of conduct for that if PDF's the way forward. There is reference to an XMP "Extensible Metadata Platform" doc which I'll have to try and get hold of to see what that's all about.... No wonder there's some awful PDF tools out there; I'd never realised quite how complicated a format it is! cheers Jules

From: "der Mouse" <mouse at rodents.montreal.qc.ca> Even in practice, since the darned thing appears to be in PDF 1.6 format (which my Acrobat 4.0 doesn't seem able to read). Vince

Yes. (It was speculation, but with a basis.) Specifically, says to me that the extracted image was produced by rendering something into GhostScript's pnmraw output `device' (and then TIFFifying). /~\ The ASCII der Mouse \ / Ribbon Campaign X Against HTML mouse at rodents.montreal.qc.ca / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B

On Thu, 2005-06-02 at 23:15 -0700, Eric Smith wrote: I'm not sure I've seen one, and I've dealt with a *lot* of TIFF files over the years. Other metadata such as the app that created the image etc. is quite common though, and I have a feeling that Photoshop puts in all sorts of extra tags (I haven't got a copy here with which to check) Whether any such tags are useful to preserve is another matter. Personally I like the accountability; I'd like to know who scanned a document, when they scanned it, what software they used to do the scan. Mainly because it may help at some future OCR stage in identifying ways of improving the process or runs of documents that are likely to cause trouble during the OCR phase. Plus of course it's nice to know who was responsible for the hard work! cheers Jules

Jan-Benedict Glaw wrote: Yes. pdfimages, from xpdf. There's not any way to reconstruct the original TIFF file exactly, because there are too many possible variations. But it is possible to reconstruct a TIFF file that contains the identical image data. In other words, decompressing the original TIFF vs. the reconstructed TIFF would result in the same bitmap. Eric

der> ...something it is noticeably worse at than the netpbm family der> (pbm, pgm, ppm). (I can hardly imagine writing png-format image der> software without using a library. I routinely do so with p?m der> formats.) Sure, but that's because the netpbm formats are designed to be maximally trivial, so they are extremely inefficient. If you want compression, it won't be trivial, ever. paul

Eric! Stop giving me those $20 bills! You know I only like $10's! To make 10's from 20's I have to go all the way to the store! {In my best William Burroughs nasal whine} It's sooooo inconvenient!

Bj?rn wrote: I disagree. It's not particularly difficult to pull the bitmaps back out of a PDF document and convert them into whatever fromat you like. For instance, the pdfimages program that is part of the free xpdf software can do it, and there are many Windows utilities as well. Eric

From: "Scott Stevens" <chenmel at earthlink.net> Sent: Sunday, June 05, 2005 9:50 AM Adobe Acrobat has no intermediate files but other open source programs do. Even if there are intermediate files the thing is still to produce the PDF, that is the package that is useful to the majority. Remember we are not creating PDF's for ourselves and we are not creating PDF's for any individual we are creating PDF's to share with the majority. Each PDF can have totally different internal structure and only the PDF wrapper gives them a consistent face. If anyone can not handle the PDF or just won't deal with them then please remember this: That's OK you don't have to have access to them, you are no worse off for not having them just don't expect others to do anything special for you. Randy www.s100-manuals.com