20 May
2005
20 May
'05
10:08 a.m.
On Fri, 2005-05-20 at 09:55 -0700, vrs wrote:
Acrobat seems to be the only one under Linux that works well with PDFs
full of image scans, but the fact that Adobe have bloated it out to
nearly 100MB for the viewer has annoyed me somewhat.
Ghostscript doesn't seem to render with particularly good quality, and
all the other viewers I've tried seem rather incapable of dealing with
the physical size of PDF files that are wrapping document scans (I
assume it's another one of those situations where they're trying to
buffer the whole damn file into memory at once)
Presumably other Unix platforms suffer the same problems as they're
largely reliant on the same tools.
How Adobe mange to make Acrobat eat up 100MB of space is beyond me
though... :-(
I can't say I've found many bad TIFF viewers for single images though
(on any platform); it's only when multiple images are put into the same
file that a lot of tools start falling over.
cheers
Jules
It's
*could* - problem I've always found is that tools to handle multi-
image TIFF files are pretty thin on the ground; they either don't bother
at all (and assume one image per file) or they try and buffer the whole
lot into memory (for 100 A4 page scans in one file, that's Not A Good
Thing). Only about 20% of tools I've come across over the years actually
try and handle TIFFs properly (as I've said in the past, even
Imagemagick isn't great because it buffers data to 32-bit colourspace in
memory regardless of the actual images' bit depth)
The spectacular badness of TIFF viewers (at least at the free-as-in-beer
level) is why I prefer PDFs. Despite their problems, the PDF viewers seem
*way* better at rendering (and printing) the pages. (Though I am sure there
are other platforms which have no decent PDF viewer.) 
20 May
20 May
10:29 a.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
On Fri, 2005-05-20 17:08:34 +0000, Jules Richardson <julesrichardsonuk at
yahoo.co.uk> wrote:
I can't say I've found many bad TIFF viewers
for single images though
(on any platform); it's only when multiple images are put into the same
file that a lot of tools start falling over.
We've now named quite a lot of applications and concepts about how to
handle scanned documents. I'd like to get the big picture:
- How do you scan a paper document? Page by page? Two pages at once
(with a sufficient large scanner)? Do you use a script or something
like that? ...or are there well-working applications out there that
aid in scanning some 100 pages? Do you directly scan b/w, or first use
grayscale/colour and then degrade that to b/w?
- How do you work on the scanned images: Do you cut off the white rim as
much as possible? How do you deal with images that are a tad rotated?
Accept that? Re-scan to hopefully get a better image? Revert rotation
in software? How do you deal with single black dots in white areas or
the other way around?
- What digital format do you like to get when it's all finished? Plain
PDF? PDF with some bookmarks? PDF with all headings as bookmarks? A
new PDF-hyperref based index? Multiple TIFF/PNG/whatever images?
Something like a web-based slide-show? ...or multiple formats
(web-based for viewing, PDF for printing, ...)?
- What do you currently use as your software:
Operating system:
PDF viewer:
TIFF viewer:
Browser/other viewers you'd love to use:
MfG, JBG
--
Jan-Benedict Glaw jbglaw at lug-owl.de . +49-172-7608481 _ O _
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O
fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
12:02 p.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
On Fri, 2005-05-20 at 19:29 +0200, Jan-Benedict Glaw wrote:
On Fri, 2005-05-20 17:08:34 +0000, Jules Richardson
<julesrichardsonuk at yahoo.co.uk> wrote:
Single page here for A4 docs; for A5 I'll do two pages at once and have
written scripts before to automate the rotating / cropping process (and
chop out some of the 'noise' in the white background to reduce file
size). I tend to use 300dpi.
(Lots of Windows scanner software seems to try and be clever and adjust
the contrast on the fly btw - which isn't so good when you're trying to
get consistency across multiple pages!)
I can't say I've found many bad TIFF
viewers for single images though
(on any platform); it's only when multiple images are put into the same
file that a lot of tools start falling over.
We've now named quite a lot of applications and concepts about how to
handle scanned documents. I'd like to get the big picture:
- How do you scan a paper document? Page by page? Two pages at once
(with a sufficient large scanner)? Do you use a script or something
like that? ...or are there well-working applications out there that
aid in scanning some 100 pages? Do you directly scan b/w, or first use
grayscale/colour and then degrade that to b/w?
I use 8 bit greyscale for all pages, 24bit colour for covers. The former
because I don't want my scanning process to harm future OCR attempts;
I'd rather leave as much info in the images as possible rather than chop
to b/w at a certain threshold and find much later down the line that
information had been lost. If there's no assumption that the docs will
never be OCR'd then that's not a problem though. I could probably get
away with 16 grey levels actually rather than 256 as a reasonable trade-
off between flexibility and storage requirements.
- How do you work on the scanned images: Do you cut
off the white rim as
much as possible? How do you deal with images that are a tad rotated?
Accept that? Re-scan to hopefully get a better image? Revert rotation
in software?
I've been known to clean up scans and sort out rotation (in particular
photocopied docs tend to be not very straight). It's a time-consuming
job though; I'm almost tempted to say it's not relevant at the scanning
stage and can be deferred (much like the OCR process).
How do you deal with single black dots in white areas
or the other way around?
I don't. Lots of storage space can be saved by altering the black and
white threshold of the image though (without checking, I think I found
treating the bottom 10% as black and the top 20% as white worked
particularly well)
- What digital format do you like to get when it's
all finished? Plain
PDF? PDF with some bookmarks? PDF with all headings as bookmarks? A
new PDF-hyperref based index? Multiple TIFF/PNG/whatever images?
Something like a web-based slide-show? ...or multiple formats
(web-based for viewing, PDF for printing, ...)?
Seperate TIFF images, one image per page in the original doc. I scan
blank pages too (!) just so that things will work out at any printing
stage. I suppose I want to capture a bit of the "feel" of the original
document as well as the raw content.
- What do you currently use as your software:
Operating system:
Linux for processing / scripting images, scanner's currently an awful
USB thing hooked up to a Windows box though (urgh).
PDF viewer:
Try to avoid it where possible. I think I'll end up settling on Acrobat
5.1 under Linux; 7.0 is way too bloated. XPDF and KDE's offering don't
handle PDFs of document scans at all well. Ghostscript seems to have
issues with rendering quality (that could just be a misconfig / RTFM
issue on my part though)
TIFF viewer:
GQView's my current favourite general image viewer; it seems fast at
decoding and does a good job as a browser / thumbnail handler. I've not
tried it on multi-page TIFFs yet though to see what it does...
Browser/other viewers you'd love to use:
Well, I'd still like a native Linux version of Paint Shop Pro... :)
cheers
Jules
12:40 p.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
In message <1116615766.30950.126.camel at weka.localdomain>
Jules Richardson <julesrichardsonuk at yahoo.co.uk> wrote:
(Lots of Windows scanner software seems to try and be
clever and adjust
the contrast on the fly btw - which isn't so good when you're trying to
get consistency across multiple pages!)
For EPSON Scan:
Professional Mode -> Configuration -> Colour -> uncheck "Continuous Auto
Exposure".
Then just click the AE button (sphere with two diagonal red arrows pointing
to it) for the first page - for the rest of the pages, just hit the Scan
button.
I use 8 bit greyscale for all pages, 24bit colour for
covers.
Same here. I sometimes scan in 8-bit then downsample to 1-bit and save in
TIFF format with CCITT Group4 Fax compression.
For the 8-bit images, I use whatever lossless compression format works best.
Seperate TIFF images, one image per page in the
original doc. I scan
blank pages too (!) just so that things will work out at any printing
stage. I suppose I want to capture a bit of the "feel" of the original
document as well as the raw content.
Hmm. Good plan.
Linux for processing / scripting images, scanner's
currently an awful
USB thing hooked up to a Windows box though (urgh).
I use Linux and Win2k, depending on my mood. I'm currently using the i486
release of Slackware 10.1, but I'm migrating to Fedora's Core 3 x86-64
distro. I figure, the PC's got a 64-bit CPU, so why not make some use of it?
'Course it means I have to build a lot of stuff from source to get the speed
benefits, but that just adds to the fun :)
Software? Imagemagick on Linux or IrfanView on Win2k. I use the SANE drivers
for my scanner (Epson backend) or EPSON Scan on W2k. The scanner is an Epson
Perfection 2400 Photo, picked because it was Linux compatible.
As far as hardware goes, my main machine (Cheetah) has an AMD Athlon64 3200+
on an Asus A8V Deluxe mainboard, with 512MB of RAM (single-channel
PC3200/DDR400 mode), an 80GB Maxtor D740X hard drive, a LiteON LDW-851S DVD
recorder (cross-flashed with firmware from the SOHW-832S Dual Layer burner to
get DVD+R9 burning functionality), and a CEC PCI-488 GPIB card. I wanted a
National Instruments GPIB, but the ones I saw second-hand were selling for
very close to the price of a brand new card...
Ghostscript seems to have
issues with rendering quality (that could just be a misconfig / RTFM
issue on my part though)
I've noticed that... It doesn't antialias when it draws to an X window for
some reason.
GQView's my current favourite general image
viewer; it seems fast at
decoding and does a good job as a browser / thumbnail handler. I've not
tried it on multi-page TIFFs yet though to see what it does...
[quick google]
Ah, you use the Gnome WM then? I use KDE, though not for any reason other
than "it came with Mandrake 8 and I've used it ever since".
Well, I'd still like a native Linux version of
Paint Shop Pro... :)
*koff* Gimp *koff*
Later.
--
Phil. | Acorn Risc PC600 Mk3, SA202, 64MB, 6GB,
philpem at philpem.me.uk | ViewFinder, 10BaseT Ethernet, 2-slice,
http://www.philpem.me.uk/ | 48xCD, ARCINv6c IDE, SCSI
... A fail-safe circuit will destroy others.
12:05 p.m.
New subject: Inventory for handling scanned documents (was: Better indexing onbitsavers)
Jan-Benedict Glaw wrote:
- How do you scan a paper document? Page by page? Two
pages at
once (with a sufficient large scanner)? Do you use a script
or something like that? ...or are there well-working
applications out there that aid in scanning some 100 pages?
Both the previous place I was at and my current employer have
B&W scanners with an autofeeder and scan to PDF. The previous
one would drop a PDF on the desktop, the current one emails
it to me. Both would do double-sided.
Do you directly scan b/w, or first use
grayscale/colour and
then degrade that to b/w?
Scan in bitonal (1-bit, B&W). Then I (usually, now) post
process to convert to G4 encoded TIFFs within a PDF.
- How do you work on the scanned images: Do you cut
off the
white rim as much as possible?
I leave them as they come. If I scan a booklet, or something
that cannot be non-destructively taken apart (and later
reassembled) then I will scan two pages at a time and
post-process manually. In that case I'll probably end up
cropping at the physical page edges.
How do you deal with
images
that are a tad rotated? Accept that? Re-scan to hopefully
get a better image? Revert rotation in software?
If it is bad I will try to rescan, especially if it is only a
few pages. I've never tried to rotate in software.
How do you
deal with single black dots in white areas or the other way
around?
Never worried about that.
- What digital format do you like to get when it's
all
finished? Plain PDF? PDF with some bookmarks? PDF with all
headings as bookmarks? A new PDF-hyperref based index?
Multiple TIFF/PNG/whatever images? Something like a
web-based slide-show? ...or multiple formats (web-based for
viewing, PDF for printing, ...)?
I produce PDF as the final format. I do not usually bother
adding bookmarks or whatever.
- What do you currently use as your software:
Operating system:
Linux (Debian), Solaris, VMS, DOS and Windows.
PDF viewer:
Acrobat or xpdf.
TIFF viewer:
IrfanView, but most things I use PDF
Browser/other viewers you'd love to use:
I'd love to see near-perfect OCR (average error
rate of say one missed/misinterpreted character
per 500 pages on 5th generation photocopies of
technical manuals from the 1960s).
If I get a second wish, I would like something that
can take multicolour text pages (like the
RSX-111 MPLUS manuals) and slice it and dice
it into multiple layers (blue/pink/red/black
etc. and put it back together as a PDF.
I still have hundreds (maybe more) pages waiting
to be processed - each is currently a 24MB (or so)
24-bit colour TIFF (at 600dpi). Help :-)
Antonio
--
---------------
Antonio Carlini arcarlini at iee.org
1:29 p.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
We've now named quite a lot of applications and
concepts about how to
handle scanned documents. I'd like to get the big picture:
- How do you scan a paper document? Page by page?
Page by page, usually. If it's bound, and two pages fit together on my
scanner, I'll often do it two pages at a time.
Do you use a script or something like that?
No...though I will use shell history mechanisms to ease the task, each
scan generally involves at least a few keystrokes.
Do you directly scan b/w, or first use
grayscale/colour and then
degrade that to b/w?
This is a judgement call. Sometimes I'll do colour, sometimes
greyscale, sometimes binary. When I do binary I'll sometimes lett he
scanner do it and sometimes I'll scan it some other way and convert it
by hand.
- How do you work on the scanned images: Do you cut
off the white rim
as much as possible?
Usually.
How do you deal with images that are a tad rotated?
Ignore the rotation, usually. Egregious cases I may rescan. In some
cases, I touch up the orientation with pnmrotate (and a subsequent
pnmcut).
How do you deal with single black dots in white
areas or the other
way around?
If the use is one for which they matter, I edit them out "by hand".
- What digital format do you like to get when it's
all finished?
Plain PDF? PDF with some bookmarks? PDF with all headings as
bookmarks? A new PDF-hyperref based index? Multiple
TIFF/PNG/whatever images? Something like a web-based slide-show?
...or multiple formats (web-based for viewing, PDF for printing,
...)?
I dislike PDF. I loathe Web-specific stuff.
I generally just keep a directory around with the iamge files.
Sometimes I'll tar it up, sometimes I'll use compression programs like
gzip or bzip2 to reduce the storage requirements.
- What do you currently use as your software:
Operating system:
NetBSD (1.4T plus a number of private hacks).
PDF viewer:
GhostScript (8.30 at the moment).
TIFF viewer:
tifftopnm, often transformed with pnm tools like pnmscale, with the
resulting p*m file displayed using a picture display program of my own.
Browser/other viewers you'd love to use:
I've yet to find really *good* such tools, probably because my
user-interface tastes are unusual.
/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML mouse at rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
23 May
23 May
2:50 p.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
On Fri, 2005-05-20 19:29:18 +0200, Jan-Benedict Glaw <jbglaw at lug-owl.de> wrote:
On Fri, 2005-05-20 17:08:34 +0000, Jules Richardson
<julesrichardsonuk at yahoo.co.uk> wrote:
We've now named quite a lot of applications and
concepts about how to
handle scanned documents. I'd like to get the big picture:
Thanks a lot for all your input. What I read out of that:
- All of you just want to read the contents.
- It would be nice if specific make-up could be applied, but
because it mainly depends on a lot of work, most of you don't
consider doing this.
- Getting it right into a web browser might be nice to have.
- Some of you also care about each individual image for
later[tm] post-processing (like OCRing it with The Next
Generation Never-Failing HyperOCR[tm](C)(REG) )
- Getting a PDF file out of those images is a must because it's
so nice'n'easy to print and copy around.
The point is: most of this actually can be done with little scripting or
programming! Most of you (especially those who scan, but also those who
try to dismantle a PDF file) stated that there are home-brewn scripts
doing most of the dirty work. But nobody actually told me something like
'Here's this URL, you'll find my stuff right there.'
...and this is what brings most of the complecity into the big picture.
Using libtiff, I guess it's not all that complicated to write a little
helper application capable of merging up to 64K single TIFF images into
eg. a multi-page TIFF with additional arrays of type-7 tags. That's kind
of magic: with this trick, you can embed any data inside a (multi-page)
TIFF, pegged to one of the sub-files (read as: one of the embedded page
images).
Armed with this, you can have /n/ TIFFs for a book's /n/ pages or one
hugh multi-page TIFF containing them all, *plus* the bonus of added
keywords, chapter captions, printed page number and the like. All these
goodies won't show up during TIFF->PDF conversion (for printing), but
they'd be well-used by some little scripts to generate a nice web-based
slide show. Even a X11-based reader is in reach (there are probably
already nice readers for things like eBooks or books for Palm Pilots,
though, I've never ever used something like that).
So we'd push the data into single TIFF files, we'd easily extract them
into text files (eg. for indexing), we'd even import auto-OCRed text
(won't be perfect, but probably better than nothing, just for pure
Google indexing).
To cut a long story short:
- Could you life, for long-term storing the data, with using
single-page TIFFs?
- These could be used to create multi-page TIFFs, PDFs, web
pages, ...
- Would you like to see something like a X11-based reader which
could support searching and equal-or-better navigation
(compared to Acrobat Reader)?
I currently don't have something scanned around; could somebode send me
(preferrably an URL) for some complete scanned doc, I'd prefer something
in the range of 20..120 pages.
Eric, I don't know how well-working your bookmark generation code is.
Can it already handle really tree-like looking bookmarks if the data was
available in tumble's input files?
MfG, JBG
--
Jan-Benedict Glaw jbglaw at lug-owl.de . +49-172-7608481 _ O _
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O
fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
3:42 p.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
Jan-Benedict Glaw wrote:
Armed with this, you can have /n/ TIFFs for a
book's /n/ pages or one
hugh multi-page TIFF containing them all, *plus* the bonus of added
keywords, chapter captions, printed page number and the like. All these
goodies won't show up during TIFF->PDF conversion (for printing),
Certainly all those "goodies" can be transferred into PDF files. For
instance, my "tumble" program can do that:
http://tumble.brouhaha.com/
- Could you life, for long-term storing the data,
with using
single-page TIFFs?
- These could be used to create multi-page TIFFs,
If you're going to use TIFF as your storage format, why not store
them as multi-page TIFF? Anyone that needs individual pages can
easily enough "burst" them with a utility like tiffsplit. If you
store them as a bunch of single files, it increases the chance that
someone will end up with only a partial document, bad pages, etc.
(same reason programs are often distributed as ZIP or tar files rather
than a bunch of smaller files).
- Would you like to see something like a X11-based
reader which
could support searching and equal-or-better navigation
(compared to Acrobat Reader)?
Are you talking about a TIFF reader? I'd like to see better PDF readers.
Evince is already much better than xpdf was, but there's probably room
for further improvement.
Eric, I don't know how well-working your bookmark
generation code is.
Can it already handle really tree-like looking bookmarks if the data was
available in tumble's input files?
Yes.
I don't like the way the tumble control files work now, which is
part of why they're not documented. I'm probably going to redesign it
to use an XML-based control language.
In my copious free time. Sigh.
Eric
24 May
24 May
12:38 a.m.
New subject: Inventory for handling scanned documents (was: Better indexing on bitsavers)
On Mon, 2005-05-23 15:42:23 -0700, Eric Smith <eric at brouhaha.com> wrote:
Super :)
Jan-Benedict Glaw wrote:
If you're going to use TIFF as your storage
format, why not store
them as multi-page TIFF? Anyone that needs individual pages can
2GB file-size limit and BigTIFF is yet in early draft stage. This
*might* be a problem at some time.
easily enough "burst" them with a utility
like tiffsplit. If you
store them as a bunch of single files, it increases the chance that
someone will end up with only a partial document, bad pages, etc.
(same reason programs are often distributed as ZIP or tar files rather
than a bunch of smaller files).
I'm not sure of available tools can handle that, esp. copying unknown
tags (esp. those of newer date like the "new" UNDEFINED tag of TIFF6.0).
But libtiff is available, so if existing tools cannot yet handle it,
they either could easily be extended or we write our own.
Are you talking about a TIFF reader? I'd like to
see better PDF readers.
Basically yes, but with the ability to use additional embedded data like
captions, printed page number and index words.
Evince is already much better than xpdf was, but
there's probably room
for further improvement.
I'd really give it a try :)
Eric, I
don't know how well-working your bookmark generation code is.
Can it already handle really tree-like looking bookmarks if the data was
available in tumble's input files?
Yes. I don't like the way the tumble control files work
now, which is
part of why they're not documented. I'm probably going to redesign it
to use an XML-based control language.
I'm really thinking about doing some testing with custom TIFF tags. With
that, an external control file may not be needed at all. Though, the
tools need to extract/import that data to/from text files of course.
In my copious free time. Sigh.
That's always a problem I guess. I just quit my contract (will remain at
my current employer's site for another month continuing hacking _and_
teaching coworkers) and of course I'd like to hack vax-linux, too. (It
was about 04:20 when I went to bed this night, 07:00 is when my alarm
clock rings up and I'm doing that for too long already...) But don't
start too fast with a new XML interface. Maybe the TIFF files will ship
all data for free at some time :)
MfG, JBG
--
Jan-Benedict Glaw jbglaw at lug-owl.de . +49-172-7608481 _ O _
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O
fuer einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
20 May
20 May
10:36 a.m.
From: "Jules Richardson" <julesrichardsonuk at yahoo.co.uk>
I can't say I've found many bad TIFF viewers
for single images though
(on any platform); it's only when multiple images are put into the same
file that a lot of tools start falling over.
The most common problems that I have had with single images have to do with
crappy scaling of the picture to fit the screen or printer. (Almost any
viewer will look OK with zoom set to 100%.) The next most serious problem
has been slowness (probably due to bad buffering decisions) but that's
mostly with a really large image (or multiple images).
For multiple image files, the slowness is often horrific, plus the interface
for switching pages completely sucks. (Forgetting the zoom setting when
going to the next page is an example that comes to mind.)
Vince
7104
days inactive
7108
days old
9 comments
7 participants
participants (7)
-
a.carlini＠ntlworld.com -
eric＠brouhaha.com -
jbglaw＠lug-owl.de -
julesrichardsonuk＠yahoo.co.uk -
mouse＠Rodents.Montreal.QC.CA -
philpem＠dsl.pipex.com -
vrs＠msn.com