One of the phrases that's become common parlance, if you like, since the
launch of PDF in the early 1990s is: "It's okay i'll just send
you a PDF". And a lot of people don't realize what a
broad church that is. It covers a multitude of sins. Yes you may get an
ultra-high quality thing that could be printed in Vogue magazine,
in 23 different colours on glossy paper, Or somebody may just say: "Oh all you
want is a copy of this, of some sort. I'll just stick it in my scanner and
send you what amounts to a jpg scan of the page."
But since you've asked for PDF I'll use the fact that PDF seems to be able to do it.
But it doesn't upgrade it - it just sends
you a bitmap scan, only in PDF format.
What should we be saying, in addition to 'I'll just send you a PDF' to reassure
people that the quality will be there somehow,
If you want that quality to be part and parcel of
your story ? I'm going to illustrate this in the end by just showing you
an original bitmap pdf of Dennis Ritchie's thesis. And when you
look at it the scan quality is very variable but you think 'yeah!
typical jpg'. But it isn't, it's PDF, so it's a broad church. You can have many
sorts of PDF so let's just run through three very important
types of PDF. If you're a purist you may be thinking:
" Well, a PDF, of course, has presumably been done on a professional piece of software like,
you know, Adobe InDesign. So it'll be sharp and perfect and everything.
And if you have high expectations and you want your
PDFs to be infinitely magnifiable up to barn-door size and have
beautifully rendered colours, well then [for] this aim in life
I think adobe used to call it PDF Normal. And my ex-grad student, Matt Hardy, still
there with Adobe, 'Hello, Matt!.' He said: "Dave, people don't understand
what 'PDF Normal' means! " So, I said: "Well what do you
call it then ?!" He said: "Oh well we use something like "PDF full text and
graphics (PDF-FTG)" In other words you've tried to choose
the features in PDF that enable you to render that sort of
material at the best possible resolution. If it's
FTG you've got to use proper fonts for characters.
If you draw line diagrams you've got to use something like Adobe Illustrator to
do them. You must draw a line as a line - not as a
set of dots. you must draw a circle as a circular
curve - not a set of dots. And you resolve
down to, if you like, coloured jpg, only for things that really are like that,
Photos, scenery, people whatever. That's absolutely fine. But if people
are in a hurry, or don't have all of this sophistication,
the other end of the spectrum is [that] you can basically use
a scanner to scan in a copy of your high-quality document
rather than regenerating it from source. You might say: "Oh! i can't be
bothered i'll just scan it". Now, if you just scan it in that's fine:
it could be a good quality scan a bad quality scan
and just like with jpg, tiff, png - all of these bitmap standards you may
have heard of and used for your photography, PDF
has its own bitmap graphic capability that subsumes all of those. So you could
send somebody a PDF which basically is little better
than a .jpg. One of the things i'm finding in
fielding queries [on this video] is that somehow, and it's great, PDF is a symbol
of quality. You know: "i asked people to send me a PDF
and all i got was this lousy bitmap!" Something like that, And the answer is,
yes, PDF will cheerfully render lousy stuff
if you give it that. PDF(I) for 'PDF Image Only'. You might think: "Well, those
are the two extremes then. Everything thoroughly retypeset -
using appropriate tools - or, at the opposite extreme:
Everything's a photo; everything's a jpg, everything's a bitmap.
It's just we're calling it a PDF bitmap.
And somebody at Adobe said: "Y'know could optical character
recognition, OCR, be of any help to us?" In the late 80s, early 90s, adobe
bought a Boston-based graphics and ocr company,whose name i've
now forgotten. But it was an excellent purchase because basically
they said: "If you have got high-quality bitmap
scans and you don't know what [app] made them,
we will have this wonderful thing called Acrobat Capture which will enable you
interactively to rebuild, to a very high standard, documents that
were done in a professional app that you don't have.
And it was a very good product. The idea
was you had a room full of people - hundreds of them even - you had a Capture
Server. So you could share the task of
recreating a 10,000 document Kennedy Library [corpus]
brought back to life with excellent precision.
It was too expensive ! The number of libraries that will
rebuild, in an entirety, a whole collection,
at super high quality, is very very limited. But somebody,
somewhere, I don't know who it was but they deserve a medal -
said: "Ah but what this will be ideal for is that, in court, judges
love having Xeroxes and they've been told not to call them Xeroxes (because
there are other photocopier companies :-) So they start saying
"Give me a photocopy !" And they love them! So, what we want
from PDF is to retain the roughness of the original but somehow,
magically, to OCR it and make it at least slightly searchable.
And somebody could say to her [the Judge]
"Yes, you Honor, you're on page 3, the word 'reprehensible' will appear on
line 3 so just type in " r-e-p- ...." "Oh! it's found it!" You see?
Well, if you've got a page full of bitmap how the heck
do you find it [the target of your search]. The answer is you have in some limited way to do
OCR on it, to see if there's anything there
that corresponds to text. So the 'middle way'
was in many ways, I think, the saving of PDF for large-scale commercial
applications. This wonderful compromise of saying: "We'll give you a bitmap
overlay, done with our technology, but underneath
that we will try our best but we won't go
completely mad. Maybe 70% to 80% OK from a good scan. We'll try and give
you some good searchability on this but not
totally perfect. We're moving on now from saying
"Well, we can't do FTG; Image only (I); is not good enough quality - we want more value -
to the rescue comes "PDF Image + Hidden Text"
And the idea is that you recognize as much as you can;
you recognize where on the page that phrase is
you have a guess at what the font is using your best 'quick and dirty'
technology and you say: "I can now make this searchable
insofar as my OCR makes sense". Ask me for "hello world" [and]
as long as i've recognized it. I will highlight for you,
back on the bitmap, Yeah! i won't just give you a listing saying
"Memorable phrases i discovered in this bitmap", I'll actually highlight them for you
because i know where they occur. That's a
very long story. In short it was a howling success,
it really was! The idea of having a bitmap that clearly hadn't been messed
with (except it had ... ) but which Her Honor
could actually get to see the relevant phrase
in context, highlighted on the bitmap, was an absolute winner, you know, in the
legal community. Let's go and find the workstation where
i'm currently trying to rescue Dennis Ritchie's thesis.
And to see what versions of it i currently have available.
This office. This is my Linux box
running Debian Linux. So this is back to 1972
software, which i just happened to have hung on to for projects of this nature
What you're seeing here is the end result of me rescuing this page.
This is my PDF(FTG). I am using exactly the right tools for properly
re-typesetting a version of this material [ditroff, pseqn, pic]
So you go there [shows sample output] But this material is mostly just
rather complicated - oh! look at that! - hideously complex mathematical equations.
You do not want to type that out character-by-character, by hand, even if
you know what it means. So, what i can do and it should scale up
reasonably well. If I say here, on the preview,
that i'd like to -- can you see i've asked it
to be eight times magnified but look the quality is still there. In fact i
think i can go one more on this preview software
Innately the PostScript / PDF imaging model
is capable of scaling stuff up that has been properly typeset. It knows how to do it.
Now, just for contrast let's take a look
at what the original looked like. [screen sows image of cover page] Now you can tell straight away - that's
a photocopy. Also old-timers in there will
immediately seize on that and say that's a Teletype print out. And it is
for some reason Dennis did the cover page on a Teletype
but when you get to the first real page, as far as we know,
you can tell that [this] is what an IBM Selectric typewriter can do.
If you magnify this ... not very good is it? ... Look at that. Can you
see it's all breaking up. I mean, this is by no means the worst
page of course. We say clearly "Image Only" is not good enough
for us. It's not searchable. So next [is] People's Exhibit Number 2 then, is,
Let's, from here, kill off that particular PDF.
[Now] Ritchie Dissertation OCR.pdf Let's see where we get to. It looks the
same and of course the page bitmap overlays
*are* the same but sneakily, and after several minutes
of processing [for a 180-page thesis] it has hidden searchability underneath.
So let us now go on page 1, and let us say, after all, we want to
just find the first "the" in this document,
Oh! it found one even earlier! Of course, I forgot that the
default mode of this [search] is not case sensitive. It's found a "The" here but with a
capital T. What i've actually managed to do is to
get a searchable bitmap version of Dennis's
thesis - and that's absolutely ideal to work from,
because I as a human can read it, and if i say "Hmm! i'm sure we came across
the word 'submarine' on this page. I can try and find it and I very likely will.
So, there you are then, you've seen totally reset material,
which is arbitrarily scalable, and doesn't crack up.
This [points at screen] is clearly not of proper typeset quality
Even at its native size here you can see there's dots missing in the
typeface and it's just not of that crisp high quality look that
one so searches after when you're outputting a PDF. You should
be trying to output a pdf with Full Text and Graphics and treat everything
the way it deserves to be treated. You should not be taking shortcuts
saying: "I don't have enough fonts - I'll just bitmap that text." The virtues
of FTG not only include quality but they
include space. Typesetting text from within a font
takes a lot less space than reproducing all the dotty bitmaps
of your broken letter t's and all this kind of stuff.
Equally you don't want to subject your graphics
to being a photo with lots of white pixels in there that have to be
compressed out and causing grief all over. No, if it's a
straight line you just say "line" and that gives you the appropriate
command in PDF to do a line. You should aim for the
highest quality that your audience, or that the
application, demands and just be aware that there is
this intermediate stage which I've found very useful is to look at a moderately
OK quality bitmap as my guide to what i
need to fix and re-type. But, at the same time, to be able to find
things inside it - just about.