Subject: [CODE4LIB] Durability of PDFs

From: Mike Taylor <[email-masked]>
Date: Mon, 15 Jun 2009 11:37:57 +0100

Dear CODE4LIB colleagues,

In one of my alternative incarnations, I am a zoological taxonomist.
One of the big issues for taxonomy right now is whether to accept as
nomenclaturally valid papers that are published only in electronic
form, i.e. not printed on paper by a publisher.

In a discussion of this matter, a colleague has claimed:

> [PDF files will not become unreadable] in the next 30-40 years.
> Possibly not in the 20 years that will follow. After that, when only
> 30-year and older documents are in the PDF format, the danger will
> increase that this information will not be readable any more. It is
> generally considered as quite unlikely that PDF will be readable in
> 100 years.

I would appreciate any comments that anyone on this list has on the
likelihood that PDF will be unreadable in 100 years.

Many thanks,

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "Can't someone act COMPLETELY OUT OF CHARACTER without arousing
	 suspicion?" -- Bob the Angry Flower, www.angryflower.com

---

From: "Benjamin O'Steen" <[email-masked]>
Date: Mon, 15 Jun 2009 11:45:06 +0100

There are items/options that can be used within a given PDF that will
drastically affect how likely it is that the PDF will still be readable.

* Inclusion of 3D applets or any Adobe Acrobat specific features
	I have seen PDFs with 3D chemical applets embedded somehow into the PDF
using a plugin. The longevity of this addon depends entirely on how long
the company/team that made the plugin, wants to support it.

* DRM of any kind - password-protected, print-disabled, etc
	These features make it hard, verging on the legally impossible to
migrate the PDF to a newer format or read it on newer versions of
Acrobat or other PDF viewers. (Legally impossible, as circumventing this
would incur the wrath of the DMCA)

* Any other odd features.

There is a profile, which doesn't allow you to add any of the above, and
it is often referred to as PDF/A (A for Archival format)

The easiest way to create these at the moment, is to use OpenOffice 3
and choose the "Save As PDF" and tick the PDF/A option.

As for not becoming unreadable.. well, this all depends on age (and so
the version) of the PDF, and your current viewing software. I have
already had situations where older PDFs cannot be viewed correctly in
newer readers (majority of these were due to older 'print-ready' pdfs
with colour-information held within)

And this doesn't include the various issues that can arise from fonts
not being included or present on the client's system, 'print' fonts that
compress letters in interesting ways ("fi" -> single character, but a
non-unicode one), images that do not display, incomplete PDFs due to bad
exports that silently fail, etc.

My advice is to keep the source files alongside, especially if they are
in (la)tex or HTML. Text is always parsable.

Ben

---

From: "K.G. Schneider" <[email-masked]>
Date: Mon, 15 Jun 2009 07:45:41 -0400

> In a discussion of this matter, a colleague has claimed

> > generally considered as quite unlikely that PDF will be readable in

Setting aside the paper/electronic argument, in terms of canonical files for
documents intended for long-term preservation, PDF seems a very weak choice.
Whether or not the actual files will "last" 100 years (I assume that we mean
that they won't degrade to the point of nonreadability), using a proprietary
binary format that doesn't readily convert to other formats seems a poor
choice. 

Why not have the documents be sourced in one of the XML-based formats such
as DocBook or DITA (well-documented, open, text-based, single-source
publication formats)? Then you can have your PDF and preserve it too.
(Donning tinfoil hat) You could even produce a handful of paper-based
documents and hide them in caves around the world. 

Karen G. Schneider

---

From: Mike Taylor <[email-masked]>
Date: Mon, 15 Jun 2009 13:08:32 +0100

K.G. Schneider writes:
 > > > Possibly not in the 20 years that will follow. After that, when
 > > > only 30-year and older documents are in the PDF format, the
 > > > danger will increase that this information will not be readable
 > > > any more. It is generally considered as quite unlikely that PDF
 > > > will be readable in 100 years.
 > 
 > Setting aside the paper/electronic argument, in terms of canonical
 > files for documents intended for long-term preservation, PDF seems
 > a very weak choice.  Whether or not the actual files will "last"
 > 100 years (I assume that we mean that they won't degrade to the
 > point of nonreadability), using a proprietary binary format that
 > doesn't readily convert to other formats seems a poor choice.

Just as a point of information, PDF is not proprietary: as of July
2008, it is an ISO standard.

 > Why not have the documents be sourced in one of the XML-based
 > formats such as DocBook or DITA (well-documented, open, text-based,
 > single-source publication formats)? Then you can have your PDF and
 > preserve it too.

The problem is how we get there from here.  Right now, the world is
full of journals that routinely make available PDFs of the articles
published in them.  Far, far fewer make available any XML-based format
-- in fact, I would imagine that only a tiny minority ever have the
documents in an XML-based format: most will go from MS-Word
submissions to PDF publications.

So the question becomes: if we want to rely on digital preservations
so that we don't need to print, do we first need to persuade the
world's journals to change their publishing practices?  If so, that
doesn't seem like a very realistic goal.

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "When I can't fondle the hand I'm fond of, I fondle the hand at
	 hand" -- E. Y. Harburg, "Finian's Rainbow"

---

From: Thomas Dowling <[email-masked]>
Date: Mon, 15 Jun 2009 08:13:01 -0400

> binary format that doesn't readily convert to other formats seems a poor

The PDF 1.7 spec was published last year as ISO 32000-1:2008.  PDF/A is also
ISO 19005-1:2005.  All just FWIW, since ISO's history with Microsoft OOXML
raises some questions about how "open" you really have to be to get an ISO number.

> documents and hide them in caves around the world. 

I expect the conversation usually starts (and often ends) with, "Hmm...'File,
Save as PDF'.  Bingo!"

-- 
Thomas Dowling
[email-masked]

---

From: Toke Eskildsen <[email-masked]>
Date: Mon, 15 Jun 2009 15:57:03 +0200

> likelihood that PDF will be unreadable in 100 years.

The problem with projections such as these are that we have very little
empiric evidence to build on. The classic fallacy is to look at any old
format that is now hard to read and use that as a guide. The problem
with this form of extrapolation is that Old Format X wasn't very
widespread in the general population, simply because the average person
did not have a computer then.

Forwarding to 2009, everyone and their dog produces a huge amount of
PDFs and JPEGs, to name two contenders for "they will be readable for
the next 1000 years".

When there's an ungodly amount of information in a given format:
Information not limited to a specific scientific field or area of
expertise, we, as a society, want to be able to read it. Forever. 

Your friend's argument that the only PDFs in the year 2070 will be 30
years old might be true in its premise, but I don't agree on his
conclusion. Why wouldn't we be interested in information from the
beginning of the millennium when we reach 2070?

That is, of course, with the premise that our society does not collapse
so that we can't maintain our current technology level. In that case
we're talking bit loss as well, which is a whole other discussion.

- Toke Eskildsen (_not_ a specialist in archival)

---

From: Jonathan Rochkind <[email-masked]>
Date: Mon, 15 Jun 2009 10:13:01 -0400

The bet is that PDFs are so popular that _someone_ (the archival 
community if no-one else, but probably someone else) will ensure that 
they continue to be readable somehow.

These are real non-trivial issues in electronic archiving though, issues 
that the archival community.  It is generally a safe assumption that 
good electronic archiving over the decades-and-more term requires some 
regular attention by an electronic archivist to make sure that files 
remain readable, and are converted to new formats when necessary. As 
well as attention to avoiding actual bit-level corruption of files. You 
can't neccesarily just dump files on a HD and ignore them and expect 
they'll be readable in 100 years, that much is true -- and true pretty 
much regardless of particular electronic format you choose.

As far as electronic formats go, I think PDF is as good as anything -- 
except maybe plain ASCII text, which is not nearly as useable (and 
doesn't allow diagrams, mathematical equations, non-English letters, 
etc). I don't know why you're colleague has decided that "30-40 years" 
is the horizon after which PDF specifically will become "unreadable", 
this seems like just a wild guess to me, but it would be interesting to 
see if he has any particular evidence to back up this claim. 

So there are real issues with electronic archiving, but unless they lead 
you to refuse to accept electronic submissions at all, you're just going 
to have to deal with them, it's not really an issue of PDF specifically, 
but it is true that "just dump files on a HD and forget about them and 
assume they'll be readable in 100 years" is not a particularly safe 
electronic archiving strategy.

Jonathan

---

From: "Doran, Michael D" <[email-masked]>
Date: Mon, 15 Jun 2009 15:01:08 -0500

> As far as electronic formats go, I think PDF is as good as 
> anything -- except maybe plain ASCII text, which is not
> nearly as useable (and doesn't allow diagrams,
> mathematical equations, non-English letters, etc).

There is no requirement that plain text be limited to the ASCII character set repertoire.  Although once they were almost synonymous, that is no longer the case [1].  Plain text can encompass anything and everything in the Unicode character set.  That includes non-Roman scripts, mathematical symbols, yada, yada, yada.

-- Michael

[1] http://en.wikipedia.org/wiki/Plain_text

# Michael Doran, Systems Librarian
# University of Texas at Arlington
# 817-272-5326 office
# 817-688-1926 mobile
# [email-masked]
# http://rocky.uta.edu/doran/

> -----Original Message-----
> From: Code for Libraries [mailto:[email-masked]] On 
> Behalf Of Jonathan Rochkind
> Sent: Monday, June 15, 2009 9:13 AM
> To: [email-masked]
> Subject: Re: [CODE4LIB] Durability of PDFs
> 
> >  _/|_	 
> ___________________________________________________________________

---

From: Jonathan Rochkind <[email-masked]>
Date: Mon, 15 Jun 2009 16:14:18 -0400

Fair enough.  Asking someone to give you a UTF-8 (or other Unicode 
encoding) plain text file though -- you better try to heuristically 
check the encoding before ingesting it, and plan on a lot of failures. 
Typical users using typical consumer software (which tends to be 
somewhat unpredictable with character encodings) can't be trusted to 
give you a UTF-8 encoding just because you specify it, or  to have any 
idea what this means or how to do it. 

And checking the to see if the 'true' encoding of a plain text file is 
what it's advertised as in an automated fashion is heuristic at best, 
and not going to be perfect. 

And you're still going to have trouble with complicated mathematical 
formulas, molecular diagrams, other diagrams, etc.

Jonathan

Doran, Michael D wrote:
>> anything -- 
>>>> Possibly not in the 20 years that will follow. After that, 
>>>>         
>> when only
>>     
>>> /o ) \/  Mike Taylor    <[email-masked]>    
>>>       
>> http://www.miketaylor.org.uk
>>     
>>> )_v__/\  "Can't someone act COMPLETELY OUT OF CHARACTER 
>>>       
>> without arousing
>>

---

From: "Ben O'Steen" <[email-masked]>
Date: Mon, 15 Jun 2009 23:51:03 +0100

Jonathon,

Likewise that paragraph reads with the same accuracy with the
following alterations

s/UTF-8|Unicode/PDF/
s/encoding/version/

I think the key thing is that garbage in == garbage out, but I feel
happier with garbage that was meant to have been unicode at some
point, compared to a pdf that was made by a Word->PDF printer driver
that craps out on large files, but does so silently.

My experiences with PDF versions:

PDF 1.3 and earlier is evil, 1.4 not too bad aside from colour issues
and its ham-fisted way of attempting to shove CMYK info into itself,
1.6 has issues to some people that I am still trying to isolate, 1.7
is rare as hens teeth and PDF/A as a spec seems to be okay, but I've
only seen a few of those in the wild and only from OpenOffice too. It
would be interesting to see how OOo's idea of PDF/A stacks against
Adobe's.

And there is PDF/X(-3?) orsimilar which I've only even seen on an
options panel, before being swiftly ignored.

And on a final note, there have been PDF files that are useless to me,
I can't wheedle out anything from them, and that are only 10 years
old. However, I have resurrected a tex-based thesis from an earlier
period without difficulty, and created a PDF/A from the source.

Bottom line is that it's best to preserve the source materials as well
as the final disseminations - you can't always guarantee a viewer will
work as expected. The trend is that newer PDF versions are better, but
be very very wary of hidden DRM. If memory serves, an eBook publisher
lost 1/4(?) of their stock, due to losing the mechanism to unlock.
Let's not have that happen to repositories...

Ben

2009/6/15 Jonathan Rochkind <[email-masked]>:
> Fair enough.  Asking someone to give you a UTF-8 (or other Unicode encoding)
> plain text file though -- you better try to heuristically check the encoding
> before ingesting it, and plan on a lot of failures. Typical users using
> typical consumer software (which tends to be somewhat unpredictable with
> character encodings) can't be trusted to give you a UTF-8 encoding just
> because you specify it, or  to have any idea what this means or how to do
> it.
> And checking the to see if the 'true' encoding of a plain text file is what
> it's advertised as in an automated fashion is heuristic at best, and not
> going to be perfect.
>>> except maybe plain ASCII text, which is not
>> There is no requirement that plain text be limited to the ASCII character
>> set repertoire.  Although once they were almost synonymous, that is no
>> longer the case [1].  Plain text can encompass anything and everything in
>> the Unicode character set.  That includes non-Roman scripts, mathematical
>> symbols, yada, yada, yada.
>>
>> -- Michael
>>
>>> From: Code for Libraries [mailto:[email-masked]] On Behalf Of
>>> Jonathan Rochkind
>>> community if no-one else, but probably someone else) will ensure that they
>>> continue to be readable somehow.
>>>
>>> that the archival community.  It is generally a safe assumption that good
>>> electronic archiving over the decades-and-more term requires some regular
>>> attention by an electronic archivist to make sure that files remain
>>> readable, and are converted to new formats when necessary. As well as
>>> attention to avoiding actual bit-level corruption of files. You can't
>>> neccesarily just dump files on a HD and ignore them and expect they'll be
>>> readable in 100 years, that much is true -- and true pretty much regardless
>>> of particular electronic format you choose.
>>>
>>> except maybe plain ASCII text, which is not nearly as useable (and doesn't
>>> allow diagrams, mathematical equations, non-English letters, etc). I don't
>>> know why you're colleague has decided that "30-40 years" is the horizon
>>> after which PDF specifically will become "unreadable", this seems like just
>>> a wild guess to me, but it would be interesting to see if he has any
>>> particular evidence to back up this claim.
>>> you to refuse to accept electronic submissions at all, you're just going to
>>> have to deal with them, it's not really an issue of PDF specifically, but it
>>> is true that "just dump files on a HD and forget about them and assume
>>> they'll be readable in 100 years" is not a particularly safe electronic
>>> archiving strategy.
>>>
>>> Jonathan
>>>
>>> Mike Taylor wrote:
>>>
>>>>

---

From: Lars Aronsson <[email-masked]>
Date: Thu, 2 Jul 2009 04:31:40 +0200

> and choose the "Save As PDF" and tick the PDF/A option.

Is there some free software that can verify if a given PDF file 
really is a valid PDF/A file?  If I want users to upload their 
documents to an archival webserver, I don't want them to upload 
bad files that aren't standards compliant and safe for the future. 

The web server needs to run a check, hopefully in less than a 
second and using predictable amounts of RAM.

Or how do you typically solve that problem?

Can PDF/A documents contain scanned images with OCR text?

-- 
  Lars Aronsson ([email-masked])
  Aronsson Datateknik - http://aronsson.se

---

From: Ed Summers <[email-masked]>
Date: Wed, 1 Jul 2009 22:56:19 -0400

You could look at JHOVE [1] for validation of the PDF/A.

[1] http://hul.harvard.edu/jhove/