Subject: [CODE4LIB] download design

From: "[Chris Stockwell]" <[email-masked]>
Date: Fri, 24 Jul 2009 17:20:33 -0400

Over the next few years, I am tasked to download 30,000 archival masters 
from Internet Archive into an archive for long-term staff access that we may 
preserve with LOCKSS. These are masters of Montana state publications. I 
have a hierarchy in mind to receive these files. The hierarchy is state 
agency\year\title\pub_date\*.pdf.  

I am intending to download the files in batches of 200 - 500 pdfs, but am 
thinking that if I slot them automatically into the archive hierarchy, misplaced 
or missing files could be very hard to find as the total grows. I will be logging 
the downloads, which should give me some control. Are there other strategies 
for ensuring that I can readily correct download errors? I am looking for 
recommendations for the simplest way to maintain reasonable control over the 
download process.

---

From: Stuart Yeates <[email-masked]>
Date: Sat, 25 Jul 2009 12:07:32 +1200

> Are there other strategies for ensuring that I can readily correct 
> download errors? I am looking for recommendations for the simplest 
> way to maintain reasonable control over the download process.

I would recommend that you also try opening the PDFs using some toolset, to check for corrupt files. There are, of course, many toolsets, including several open source ones with command line interfaces which make them readily scriptable. 

cheers
stuart

---

From: Ed Summers <[email-masked]>
Date: Fri, 24 Jul 2009 21:25:27 -0400

Hi Chris:

Assuming your files are available in a directory like:

  http://ia360943.us.archive.org/1/items/talis_openlibrary_contribution/

why not use the manifest file:
  http://ia360943.us.archive.org/1/items/talis_openlibrary_contribution/talis_openlibrary_contribution_files.xml

It will help you determine what files need to be downloaded, and the
expected md5 checksum for the file once downloaded. This will at least
guarantee that you've got the files you need and that they've made it
over the wire ok. If you are really wanting to be thorough you could
run the PDFs through something like JHOVE to make sure they look legit
I guess.

//Ed

---

From: raj kumar <[email-masked]>
Date: Fri, 24 Jul 2009 18:43:45 -0700

> recommendations for the simplest way to maintain reasonable control  

A couple things:
If you already have archive.org identifiers picked out, you can use  
something like this python script to download them all from IA:
http://blog.openlibrary.org/2008/11/24/bulk-access-to-ocr-for-1-million-books/

You can use the archive.org advanced search engine to produce xml,  
json, or csv file with all identifiers for a particular contributing  
institution:
http://www.archive.org/advancedsearch.php

eg. all identifier for Montana State Library (http://www.archive.org/details/MontanaStateLibrary 
) as an xml file (change rows=10 to rows=10000 to get them all):
http://www.archive.org/advancedsearch.php?q=collection%3A%22montanastatelib%22&fl%5B%5D=creator&fl%5B%5D=identifier&fl%5B%5D=title&sort%5B%5D=&sort%5B%5D=&sort%5B%5D=&rows=10&fmt=xml&xmlsearch=Search

Also, if you have an archive.org identifier, then you can get the  
files.xml that contains md5 and sha1 hashes, so you can verify your  
download:

To pull the files.xml, use a /download/id/id_files.xml url. e.g.:

http://www.archive.org/download/librariesoffutur00lickuoft/librariesoffutur00lickuoft_files.xml

-raj

---

From: Eric Lease Morgan <[email-masked]>
Date: Sat, 25 Jul 2009 08:02:57 -0400

On Jul 24, 2009, at 9:43 PM, raj kumar wrote:

>> Over the next few years, I am tasked to download 30,000 archival  
>> masters from Internet Archive into an archive for long-term staff  
>> access that we may preserve with LOCKSS. These are masters of  
>> Montana state publications. I have a hierarchy in mind to receive  
>> these files. The hierarchy is state agency\year\title\pub_date\*.pdf.
>>
>> I am intending to download the files in batches of 200 - 500 pdfs,  
>> but am thinking that if I slot them automatically into the archive  
>> hierarchy, misplaced or missing files could be very hard to find as  
>> the total grows. I will be logging the downloads, which should give  
>> me some control. Are there other strategies for ensuring that I can  
>> readily correct download errors? I am looking for recommendations  
>> for the simplest way to maintain reasonable control over the  
>> download process.
>
> A couple things:
>

'Sounds fun, and such a project is something I advocate not only for  
retrospective preservation purposes put for general collection  
building as well, but that is another story.

Without some sort of metadata it will not be possible for you to save  
your files in the hierarchy outlined above. State agency. Year. Title.  
Publication date. One the other hand, if metadata containing these  
values is readily accessible in the downloaded file itself or, as Ed  
mentioned, a part of some sort of manifest (or MARC record), then you  
are golden. I used Raj's script as a model for a similar process [1]:

   * write a cool query against Open Library returning identifiers
   * feed identifiers to mirroring program; I used wget
   * download file as well as metadata
   * parse metadata and process associated file accordingly

If you're really luck, then the "cool query" written against Open  
Library will also return the necessary metadata and you could use that  
as a guide to save your file

Good luck.

[1] similar process - http://infomotions.com/blog/2009/06/interent-archive-content-in-discovery-systems/

-- 
Eric Lease Morgan