Subject: [CODE4LIB] GPO PURLs

From: Keith Jenkins <[email-masked]>
Date: Tue, 25 Aug 2009 17:17:43 -0400

Is it just me, or is the GPO PURL resolver down?  I keep getting a
timeout error...

For example:
    http://purl.access.gpo.gov/GPO/LPS4468

Does anyone know of any alternate place to lookup the real URL for GPO PURLs?

Keith

---

From: Keith Jenkins <[email-masked]>
Date: Thu, 27 Aug 2009 16:37:14 -0400

Thanks to everyone who helped me confirm that the GPO PURL server is
down.  An official announcement on the GPO Listserv said:
  "The PURL Server is currently inaccessible. GPO is working with IT
staff to restore service as soon as possible. We regret any
inconvenience caused by the server problems. An updated listserv will
be sent once service is restored."

While the server is down, here is one workaround (thanks to Patricia Duplantis):
  1. Go to http://catalog.gpo.gov/
  2. Click "Advanced Search"
  3. Search for word in "URL/PURL", enter the PURL
  4. Click "Go"
  5. The original URL at the time of cataloging should appear in a 53x note.

This incident, however, illuminates a weakness in PURL systems: access
is broken when the PURL server breaks, even though the documents are
still online at their original URLs.

Maybe someone more familiar with PURL systems can tell me... is there
any way to harvest data from a PURL server, so that a backup/mirror
can be available?

Keith

---

From: Ed Summers <[email-masked]>
Date: Sat, 29 Aug 2009 08:59:52 -0400

> any way to harvest data from a PURL server, so that a backup/mirror

This would be a great question for the purl-dev discussion list too:

  http://www.purlz.org/mailman/listinfo/purl-dev

//Ed

---

From: James Jacobs <[email-masked]>
Date: Mon, 31 Aug 2009 15:06:29 -0700

Hi all, (cross-posted to purl-dev)

I'm a documents librarian (and member of the Depository Library Council) 
and usually just a lurker over here. Thanks Keith and Patricia for the 
easy workaround. I shared this with govdoc-l and on my blog:

http://freegovinfo.info/node/2704

See especially the comment that as of today, only 3,677 PURLs out of 
116,237 have been restored (3.1%). I would love to hear your 
thoughts/ideas for how this kind of critical system failure can be 
averted in the future from a technological standpoint. Is it possible to 
mirror a purl server? Will the same issue occur when GPO moves to 
handles in FDsys (http://www.handle.net/)? Will a distributed 
infrastructure as I've briefly mapped out be able to handle these types 
of critical system crashes better?

Please let me know and I'd be happy to share your ideas with GPO and the 
documents community.

Best,

James Jacobs

-- 
James R. Jacobs
International Documents Librarian
Green Library, Stanford University
P: (650) 725-1030 E: [email-masked]
AIM: LibrarianJames T: @freegovinfo

The more beautiful questions demand the more beautiful answers,
and if we can learn to ask them, we stand a chance of steering
clear of shipwreck on our jury-rigged and not so distant star.
--Lewis Lapham, Lapham's Quarterly I(3), Summer, 2008, p.17.

-----------------------------------------------
This message may have been intercepted and read by U.S. government
agencies including the FBI, CIA, and NSA without notice or warrant or
knowledge of sender or recipient.

  (\
{|||8-
  (/

---

From: Jonathan Lebreton <[email-masked]>
Date: Tue, 1 Sep 2009 09:13:57 -0400

This is indeed an interesting problem - we are all dependent on a
centralized service node.  

Just got off the phone with GPO 9 am 9/1/09.  
I was told they are now up to 50% or PURLs restored but the script is
running very slowly line-by-line since the server (they're updating the
production server while it is up) is experiencing unusually heavy load
from the user community and bots scheduled to troll at beginning of the
month.  

Jonathan LeBreton 
Sr. Associate University Librarian
Temple University Libraries
voice: 215-204-8231
fax: 215-204-5201
email:  [email-masked]
email:  [email-masked]

> -----Original Message-----
> From: Code for Libraries [mailto:[email-masked]] On Behalf
Of
> James Jacobs
> Sent: Monday, August 31, 2009 6:06 PM
> To: [email-masked]
> Subject: Re: [CODE4LIB] GPO PURLs
>

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 1 Sep 2009 11:53:12 -0400

My opinion...  Of course it's possible to mirror a purl server, although 
I can't say how easy it is for that particular software.  And yes, this 
problem could still occur with a different architecture that 
accomplishes the same basic thing, like FDsys.

If GPO is going to encourage linking via purl, as they have been for 
years, then GPO needs to take responsibility for mirroring their purl 
server and generally maintaining it with a very high level of 
reliability. This can be done with failover backup servers with mirrored 
data, ideally failover backup servers at distinct physical locations 
with seperate power and network connections.  Corporate IT does this for 
important services that demand 99.9%+ uptime all the time, there's 
nothing specially novel about doing it for the GPO purl server, the 
'best practices' are established. There's nothing to it except for that.

Of course, the organization involved needs to define what kind of uptime 
they expect (99.999%? 95%?) and provision resources (staff time and 
hardware etc) correspondingly.  If GPO lacks the resources to maintain 
the kind of uptime the community thinks should be maintained, then....   
well, that's how it goes.   Or if GPO hasn't even thought about it like 
this, because they lack the staff expertise to even plan accordingly...  
again, them's the breaks. Doing things right takes money, including 
money for staff with appropriate expertise.

Of course, one failure in X (10?) years is fairly good reliability... 
depending on how long it takes them to get everything back working 100%. 
If it's back by tomorrow, one outage in 10 years pretty good. If it 
takes a week to get back, not so good.

I am not going to get on the 'see, this shows why purl is bad' 
bandwagon. Sure, it's a single organizational point of failure. But 
there are many single points of failure in GPO's services, aside from 
purl. If it wasn't purl breaking, then the actual server that hosted the 
docs could still go down, the server the purl's are pointing to. Does 
GPO maintain appropriate failover backup for the actual document hosting 
servers? Who knows.  So if you want to talk about 'distributed 
infrastructure' it's not just about purl server, it's about document 
hosting, and probably a dozen other things GPO does. 

Sure distributed infrastructure would be cool, and would, for instance, 
help maintain access even if somehow GPO itself disappeared or became 
evil or something. But geeks like to talk about distributed 
infrastructure becuase it's an 'interesting problem', meaning that the 
standard 'best practice' way to do it _isn't_ entirely clear -- it's a 
more complicated problem then, for example, simply maintaining robust 
failover backup infrastructure. If GPO lacks the resources/expertise to 
even maintain robust failover backup infrastructure....  pressuring GPO 
(and/or the federal government that funds/oversees them!)  to adequately 
maintain a failover backup infrastructure seems a lot more realistic way 
to achieve reliable access than pressuring them to do R&D in as yet not 
entirely clear methods of creating a distributed infrastructure!

Jonathan

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 1 Sep 2009 12:00:37 -0400

I'd add that not only does it sound like GPO maintained no failover 
backup, it sounds, based on Jonathan Lebreton's report,  like they 
didn't even maintain an offline backup, since they're needing to 
regenerate the purl database from raw data, rather than simply restoring 
from a backup, which would generally be much quicker then the process 
that Jonathan Lebreton seems to be describing.

 From what info we have, it sounds like GPO simply, well, was very very 
far from 'best practices' for a service meant to be robustly reliable. 
On the other hand, we're just going from sort of third hand hearsay, 
maybe they were doing things more right than it sounds, but some kind of 
catastrophic unexpected 'perfect storm' still happened to bring 
everything down. Maybe 48 hours of outage in 10 years (how long has GPO 
purl been running? Have there been outages like this before?) is 
appropriate reliability for the level of importance of this service. I 
dunno.

Jonathan

Jonathan Lebreton wrote:
>> infrastructure as I've briefly mapped out be able to handle these
>>     
> types
>   
>>>    "The PURL Server is currently inaccessible. GPO is working with
>>>       
> IT
>   
>>> inconvenience caused by the server problems. An updated listserv
>>>       
> will
>   
>>> While the server is down, here is one workaround (thanks to Patricia
>>>       
>> Duplantis):
>>     
>>>    5. The original URL at the time of cataloging should appear in a
>>>       
>> 53x note.
>>     
>>> This incident, however, illuminates a weakness in PURL systems:
>>>       
>> access
>>     
>>> Maybe someone more familiar with PURL systems can tell me... is
>>>       
> there
>

---

From: Roy Tennant <[email-masked]>
Date: Tue, 1 Sep 2009 09:11:35 -0700

I think this episode also illustrates, once again, that preservation is not
about technology at all, it's about *institutional commitment*. The kind of
institutional commitment that would have implemented and maintained the
kinds of procedures that Jonathan described. Without institutional
commitment, no technology on earth can save you.
Roy

---

From: Roy Tennant <[email-masked]>
Date: Tue, 1 Sep 2009 09:11:35 -0700

---

From: Keith Jenkins <[email-masked]>
Date: Tue, 1 Sep 2009 12:36:48 -0400

On Tue, Sep 1, 2009 at 11:53 AM, Jonathan Rochkind<[email-masked]> wrote:
> depending on how long it takes them to get everything back working 100%. If
> it's back by tomorrow, one outage in 10 years pretty good. If it takes a
> week to get back, not so good.

It's been 8 days so far... hopefully it will we back to normal soon.
I'm not sure how long GPO has been serving PURLs, but if we assume
this is the first failure in 10 years, then that's still 99.8% uptime,
which isn't bad.  If the 0.2% were spread evenly across ten years, it
would hardly be noticable, but when it happens all at once, it
certainly does seem worse.

Keith

---

From: "Edward M. Corrado" <[email-masked]>
Date: Tue, 1 Sep 2009 15:29:02 -0400

Roy++

I agree while we might use technology to preserve things, it is only a 
tool to help preserve things. It is at best the how, not the which, 
what, why, and when.

Edward