Subject: [CODE4LIB] FW: PURL Server Update 2

From: "Duplantis, Patricia A." <[email-masked]>
Date: Tue, 1 Sep 2009 14:03:49 -0400

FYI...

Patricia A. Duplantis

Librarian (Automation)

Library Technical Services Support Section,

Library Technical Information Services

U.S. Government Printing Office

732 North Capitol Street NW

Mail Stop: IDBS

Washington, DC 20401

Phone: 202-512-2010 ext. 33268

Fax: 202-512-1432

Email: [email-masked]

Have you visited a Federal depository library lately?

http://www.gpoaccess.gov/libraries.html

Easy as FDL!

Federal Depository Libraries: Free Information, Dedicated Service, Limitless Possibilities

-----Original Message-----
From: Announcements from the Federal Depository Library Program [mailto:[email-masked]] On Behalf Of FDLP Listserv
Sent: Tuesday, September 01, 2009 1:46 PM
To: [email-masked]
Subject: PURL Server Update 2

This message was cross-posted to FDLP-L and GOVDOC-L.

-----------------------------------------------------

Since last Monday (August 24th), when the PURL server suffered a significant hardware failure, GPO staff has been working at the highest level of priority to re-establish server access.

Though the hardware configuration was restored, GPO has worked continuously, including this past weekend, to restore the system software configurations for this legacy application. Thus far this has resulted in re-established access to a small percentage of PURLs. Although back-ups of critical files ensured no loss of data, the reconfiguration of the software on the new server is still being enabled. GPO's approach is to properly construct a new configuration and to restore all the PURLs to enable use of the automated resolution process. GPO recognizes the criticality of having stable and redundant systems to access online content for the information community.

GPO knows that many institutions have automated URL checkers that run against the PURL server. Please be aware that the PURL restoration process is severely slowed by checkers repeatedly hitting the PURL server.

On- and off-site redundant back-up of all critical hardware and systems is and will continue to be performed by GPO. Additionally, GPO is exploring hosted services and expansion of real time failover capability. GPO is also finalizing requirements for migrating legacy applications, like PURLs, to FDsys for later system capability releases. Once FDsys is fully enabled, GPO will have migrated into a more modern and scalable/failover infrastructure.

The New Electronic Titles (NET) service will be delayed until the PURLs resolve properly. We sincerely regret the inconvenience caused by the PURL server outage. Additional postings will be made to keep the community informed as restoration of the server progresses.

_________________________________

If you have questions or comments, please use the askGPO help service

at: <http://www.gpoaccess.gov/help>. When submitting a question,

please choose the category "Federal Depository Libraries" and the

appropriate subcategory, if any, in order to ensure that your question

is routed to the correct area.

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 1 Sep 2009 15:44:56 -0400

Duplantis, Patricia A. wrote:
> On- and off-site redundant back-up of all critical hardware and systems is and will continue to be performed by GPO. 

I don't really understand how this is consistent with:

> Though the hardware configuration was restored, GPO has worked continuously, including this past weekend, to restore the system software configurations for this legacy application. Thus far this has resulted in re-established access to a small percentage of PURLs. Although back-ups of critical files ensured no loss of data, the reconfiguration of the software on the new server is still being enabled. GPO's approach is to properly construct a new configuration and to restore all the PURLs to enable use of the automated resolution process. 
>   

Doesn't sound like they had the right backups of the right stuff to me.  
Sounds like _something_ wasn't backed up, necessitating "reconfiguration 
of the software on the new server."  At least with the stuff I work 
with, I would have backed up anything requiring 'reconfiguration' too, 
so all it took was a reload of the backup. Of course hindsight is 20/20, 
hopefully GPO has improved their backup procedures, although I'd be more 
confident if they had something like "Based on this experience, we will 
be reviewing and enhancing our backup procedures to ensure that in 
similar circumstances in the future full restoration from backups can 
occur within minutes."

Jonathan

---

From: stuart yeates <[email-masked]>
Date: Wed, 2 Sep 2009 08:09:53 +1200

> GPO knows that many institutions have automated URL checkers 
 > that run against the PURL server. Please be aware that the
> PURL restoration process is severely slowed by checkers 
> repeatedly hitting the PURL server.

Presumably if there are any parties running automated tools which 
neglect to check robots.txt those institutions are going to be named and 
shamed and/or permanently filtered?

cheers
stuart
-- 
Stuart Yeates
http://www.nzetc.org/       New Zealand Electronic Text Centre
http://researcharchive.vuw.ac.nz/     Institutional Repository

---

From: "Edward M. Corrado" <[email-masked]>
Date: Tue, 1 Sep 2009 15:57:21 -0400

Jonathan Rochkind wrote:
>> On- and off-site redundant back-up of all critical hardware and 
>> systems is and will continue to be performed by GPO. 
>
>> Though the hardware configuration was restored, GPO has worked 
>> continuously, including this past weekend, to restore the system 
>> software configurations for this legacy application. Thus far this 
>> has resulted in re-established access to a small percentage of PURLs. 
>> Although back-ups of critical files ensured no loss of data, the 
>> reconfiguration of the software on the new server is still being 
>> enabled. GPO's approach is to properly construct a new configuration 
>> and to restore all the PURLs to enable use of the automated 
>> resolution process.   
>
> Doesn't sound like they had the right backups of the right stuff to 
> me.  Sounds like _something_ wasn't backed up, necessitating 
> "reconfiguration of the software on the new server."  At least with 
> the stuff I work with, I would have backed up anything requiring 
> 'reconfiguration' too, so all it took was a reload of the backup. Of 
> course hindsight is 20/20, hopefully GPO has improved their backup 
> procedures, although I'd be more confident if they had something like 
> "Based on this experience, we will be reviewing and enhancing our 
> backup procedures to ensure that in similar circumstances in the 
> future full restoration from backups can occur within minutes."
I'm with you Jonathan. While there are probably a lot of valid reasons 
behind what they are doing and why it was necessary to reconfigure the 
software for the new server, it is clear that their backup and recovery 
plan was not sufficient to get the service up and running in a timely 
manner. Obviously they are not the first place to have this problem (see 
Ma.gnoliaand JournalSpace for even worse situations) and they won't be 
the last.

This should be a lesson to each and everyone of us who are in charge of 
maintain systems to actually try to restore a system from your backups 
on a some-what regular basis to ensure that you can get important 
systems up and running in a timely manner. From personal experience, I 
can tell you that you can learn a lot from these tests, and on at least 
one occasion they saved my ____ when I had a critical system fail.

Edward

---

From: "Houghton,Andrew" <[email-masked]>
Date: Tue, 1 Sep 2009 16:47:41 -0400

> From: Code for Libraries [mailto:[email-masked]] On Behalf Of
> Edward M. Corrado
> Sent: Tuesday, September 01, 2009 3:57 PM
> To: [email-masked]
> Subject: Re: [CODE4LIB] FW: PURL Server Update 2
> 

I would be surprised if GPO wasn't backing up their data or even imaging
their server drives on a periodic basis.  However, in my experience with 
operation folks over many years, it isn't as simple as grabbing a backup
tape, putting it in a drive, and copying the data.  Just copying the
data, depending on how large it is could take some time.

1) Data drive fails. Grab a new drive from stock, reformat the drive,
   grab the backup tape, reload the data, then figure out what was lost
   between the time it was backed up and it went down.  If it were a SQL
   database can you recover from any intact journals?  Yes, then apply
   the journals.  What if this isn't the only service running on the
   server, you got to restore the data for those applications too.

2) OS drive fails. Grab a new drive from stock, reformat the drive, grab
   the backup tape or image and reload the OS, then figure out what
   security or application patches were applied between the time the 
   backup/image was taken and today.  Reinstall the missing patches.

3) Total hardware failure.  Do (1) and (2) above.  Your organization
   might keep pre-built spare servers around, but you still have to make
   sure that they are up-to-date with security and application patches,
   that they have the correct applications installed, you need to change
   the IP address, the server name and other little details like that.

4) Server compromised.  Worst case scenario.  They need to preserve all
   the drives so they can analyze them and turn over information to 
   police.  They are not going to trust the backup/image since they don't 
   know how long the server was compromised.  So they are most likely 
   going to rebuild the server from scratch and insure that it has *all* 
   the latest security and application patches, in addition to doing (1).

We had a research server get compromised a few years ago and it took
several weeks to get it back online due to rebuilding it from scratch.
Nothing is as simple as it seems...

Andy.

---

From: Kyle Banerjee <[email-masked]>
Date: Tue, 1 Sep 2009 14:44:02 -0700

>   police.

This is where written policies and reality often diverge. Getting LE
involved is tantamount to throwing away production equipment making a
bad situation much worse on multiple levels. I'd be willing to bet
that most people quietly clean and rebuild, especially if they're not
budgeted for a backup system...

kyle

---

From: "Edward M. Corrado" <[email-masked]>
Date: Tue, 1 Sep 2009 19:51:20 -0400

On Tue, Sep 1, 2009 at 4:47 PM, Houghton,Andrew<[email-masked]> wrote:

...
>   the latest security and application patches, in addition to doing (1).

There was nothing in the message they sent that leads me to believe
this was the case. I can only go on what they say, and what they said
was:

"Since last Monday (August 24th), when the PURL server suffered a
significant hardware failure, GPO staff has been working at the
highest level of priority to re-establish server access."

Thus I have to believe them that they did not have a compromised
server and instead they had a hardware failure. I have no idea why
they couldn't just restore from backup which would at least gotten
them back to where they were from the last backup (which presumably
was at most a week ago, if not someone should have a lot of explaining
to do to someone). Having to recreate a week of work wouldn't take
that long and presumably the system could have been used while that
happened. I hope the provide a post mortem because I like to learn fro
these types of things so I can be better prepared if something like
this happens to a system I'm responsible.

> Nothing is as simple as it seems...

Well, sometimes it is easier, but yes, I understand that sometimes
these are harder than one would think. This is why you have to make
sure, and test, that you have an adequate backup and recovery plan.
This is basic systems administration. 8+ days to recover from a
hardware issue a service that the GPO has been encouraging large
numbers of people to use, seems more than excessive. For whatever
reason they were not adequately prepared for "a significant hardware
failure" on this service.

Incidentally, this is one of the reasons I am very keen on
virtualization. Moving a VMware image (or what ever competitor you
prefer) to new hardware is a lot less complicated than moving physical
machines.

Edward

---

From: Ross Singer <[email-masked]>
Date: Tue, 1 Sep 2009 21:36:06 -0400

On Tue, Sep 1, 2009 at 7:51 PM, Edward M. Corrado<[email-masked]> wrote:

> to do to someone).

I didn't want to join this speculation party, but here goes.

It's quite possible that part of the problem here is that the
"significant hardware failure" meant that the replacement was a
completely different architecture (let's say for argument's sakes that
the server that failed was AS/400 and the replacement was Solaris on
an Intel server) because IT policy (or, you know, reality) dictated
that the old hardware would be replaced if it failed.

So then we're not just talking about backing up from tape -- things
need to be compiled -- there are perhaps problems with legacy C
libraries, character sets, *whatever*.

When I was working at Emory, we had a grant funded project that
indexed a handful of collections of SGML EAD files in an app called
iSearch (http://www.etymon.com/tr.html#).  When the (admittedly
neglected) VA Linux server it ran on had a major problem it was
insanely non-trivial to get this completely orphaned application
running in a contemporary operating system (in this case, RedHat).
Old versions of iSearch /would not under any circumstances/ compile --
new ones couldn't read the old data.  The application was down for --
I don't know -- months, IIRC.  Granted, this was nowhere near the
priority of GPO's PURL server -- but you can't stop time to solve
these sorts of Catch-22s, either.

Things happen.  Catastrophes generally have the added advantage of
ensuring they don't happen again for a while.

-Ross.

---

From: "Edward M. Corrado" <[email-masked]>
Date: Tue, 1 Sep 2009 22:14:42 -0400

> ensuring they don't happen again for a while.

Ross,

For arguments sake, let's  say youe speculation is correct. In this  
case this would be a huge management failure. If you are running a  
major service you need to plan for hardware failure. This isn't rocket  
science, this is sysadmin 101. Hardware fails.

Edward

---

From: Chris King <[email-masked]>
Date: Wed, 2 Sep 2009 09:02:06 -0400

stuart yeates wrote:
>> PURL restoration process is severely slowed by checkers repeatedly 
>> hitting the PURL server.
> 

Any "out-of-the-box" installation of Innovative's Millennium that 
includes "URL Verify" is set automatically to check all 856 links on the 
first of the month.

Chris

-- 
Christopher King
Public Services Librarian
Appalachian School of Law Library
1221 Edgewater Drive
Grundy VA 24614
(276) 935-6688 x1314
[email-masked]

---

From: David Fiander <[email-masked]>
Date: Wed, 2 Sep 2009 09:31:32 -0400

And how does that absolve such sites from being "named and shamed" or
permanently filtered? If Millenium is acting like a robot in its
monthly maintenance processes, then it should be checking robots.txt.

On Wed, Sep 2, 2009 at 9:02 AM, Chris King<[email-masked]> wrote:
>>  > that run against the PURL server. Please be aware that the
>>>
>> Presumably if there are any parties running automated tools which neglect
>> to check robots.txt those institutions are going to be named and shamed
>> and/or permanently filtered?
>
> Any "out-of-the-box" installation of Innovative's Millennium that includes
> "URL Verify" is set automatically to check all 856 links on the first of the
> month.
>
> Chris
>
>
>
> --
> Christopher King

---

From: "Houghton,Andrew" <[email-masked]>
Date: Wed, 2 Sep 2009 09:55:39 -0400

> David Fiander
> Sent: Wednesday, September 02, 2009 9:32 AM
> If Millenium is acting like a robot in its

User agents are *not required* to check robots.txt nor are servers
*required* to provide a robots.txt.  There are no expectations for
robots.txt other than a gentlemen's agreement that if a server
provides one it should be consulted when any content is access from 
the server.

However, if you have publicly accessible URIs it is highly unlikely
that you would restrict access to those of URIs in your robots.txt.
It kink-of defeats the purpose of the URIs being *public*.  You
might put those URIs in robots.txt when the URIs have been deprecated 
and are being redirected to another URI, e.g., you redesigned your
Web site, but 1) I would argue that it would be better for your user
agents to see the redirect so they can update themselves, and 2) GPO
is running a PURL server, where the URIs are suppose to be *permanent*
and *publicly* accessible.

Robots.txt is a nice idea, but if you are having an issue with a user
agent, the network folks will most likely update the router rules to 
block the traffic rather than let it get thru to the server.

<http://www.robotstxt.org/orig.html>

Andy.

---

From: Thomas Dowling <[email-masked]>
Date: Wed, 2 Sep 2009 10:24:33 -0400

On 09/02/2009 09:55 AM, Houghton,Andrew wrote:
> User agents are *not required* to check robots.txt...

That's probably why David said "should".  Or, "SHOULD".  :-)

The III crawler has been a pain for years and Innovative has shown no interest
in cleaning it up.  It not only ignores robots.txt, but it hits target servers
just as fast and hard as it can.  If you have a lot of links that a lot of III
catalogs check, its behavior is indistinguisable from a DOS attack.  (I know
because our journals server often used to crash about 2:00am on the first of
the month...)

> It kink-of defeats the purpose of the URIs being *public*.

True.  I think it's reasonable at this point to ask crawlers to support both
robots.txt, and also sitemap.xml for information on public pages and how often
to check them.

-- 
Thomas Dowling
[email-masked]

---

From: "Houghton,Andrew" <[email-masked]>
Date: Wed, 2 Sep 2009 11:26:58 -0400

> Sent: Wednesday, September 02, 2009 10:25 AM
> catalogs check, its behavior is indistinguishable from a DOS attack.  (I
> know

I see that I didn't fully make the connection to the point I was 
making... which is that there are hardware solutions to these 
issues rather than using robots.txt or sitemap.xml.  If a user 
agent is a problem, then network folks should change the router 
to ignore the user agent or reduce the number of requests it is
allowed to make to the server.

In the case you point to with III hitting the server as fast as 
it can and it looking like a DOS attack to the network which
caused the server to crash, then 1) the router hasn't been setup 
to impose throttling limits on user agents, and 2) the server 
probably isn't part of a server farm that is being load balanced.

In the case of GPO, they mentioned or implied, that they were
having contention issues with user agents hitting the server
while trying to restore the data.  This contention could be
mitigated by imposing lower throttling limits in the router on 
user agents until the data is restored and then raising the 
limits back to the whatever their prescribed SLA (service level
agreement) was.

You really don't need to have a document on the server to tell 
user agents what to do.  You can and should impose a network 
policy on user agents which is far better solution in my opinion.

Andy.

---

From: Bill Dueber <[email-masked]>
Date: Wed, 2 Sep 2009 12:09:49 -0400

Andy, I think there are three issues here:

1. Should the GPO put in place, at least at the moment, some throttling for
user agents behaving like dicks?
2. Should III (and others), when acting as a user agent, be such a dick?
3. How do I know if I'm being a dick?

The answers folks are offering, I think, are (1) Yes, (2) No, and (3) It's
hard to know, but you should always check robots.txt, and you should always
throttle yourself to a reasonable level unless you know the target can take
the abuse.

For the majority of the web, for the majority of the time, basic courtesy
and the gentleperson's agreement ensconced in robots.txt works fine -- most
folks who write user agents don't want to be dicks. When this informality
doesn't work, as you point out, there are solutions you can implement at
some edge of your network. Of course, at that point the requests are already
flooding through to *somewhere*, so getting things stopped as close to the
point of origin is key.

-- 
Bill Dueber
Library Systems Programmer
University of Michigan Library

---

From: David Jones <[email-masked]>
Date: Wed, 2 Sep 2009 10:33:45 -0700

>>> On Wed, Sep 2, 2009 at  9:09 AM, Bill Dueber <[email-masked]> wrote: 

In Innovative's defense, the URL verifier is a very limited and configurable 'robot'. 

- It only verifies the HTTP header response codes for the URLs in the library's records. It does not crawl, fetch or spider from there.
- It can be configured by Innovative to run at different times. It can be configured to not run automatically at all.
- It can be configured to not crawl URLs that match certain patterns (e.g., handle.net, purl.org, purl.access.gpo.gov) that will always return a non-200 response or are otherwise pointless to verify (e.g., loc.gov/catdir/).

It's a tool that is configurable. It's the libraries' responsibility to use it wisely.

I do believe that Innovative should do more to educate their users on its appropriate use and should ship it disabled by default, but that doesn't release the libraries from their ultimate responsibility.

HTH,
David

_____________________________________________________________________
David Jones                                     mailto:[email-masked]
Library Systems Manager                  http://www.scu.edu/library/
University Library                               fax:   408-551-1805
Santa Clara University                            phone: 408-551-7167
500 El Camino Real
Santa Clara CA 95053-0500
_____________________________________________________________________
Reality is that which, when you stop believing in it, doesn't go away.
-- Philip K. Dick

---

From: Jonathan Lebreton <[email-masked]>
Date: Wed, 2 Sep 2009 15:50:43 -0400

Andrew Houghton wrote: 

> In the case of GPO, they mentioned or implied, that they were having 
> contention issues with user agents hitting the server while trying to 
> restore the data.  This contention could be mitigated by imposing 
> lower throttling limits in the router on user agents until the data is

> restored and then raising the limits back to the whatever their 
> prescribed SLA (service level
> agreement) was.

The GPO tech I spoke with mentioned this contention issue explicitly.
He had just emerged from a meeting on the PURL problem (which meeting
sounded like an all-hands on deck affair).  He mentioned that there had
been discussion of the contention issue in the meeting but that they had
decided not to block the offending IPs (whether because they could not
do so effectively in time or because of a philosophical issue I did not
inquire).  Throttling the user agents  was not mentioned to me as a
possiblity.  

In fact, unless I'm mistaken, the PURL server does appear to be
completely inaccessible now, in advance of advertised downtime this
afternoon 5-7 EST  

Jonathan LeBreton 
Sr. Associate University Librarian
Temple University Libraries
voice: 215-204-8231
fax: 215-204-5201
email:  [email-masked]
email:  [email-masked]

> Houghton,Andrew
> Sent: Wednesday, September 02, 2009 11:27 AM
> > The III crawler has been a pain for years and Innovative has shown
no
> > interest
> > in cleaning it up.  It not only ignores robots.txt, but it hits
> target
> > servers
> > just as fast and hard as it can.  If you have a lot of links that a
> lot
> > of III