Subject: [CODE4LIB] WARC file format now ISO standard

From: "[email-masked]" <[email-masked]>
Date: Tue, 2 Jun 2009 13:59:32 -0700

hi code4lib,

if you're archiving web content, please use the WARC format.

thanks,
/[email-masked]

WARC File Format Published as an International Standard
http://netpreserve.org/press/pr20090601.php

ISO 28500:2009 specifies the WARC file format:

* to store both the payload content and control information from
   mainstream Internet application layer protocols, such as the
   Hypertext Transfer Protocol (HTTP), Domain Name System (DNS),
   and File Transfer Protocol (FTP);
* to store arbitrary metadata linked to other stored data
   (e.g. subject classifier, discovered language, encoding);
* to support data compression and maintain data record integrity;
* to store all control information from the harvesting protocol
   (e.g. request headers), not just response information;
* to store the results of data transformations linked to other
   stored data;
* to store a duplicate detection event linked to other stored
   data (to reduce storage in the presence of identical or
   substantially similar resources);
* to be extended without disruption to existing functionality;
* to support handling of overly long records by truncation or
   segmentation, where desired.

more info here:
http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml

---

From: Karen Coyle <[email-masked]>
Date: Tue, 2 Jun 2009 14:15:20 -0700

Unfortunately, being an ISO standard, to obtain it costs 118 CHF (about 
$110 USD). Hard to follow a standard you can't afford to read. Is there 
an online version somewhere?

kc

-- 
-----------------------------------
Karen Coyle / Digital Library Consultant
[email-masked] http://www.kcoyle.net
ph.: 510-540-7596   skype: kcoylenet
fx.: 510-848-3913
mo.: 510-435-8234
------------------------------------

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 2 Jun 2009 14:18:13 -0700

hi Karen,

understood.

the final draft of the spec is available here:
http://www.scribd.com/doc/4303719/WARC-ISO-28500-final-draft-v018-Zentveld-080618

and other (similar) versions here:
http://archive-access.sourceforge.net/warc/

/[email-masked]

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 2 Jun 2009 17:27:35 -0400

But you have to pay $200 for the document that lists changes from last draft 
to first official version.

(Ok, Ok, it was just a joke. But you do get the point.)

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 2 Jun 2009 14:35:34 -0700

point well taken. :)

there were no significant changes to the WARC format
between the last draft and the published standard.

you can use Heritrix WARCReader, or WARC Tools warcvalidator
to verify that you have created a valid WARC in accordance
with the spec.

/[email-masked]

On 6/2/09 2:27 PM, Ray Denenberg, Library of Congress wrote:
> But you have to pay $200 for the document that lists changes from last 
> draft to first official version.
> 
> ----- Original Message ----- From: "[email-masked]" <[email-masked]>
>>> Unfortunately, being an ISO standard, to obtain it costs 118 CHF 
>>> (about $110 USD). Hard to follow a standard you can't afford to read. 
>>> Is there an online version somewhere?
>>>
>>> kc
>>>
>>> [email-masked] wrote:
>>>> hi code4lib,
>>>>

---

From: David Fiander <[email-masked]>
Date: Tue, 2 Jun 2009 17:47:03 -0400

This is a common problem with ISO standards, and the common solution
is to do just this: release the final "draft" before it's approved by
ISO as an official standard. That's what the ISO Forth programming
language group did as well.

- David

On Tue, Jun 2, 2009 at 5:35 PM, [email-masked] <[email-masked]> wrote:
>>>> $110 USD). Hard to follow a standard you can't afford to read. Is there an
>>>> online version somewhere?
>>>>
>>>> kc
>>>>
>>>> [email-masked] wrote:
>>>>>
>>>>> hi code4lib,
>>>>>

---

From: Bill Dueber <[email-masked]>
Date: Wed, 3 Jun 2009 09:35:57 -0400

So, can we expect a leaked final "draft" of RDA, do you think :-)

On Tue, Jun 2, 2009 at 5:47 PM, David Fiander <[email-masked]> wrote:

> (about

-- 
Bill Dueber
Library Systems Programmer
University of Michigan Library

---

From: Chris Gray <[email-masked]>
Date: Wed, 3 Jun 2009 09:50:19 -0400

Bear in mind, too, that standards like this are written for people who 
will be writing low level software tools to work with the WARC format.  
If WARC gets widely adopted, the crucial knowledge for users will be 
refined out and presented in more digestible forms.  (Personally, I'll 
wait for W3Schools to do a tutorial.)

There's also usually a significant delay between striking a standard and 
the wide availability of the needed tools for common uses and 
platforms.  (As an exercise, list all the browsers fully implementing 
[Pick your favorite standard here.  I'll go with:-] XSLT, which became a 
standard in 1999.)

Chris Gray
Library Systems
University of Waterloo

"The nice thing about standards is that you have so many to choose from."
-Andrew Tanenbaum