Subject: Re: [CODE4LIB] marc21 and usmarc

From: "Brown, Alan" <[email-masked]>
Date: Fri, 23 Jan 2009 09:39:54 -0000

At 22 January 2009 16:28 Eric Lease Morgan wrote:

> Does anybody here know the difference between MARC21 and USMARC?
> 
> I am munging sets of MARC bibliographic data from a III catalog with
> holdings data from the same. I am using MARC::Batch to read my bib'
> data (with both strict and warnings turned off), insert 853 and 863
> fields, and writing the data using the as_usmarc method. Therefore, I
> think I am creating USMARC files. I can then use marcdump to... dump
> the records. It returns 0 errors.     

Eric, This isn't an encoding thing is it? I know that a number of III
catalogues still encode their diacritics using the MARC8 version of
USMARC. We have changed ours to Unicode now, but we did have an issue of
the catalogue outputting unicode records that weren't tagged as such in
the leader and so couldn't be identified as proper MARC21 (current
version of USMARC). III have solved this with their latest release. This
issue had me scratching my head with a lot of my MARC::Record scripts,
but generally they failed quite spectacularly.

regards

Alan Brown
-- 
Alan Brown
Library Systems Liaison Officer
Resource Services
Bury Libraries
Textile Hall
Manchester Rd
Bury
BL9 0DG
Tel 0161 253 5877
Fax 0161 253 6003
http://www.bury.gov.uk/libraries
http://library.bury.gov.uk
-----------------------------------------------------------------
Why not visit our website www.bury.gov.uk
-----------------------------------------------------------------
The information contained in this e-mail and any files transmitted
with it is for the intended recipient(s) alone. It may contain
confidential information that is exempt from the disclosure under
English law and may also be covered by legal,professional or other privilege.
If you are not the intended recipient, you must not copy, distribute or take any
action in reliance on it. 
If you have received this e-mail in error, please notify us immediately by using 
the reply facility on your e-mail system.
If this message is being transmitted over the Internet, be aware that it may be 
intercepted by third parties.
As a public body, the Council may be required to disclose this e-mail or any 
response to it under the Freedom of Information Act 2000 unless the information
in it is covered by one of the exemptions in the Act. By responding to this
e-mail you accept that your response may be subject of recording/monitoring to
ensure compliance with the Council's ICT Security Policy. 
Electronic service accepted only at [email-masked] and on fax number 
0161 253 5119 .
*************************************************************

---

From: Eric Lease Morgan <[email-masked]>
Date: Fri, 23 Jan 2009 06:52:42 -0500

> but generally they failed quite spectacularly.

Actually, I believe I am suffering from a number of different types of
errors in my MARC data: 1) encoding issues (MARC8 versus UTF-8), 2)
syntactical errors (lack of periods, invalid choices of indicators, etc.),
3) incorrect data types (strings entered into fields denoted for integers,
etc.) Just about the only thing I haven't encountered are structural errors
such as invalid leader, and this doesn't even take into account possible
data entry errors (author is Franklin when Twain was entered).

Yes, I do have an encoding issue. All of my incoming records are in MARC8.
I'm not sure, but I think the Primo tool expects UTF-8. I can easily update
the encoding bit (change leader position 09 from blank to a), but this does
not change any actual encoding in the bibliographic section of my data.
Consequently, after updating the encoding bit and looping through my munged
data MARC::Record chokes on records with the following error where UTF-8 is
denoted but include MARC8 characters:

  utf8 "\xE8" does not map to Unicode at
  /usr/lib/perl5/5.8.8/i686-linux/Encode.pm line 166.

Upon looking at the raw MARC see the the offending record includes the word
Münich. What can I do to transform MARC8 data into UTF-8? What can I do to
trap the error above, and skip these invalid records?

-- 
Eric Lease Morgan

---

From: Tod Olson <[email-masked]>
Date: Fri, 23 Jan 2009 06:14:35 -0600

> trap the error above, and skip these invalid records?

We've had good luck with the yaz-marcdump utility that's included with  
the YAZ toolkit.  We're  using it to convert our exported Horizon  
records from MARC8 to UTF-8 before we import into AquaBrowser.  The  
tool is easy to compile, blindingly fast, forgiving of common MARC  
errors, and changes the coding correctly. It's been serving us well.

-Tod

Tod Olson <[email-masked]>
Systems Librarian
University of Chicago Library

---

From: Jonathan Rochkind <[email-masked]>
Date: Fri, 23 Jan 2009 10:32:20 -0500

This is definitely a pretty typical experience, alas.

Despite all of the library commmunities voiced obsession with doing 
things 'by the book' according to standards, anyone that's actually 
tried to work with an actually existing large corpus of MARC data.... 
finds that is is all over the place, and very non-compliant in many ways.

One of the most annoying things to deal with is that encoding issue. A 
US-MARC/MARC21 record can actually be in MARC-8 encoding OR in UTF-8, 
and there is actually a field (fixed field I think) to declare which 
encoding is used. But in actually existing MARC records, it is not 
uncommon for a record to declare itself as being in one encoding, but 
actually is in the other. This makes MARC records very difficult to deal 
with, definitely.

Jonathan

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu

---

From: Mark Jordan <[email-masked]>
Date: Fri, 23 Jan 2009 08:57:03 -0800

----- "Jonathan Rochkind" <[email-masked]> wrote:

 A
> encoding is used. 

leader pos 09

Mark

---

From: Kyle Banerjee <[email-masked]>
Date: Fri, 23 Jan 2009 08:58:37 -0800

> data entry errors (author is Franklin when Twain was entered).

This MARC stuff is more confusing than it needs to be. As far as the
original question about the difference between USMARC and MARC21,
there is none for all practical purposes. In the mid 90's, the USMARC
and CANMARC communities tried to eliminate differences between them to
improve standardization. The outcome was called MARC21.

Structurally, it's all the same stuff. The differences they're talking
about resolving between CANMARC and USMARC refer to what MARC tags
correspond with which data fields rather than substantive differences
in structure..

The MARC format itself is just a container, and it does not require
that the fields be numeric -- that title is in 245 is simply a
cataloging practice. Although catalogers always use numbers, the
structure of the MARC format allows other characters to be used.

> Despite all of the library commmunities voiced obsession with doing things
> 'by the book' according to standards, anyone that's actually tried to work
> with an actually existing large corpus of MARC data.... finds that is is all
> over the place, and very non-compliant in many ways.

This sums up the problem nicely. For all their carping about detail,
accuracy, and the like catalogers are not consistent once you get
beyond a few basic metadata fields.

This is because catalogers like to believe they can exert far more
bibliographic control than is realistically possible. As a result,
they have developed hopelessly complex procedures that would cause any
Byzantine ruler to break down in tears.

Have you ever seen the books catalogers do to do their jobs? There's
not just AACR2, but also the Library of Congress Rule Interpretations,
the Subject Cataloging Manual, LCCS, Cutter Tables, code lists for
various fields, CONSER manual, Romanization tables, Bib formats and
standards, and there are a zillion specialized resources. BTW, there
is nothing unusual about using all the resources mentioned above to
catalog a single piece.

If you mention inconsistency to a cataloger, you'll trigger a
monologue on quality control and who isn't doing what properly.
However, you know the system is poorly designed when people who've
been cataloging for more than 10 years can't get it right. In any
case, the consistency is so bad that you're better off running
heuristic procedures on data strings than trusting special purpose
fields. Even fields as basic as encoding level that all catalogers
know are not trustworthy enough to rely on.

Catalogers. Can't live with 'em. Can't shoot 'em....

kyle (ex-cataloger who created literally thousands of original records
in OCLC during a former lifetime)
-- 
----------------------------------------------------------
Kyle Banerjee
Digital Services Program Manager
Orbis Cascade Alliance
[email-masked] / 541.359.9599

---

From: Karen Coyle <[email-masked]>
Date: Tue, 27 Jan 2009 07:57:39 -0800

There's also the fact that validation must be built into individual 
systems, since there is no schema or dtd against which everyone can 
validate the MARC record. Thus, every system validates somewhat 
differently (and some not at all). Although MARC is a machine-readable 
record, we don't have a machine-actionable standard for our data. This, 
I must admit, just boggles the mind in this day and age.

kc

-- 
-----------------------------------
Karen Coyle / Digital Library Consultant
[email-masked] http://www.kcoyle.net
ph.: 510-540-7596   skype: kcoylenet
fx.: 510-848-3913
mo.: 510-435-8234
------------------------------------

---

From: Eric Lease Morgan <[email-masked]>
Date: Tue, 27 Jan 2009 11:04:33 -0500

> I must admit, just boggles the mind in this day and age.

Can somebody say "MARCXML or MODS complete with a schema"? Such solutions
offer at least syntactic validation if not also semantic validation. Oh
well.

-- 
Eric Lease Morgan

---

From: Alexander Johannesen <[email-masked]>
Date: Tue, 27 Jan 2009 17:09:09 +0100

On Tue, Jan 27, 2009 at 17:04, Eric Lease Morgan <[email-masked]> wrote:
> Can somebody say "MARCXML or MODS complete with a schema"?

Well, we can say it, and I think we *have* said it for a very long
time, but it doesn't seem to change anything. Damn those words.

> Such solutions offer at least syntactic validation if not also
> semantic validation. Oh well.

I would say a little bit more than "oh well" (but I don't really have;
you know how I feel :), but I would love to hear what the vendors are
thinking about this all. They seem to very, very quiet about it all
(without speculating to why ...)

regards,

Alex
-- 
---------------------------------------------------------------------------
 Project Wrangler, SOA, Information Alchemist, UX, RESTafarian, Topic Maps
------------------------------------------ http://shelter.nu/blog/ --------

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 27 Jan 2009 12:06:17 -0500

> (without speculating to why ...)

Because their customers are not demanding it, and they often don't have 
the technical expertise to understand why it matters anyway.  But mainly 
because their customers are not demanding it.

Jonathan

---

From: Alexander Johannesen <[email-masked]>
Date: Tue, 27 Jan 2009 18:11:04 +0100

On Tue, Jan 27, 2009 at 18:06, Jonathan Rochkind <[email-masked]> wrote:
> Because their customers are not demanding it, and they
> often don't have the technical expertise to understand
> why it matters anyway.  But mainly because
> their customers are not demanding it.

So, um, could librarians everywhere start being just a tad bit more
demanding about this stuff? You know, before your profession becomes
obsoleted from this planet?

Actually, I was wondering what areas MODS can't handle which MARC
does, hijack and / or change MODS to fit it (what I know of it seems a
bit limiting, but through XML certainly extensible). Shouldn't folks
start by demanding at least MODS (or XOBIS if we're *really* crazy :)?

Regards,

Alex
------------------------------------------ http://shelter.nu/blog/ --------

---

From: Kyle Banerjee <[email-masked]>
Date: Tue, 27 Jan 2009 09:56:10 -0800

> obsoleted from this planet?

The basic problem is that the real value of a catalog is its
consistency, and the legacy data in these fields is already too
inconsistent to have much value.

The good news is that despite the fact that some fields are just
hopeless, many things in the catalog are decent. The consistency of
structure and the quality of content in author, title, and subject
fields is significantly better in a library catalog than it is in
other sources. That is why faceting works pretty well.

Where things really start falling apart are the special purpose
fields. Between some libraries deciding these aren't worth filling in,
catalogers not being aware of them, etc, they are inconsistently
applied. Even among the few that entered relatively consistently (such
as 043), there is usually a better source of data (e.g. 6XX |z)
somewhere else in the record

If data aren't consistent enough, it's just not useful no matter how
good your system is. Plus, any system that actually knew all the MARC
tag and indicator twiddling would be hideously complex. Do you really
want to design a system that exploits the special tag indicating if a
piece is a Festschrift, has color illustrations, or a portrait? Do you
think that exploiting the full power of the 007 would not confuse the
heck out of everyone or that a special field which exists only to
discuss funding associated with the content of the piece needs special
treatment and that any work units or task numbers associated with that
funding need to be stored separately?

The data in the fields above are hideously inconsistent, but even in a
perfect world where it is all correct, I'd argue it's still not worth
screwing with -- especially since there are plenty of basic aspects of
the patron and staff experience that need serious work.

> start by demanding at least MODS (or XOBIS if we're *really* crazy :)?

Frankly, the important stuff is there and it would be possible to
modify MODS to accommodate the things that aren't. The main reason
you're stuck with MARC is that there are a lot of legacy loaders out
there so even if all transmission was done in MODS, you'd still have
to convert it to MARC.

There are arguments to do so, but the business case is not strong.
That data providers won't send MODS until libraries demand it.
Libraries won't demand it until their systems use it. Systems won't
use it until libraries demand it because that's what their data
providers require.

It's a vicious circle, so we're stuck with MARC. The only people who
aren't happy with this arrangement are those who are trying to create
something new. Many librarians who think they use MARC every day have
no idea that it is a binary format that is unfriendly to eyes and
machines.

Just because something makes sense does not mean it will happen. The
QWERTY keyboard is terrible, every modern operating system can support
the technically superior Dvorak layout, yet we never switch. I'm a bit
more optimistic getting the content of the catalog record into a
container better than MARC, but that will take a long time.

kyle

---

From: Alexander Johannesen <[email-masked]>
Date: Tue, 27 Jan 2009 19:14:13 +0100

> There are arguments to do so, but the business case is not strong.

Well, I'd say the future of the library world is a good business case,
and I know several people (high and low) fully aware of it, but I
think it's hard to take any step in either direction that would be
deemed worth it. Toguh one, indeed.

> use it until libraries demand it because that's what their data

Well, I've been yelling for vendors to get more involved for a long
time, but there's a lot of blankness coming from them. I guess they're
happy with the current tie to MARC (binding the libraries to them
forever) until the business is gone ...

> something new. Many librarians who think they use MARC every day
> have no idea that it is a binary format that is unfriendly to eyes and
> machines.

MARC may be MAchine Readable, but not MAchine Understandable or even
MAchine Usable.

I had an idea some time ago to create a dummy / fake MARC record with
much more to it (like extensions and special tags systems can react
to, such as validation) and pass it around the infrastructure to see
what in it survives (the golden rule is to ignore what you don't
understand, although I know a few MARC systems who filter out what
they don't understand (!!!) because, well, these systems were mostly
built back when a megabyte of storage and / or memory had a price of
about a cataloger or two. Friggin' crazies!). Anyone in? :)

Regards,

Alex
------------------------------------------ http://shelter.nu/blog/ --------

---

From: Nate Vack <[email-masked]>
Date: Tue, 27 Jan 2009 12:48:47 -0600

> other sources. That is why faceting works pretty well.

This is a very good point -- the parts of the MARC record that people
actually *use* tend to be reasonably OK. When they aren't, they tend
to get corrected. Hence, you can use the good parts of the records to
do useful things.

The crappy, inconsistent bits of the record? Ignore 'em. Don't expose
them. Let actual use dictate what gets exposed, and how.

Sure, MARC is an ugly, nasty binary format, and could be replaced by
Something Better. MP3 is another nasty format -- but somehow, Ogg
Vorbis hasn't replaced it, despite its technical superiority. iTunes
manages to be a very good media player, even though MP3s suck as much
as they did 15 years ago. Good collection search and management tools
can be built on top of MARC.

While it's _fun_ to complain about ugly data structures, it's _more
productive_ to build libraries and products that abstract away the
ugliness.

I don't think the merits of MARC will determine the future of
libraries any more than the merits of MP3 will determine the future of
the music industry.

Cheers,
-Nate

---

From: Karen Coyle <[email-masked]>
Date: Tue, 27 Jan 2009 10:50:41 -0800

> there so even if all transmission was done in MODS, you'd still have
I am less optimistic about MODS than Kyle. Having watched it be made, I 
think it's more than just a bit of a kludge, and carries forward a lot 
of the problems of MARC21. I also don't think that it has a strong model 
or philosophy behind it. I think we can do much, much better. What is 
stopping us is what comes up here: you can create a better record, but 
that doesn't mean that library systems will use it. Even so, I'm up for 
trying to create that better record, and I'm even up for creating one 
that is compatible with library cataloging practices, at least in their 
intent. Some of us talked about this on the exhibits floor of ALA just 
in the last few days.

I will start by re-organizing a document I did a few years ago but that 
was never publicly released. I'll do a new, public version and post it, 
then wiki it so we can have the discussion. Also, I think that the 
cataloger scenarios in the DC/RDA wiki are beginning to show what one 
can do with the FRBR assumption behind the record.

kc

---

From: Gabriel Farrell <[email-masked]>
Date: Tue, 27 Jan 2009 17:00:08 -0500

> can do with the FRBR assumption behind the record.

This sounds like a great idea, Karen, and I'm looking forward to seeing         
the document and discussing it.  If a record format can demonstrate a           
significant leap forward then it will be adopted.                               

Keep this list in the loop about the public version.

Gabriel

---

From: "Riley, Jenn" <[email-masked]>
Date: Tue, 27 Jan 2009 18:32:48 -0500

> or philosophy behind it. I think we can do much, much better. 

I agree with Karen's characterization of how MODS has developed since its inception. The good news is that will hopefully change soon. The newly formed MODS/MADS Editorial Committee is developing a design principles document that will help guide future versions of MODS and MADS. We'd gratefully welcome feedback on what those principles should be. The MODS list [1] is probably the best place for that discussion to take place, but some of the Committee members are on this list too, so ideas brought up here won't be lost on that group. I suspect we'll have a draft document to share on the MODS list in the next month or so, but ideas for what should be on it before then are even more valuable. :-)

[1] http://listserv.loc.gov/listarch/mods.html

Jenn 
(Chair, MODS/MADS Editorial Committee)

========================
Jenn Riley
Metadata Librarian
Digital Library Program
Indiana University - Bloomington
Wells Library W501
(812) 856-5759
www.dlib.indiana.edu

Inquiring Librarian blog: www.inquiringlibrarian.blogspot.com