Subject: [CODE4LIB] [Fwd: [ol-tech] Modified RDF/XML api]

From: Karen Coyle <[email-masked]>
Date: Mon, 10 Aug 2009 13:42:44 -0700

I'd really appreciate some testing and any comments folks have. Thanks, kc

-------- Original Message --------
Subject: 	[ol-tech] Modified RDF/XML api
Date: 	Mon, 10 Aug 2009 13:32:25 -0700
From: 	Karen Coyle <[email-masked]>
Reply-To: 	Open Library -- technical discussion <[email-masked]>
To: 	Open Library -- technical discussion <[email-masked]>

The Open Library API that returns bibliographic data in RDF/XML has been 
greatly modified. To make use of it, simply add '.rdf' to the end of the 
bibliographic identifier:
    http://openlibrary.org/b/OL6807502M.rdf

and of course it can be used in a curl command as:
    curl 'http://openlibrary.org/b/OL6807502M.rdf'

The output relies heavily on DCMI Metadata Terms ("dcterms") [1] for its 
properties, but also makes use of Bibliontology [2] and RDA Vocabulary 
[3] where those present a better fit. Because it is in XML, this 
provides what we hope will be an easy-to-use format that makes use of 
commonly known metadata terms.

Not all data elements are included in the output, in particular there is 
no attempt at this point to export the tables of contents fields. That 
could be a future enhancement.

We would appreciate any comments you have on this API, as well as 
suggestions and corrections.

There are a number of different bibliographic formats that we could 
return in our APIs: MARCXML, MODS, EndNote import, ... I would be 
particularly interested to hear if there are particular formats that you 
would find immediately useful.

Thanks,
kc
[1] http://dublincore.org/documents/dcmi-terms/
[2] http://bibliontology.com/
[3] http://metadataregistry.org/schema/show/id/1.html

-- 
-----------------------------------
Karen Coyle / Digital Library Consultant
[email-masked] http://www.kcoyle.net
ph.: 510-540-7596   skype: kcoylenet
fx.: 510-848-3913
mo.: 510-435-8234
------------------------------------

_______________________________________________
Ol-tech mailing list
[email-masked]
http://mail.archive.org/cgi-bin/mailman/listinfo/ol-tech

-- 
-----------------------------------
Karen Coyle / Digital Library Consultant
[email-masked] http://www.kcoyle.net
ph.: 510-540-7596   skype: kcoylenet
fx.: 510-848-3913
mo.: 510-435-8234
------------------------------------

---

From: Aaron Rubinstein <[email-masked]>
Date: Mon, 10 Aug 2009 16:55:49 -0400

Thanks, Karen, this is a really exciting development.

I was wondering, though, whether there is a plan to link any of the  
Open Library data with existing linked data sets, for example:  
dbpedia.org for authors/titles and id.loc.gov for subjects?  I realize  
that exposing data in RDF and exposing linked open data do not  
necessarily need to be synonymous but the richness of the Open Library  
data set begs the question.

Thanks again!

Aaron

-- 
Aaron Rubinstein
Digital Project Manager
W.E.B. Du Bois - Verizon Digitization Project
Special Collections and University Archives
University of Massachusetts, Amherst
Tel: (413)545-9637
Email: [email-masked]
Web: http://www.library.umass.edu/spcoll/

Quoting Karen Coyle <[email-masked]>:

> The Open Library API that returns bibliographic data in RDF/XML has  
> been greatly modified. To make use of it, simply add '.rdf' to the  
> end of the bibliographic identifier:
> The output relies heavily on DCMI Metadata Terms ("dcterms") [1] for  
> its properties, but also makes use of Bibliontology [2] and RDA  
> Vocabulary [3] where those present a better fit. Because it is in  
> XML, this provides what we hope will be an easy-to-use format that  
> makes use of commonly known metadata terms.
>
> Not all data elements are included in the output, in particular  
> there is no attempt at this point to export the tables of contents  
> fields. That could be a future enhancement.
>
> particularly interested to hear if there are particular formats that  
> you would find immediately useful.
>
> Thanks,
> kc

---

From: Karen Coyle <[email-masked]>
Date: Mon, 10 Aug 2009 15:15:58 -0700

Aaron,

there has been talk about exposing the data in Open library as LOD, but 
the project is currently focused more on creating an interactive site 
for humans rather than a manipulable data store, although ideally those 
two could co-exist. Right now there is some linking to Wikipedia 
personal names but not to dbpedia (as far as I know, but things change 
quickly...) although as you know the Wikipedia link is just one short 
step away from dbpedia. The number of names in Wiki/db is relatively 
small, and the programmer working on the data manipulation, Edward 
Betts, has found about 80K matches. Compared to the number of names in 
the LC name authorities file that is really small. I would like to make 
the link from the names in the bib records to the LCNA records to pick 
up those record numbers as identifiers. No one wants to use the "library 
form" in name displays, so without those identifiers we will lose the 
uniqueness that name authority provides. Presumably, LCNA will 
eventually be online and the data could be linked, thus picking up 
alternative name forms. Similar experimentation is being done with place 
names from subject headings.

The OL group has talked about linking to id.loc.gov, but in fact  the 
project folks are mainly interested in getting away from the LCSH 
structuring of subjects. I admit that I find it hard to defend LCSH in 
the discussions about subjects and subject searching. ;-) I wish that we 
had better access to both LC and Dewey classifications: the class 
numbers, what they mean in words, and how they link to subject headings.

For anyone who wants to use the OL as simply data, there is a full 
database dump done periodically, which you can find here:
   http://openlibrary.org/dev/docs/jsondump

I don't know much about it, but there is work being done with these 
dumps at http://ol.dataincubator.org/

On another note, since the whole data set is quite large, one of the 
current efforts is to make it possible to identify collections that 
could be downloaded as sets of metadata or harvested using OAI. We might 
be able to find better uses for those collections than for the entire db 
which, as is the case with most large bibliographic databases, has a 
fair amount of chaff for any given definition of wheat.

kc

---

From: Ed Summers <[email-masked]>
Date: Mon, 10 Aug 2009 21:31:03 -0400

Thanks for announcing this exciting news Karen. Are there any plans to
improve the Author data view? For example the bibliographic
description you pointed us at [1] references Lawrence Lessig using:

  http://openlibrary.org/a/OL1518080A

Which is really nice, and honestly enough to say you are doing
linked-data IMHO. But when you resolve that URI you get a chunk of
rdf/xml which isn't valid (no rdf namespace declared). If this could
be fixed it might even nice to deliver up Authors using something like
FOAF if possible.

Also have you considered linking to the book covers in the
bibliographic data view using something like foaf:depiction and/or
foaf:thumbnail? For example:

  <http://openlibrary.org/b/OL6807502M> foaf:depiction
<http://covers.openlibrary.org/b/olid/OL6807502M-L.jpg> .

Thanks again for taking the time to announce this new support.

//Ed

[1] http://openlibrary.org/b/OL6807502M.rdf

---

From: Karen Coyle <[email-masked]>
Date: Tue, 11 Aug 2009 07:40:47 -0700

Ed, I have NO IDEA how you got to rdf/xml from the OL author link -- do 
tell, and I'll take a look! There is no RDF/XML export template for 
authors, but one could be created. The URI/URL is simply the address of 
the author page, and also considered the author identifier on OL.

Thanks for the covers suggestion -- I'll see if that can be added.

kc

---

From: Ed Summers <[email-masked]>
Date: Tue, 11 Aug 2009 11:02:46 -0400

On Tue, Aug 11, 2009 at 10:40 AM, Karen Coyle<[email-masked]> wrote:
> tell, and I'll take a look! There is no RDF/XML export template for authors,
> but one could be created. The URI/URL is simply the address of the author
> page, and also considered the author identifier on OL.

The nice thing about this linked data stuff is all you have to do is
follow your nose:

--

ed@rorty:~$ curl --include --header "Accept: application/rdf+xml"
http://openlibrary.org/a/OL1518080A
HTTP/1.1 200 OK
Content-Type: application/rdf+xml; charset=utf-8
Date: Tue, 11 Aug 2009 15:01:32 GMT
Server: lighttpd/1.4.19
Transfer-Encoding: chunked
Connection: Keep-Alive
Age: 0

<?xml version="1.0" encoding="utf-8"?>
<rdf:RDF
  xmlns:ol='http://openlibrary.org/type/author'

    <ol:name>Lawrence Lessig</ol:name>

    <ol:personal_name>Lawrence Lessig</ol:personal_name>

    <ol:key>/a/OL1518080A</ol:key>

    <ol:type>http://openlibrary.org/type/author.rdf</ol:type>

    <ol:id>5209974</ol:id>

</rdf:RDF>

--

//Ed

---

From: Corey Harper <[email-masked]>
Date: Tue, 11 Aug 2009 11:07:03 -0400

Hi Karen,

Simply adding .rdf to the end of that link returns the RDF/XML Ed 
mentioned, though invalid due to the namespace issue.  Additionally, it 
looks to me like content negotiation is functioning properly against the 
author URIs as well as the bibs, which is fantastic.

I'd agree with Ed's postulation that you're basically already doing 
linked-data.  The fact that the following does what I'd expect is great:

> curl -H "Accept: application/rdf+xml" http://openlibrary.org/a/OL1518080A

Adding to the chorus, thanks for announcing this.

-Corey

-- 
Corey A Harper
Metadata Services Librarian
New York University Libraries
20 Cooper Square, 3rd Floor
New York, NY 10003-7112
212.998.2479
[email-masked]

---

From: "Coombs, Karen A" <[email-masked]>
Date: Tue, 11 Aug 2009 10:12:59 -0500

Karen,

I hate to say it but as someone who is working with the WorldCat Search API
at this point I sure could create code that used Open Library data faster if
you used one of the same metadata formats that the WorldCat Search API uses.
While the two services seem different, (I haven't really figured out how to
effectively search with the Open Library API) there is a learning curve just
working with different metadata formats. It would be good not to have to
teach my staff about another format if I don't have to. That being said, I'd
love to see MODS, that is if it is really good MODS where names are broken
out into their component parts (given, family) and roles are included.

Karen

-- 
Karen A. Coombs
Head of Libraries' Web Services
University of Houston
114 University Libraries
Houston, TX  77204-2000
Phone: (713) 743-3713
Fax: (713) 743-9811
Email: [email-masked]

---

From: Ed Summers <[email-masked]>
Date: Tue, 11 Aug 2009 11:44:29 -0400

> out into their component parts (given, family) and roles are included.

For the sake of providing an alternate perspective, while it would be
nice to see an OpenSearch description for OpenLibrary to complement
the WorldCat Search API I think it would be a mistake to suggest that
OpenLibrary should use SRU and/or MARCXML as an output format just to
provide some commonality w/ OCLC. Perhaps there is more to Karen's
suggestion though.

OpenLibrary is already using the DublinCore vocabulary in its
metadata, just like WorldCat Search API, which seems enough to me. I'm
personally pretty interested to see OpenLibrary taking a more organic
approach to vocabulary selection, mixing and matching vocabulary
elements rather than imposing a particular metadata world-view. I'm
also pleased to see OpenLibrary's approach to thinking about the
resources they are publishing on the web ("A URL For Every Book"), and
providing metadata for those resources in a way that fits in
seamlessly with the web.

//Ed

---

From: Karen Coyle <[email-masked]>
Date: Tue, 11 Aug 2009 09:09:03 -0700

Karen,

I assume that MARCXML is one of those formats? As for MODS, that's 
possible although at the moment names would not be broken down. OL is 
working on altering its storage of names to include separate given and 
family portions, so that will be possible in the future.

Note that an initial MARCXML implementation will not be full MARC, in 
the sense that the OL record doesn't store most of the MARC fixed field 
data. In the case where the original input record was a MARC record, 
more work needs to be done to combine that original record with OL 
enhancements and export as full MARC.

kc

---

From: Karen Coyle <[email-masked]>
Date: Tue, 11 Aug 2009 09:24:10 -0700

OK! thanks. There must be some default operating there... RDF for 
authors is now on to do list! Here are the data elements available:

name
alternate names
website
birth date
death date
wikipedia link

FOAF doesn't cover death dates... RDA has death dates, alternate names. 
Should FOAF be used where possible, adding in RDA to fill in? There are 
a lot of elements they have in common.

kc

---

From: Ross Singer <[email-masked]>
Date: Tue, 11 Aug 2009 13:00:26 -0400

Karen,

The Bio vocabulary might help with the birth/death dates:
http://vocab.org/bio/0.1/.html

And foaf:isPrimaryTopicOf

http://xmlns.com/foaf/spec/#term_isPrimaryTopicOf

might be a good way to relate to the wikipedia page.

I don't have any recommendation for alternate names (and would be
interested in knowing of any, myself).

All this isn't to discourage using the RDA vocabulary for any of this,
but my concern is that its complexity, lack of documentation and
kitchen sink approach will be daunting, especially for people coming
from outside the library domain.

I sort of look at RDA as the "ontology of last resort".

-Ross.

On Tue, Aug 11, 2009 at 12:24 PM, Karen Coyle<[email-masked]> wrote:
> OK! thanks. There must be some default operating there... RDF for authors is
> now on to do list! Here are the data elements available:
>
> name
> alternate names
> website
> birth date
> death date
> wikipedia link
>
> Should FOAF be used where possible, adding in RDA to fill in? There are a
> lot of elements they have in common.
>
> kc
>
> Ed Summers wrote:
>>
>>> authors,

---

From: Karen Coombs <[email-masked]>
Date: Tue, 11 Aug 2009 12:57:59 -0500

Yes, the WorldCat Search API offers MARCXML. It is a bear to work with,
which is why MODS seems like a slightly better route. Smarter than DC but
not has beastly as MARCXML. I actually don't anything from the MARC fixed
fields right now. So the fact that that information is missing isn't a
problem.

In terms of what Ed Summers said, I agree wholeheartedly. OpenLibrary
doesn't need to match what OCLC is doing, but they should be aware of what
is going on there and think about how stuff might dovetail nicely for
developers.

I'll also second his OpenSearch request. Love the ease of using OpenSearch.
My favorite output to work with from WorldCat Search API now are the
Atom/RSS outputs. These standards are so flexible (you can include other
schemas in them) and well known beyond libraryland.

Karen

---

From: Karen Coyle <[email-masked]>
Date: Thu, 13 Aug 2009 12:10:41 -0700

> <http://covers.openlibrary.org/b/olid/OL6807502M-L.jpg> .

Does it work for folks if this returns either a cover OR a blank? (1x1 
jpg). It may be awkward to test first for an actual cover. Also, if it's 
ok to not test for a cover, does anyone have a preference over the blank 
or a 404 error? I think the API can do both.

BTW, basic author RDF is coded and in beta. I'll announce when it goes 
on the public site.

kc

---

From: Ed Summers <[email-masked]>
Date: Thu, 13 Aug 2009 21:39:19 -0400

On Thu, Aug 13, 2009 at 3:10 PM, Karen Coyle<[email-masked]> wrote:
> Does it work for folks if this returns either a cover OR a blank? (1x1 jpg).
> It may be awkward to test first for an actual cover. Also, if it's ok to not
> test for a cover, does anyone have a preference over the blank or a 404
> error? I think the API can do both.

My personal preference would be to include the assertion about the
cover image only if it's true. It might be better to say nothing than
having to live with the false positives. But maybe some other people
feel differently about it.

//Ed

---

From: Keith Jenkins <[email-masked]>
Date: Fri, 14 Aug 2009 09:23:44 -0400

I agree with Ed.  It would be best to omit the statement about the
cover image if it doesn't actually exist.

Keith

---

From: Karen Coyle <[email-masked]>
Date: Fri, 14 Aug 2009 06:50:29 -0700

OK, so we'll only include cover images if we can test for them. - kc

---

From: Gabriel Farrell <[email-masked]>
Date: Sun, 16 Aug 2009 11:21:24 -0400

> providing metadata for those resources in a way that fits in

Agreed on the mixing and matching.  That's what all these wild
vocabularies live for!

Gabriel