Subject: [NGC4LIB] Aggregation of metadata

From: Marja Haapalainen <[email-masked]>
Date: Tue, 16 Feb 2010 08:53:00 +0100

Last year Serials Solutions made quite an impact on the library community with the launch of Summon, to be used as a single point of access for data from various sources. ExLibris, as you know, quickly followed up with a press release about Primo Central. Since then we have also seen similar initiatives from other vendors such as OCLC and EBSCO. We are currently looking at different ways of aggregating metadata (mainly about e-articles, e-journals and e-books) from different sources/publishers. Apart from the commercial vendors, who seem to provide access to their database only through API:s, we are interested in the possibility of storing and indexing the data ourselves. Though this seems to imply some effort in collecting the data, it also gives more flexibility and control over the search services we develop. It would also mean giving libraries more control of how metadata is produced, used and re-used.

We are therefore looking for other, library or consortia driven, initiatives regarding the collection and aggregation of metadata (not only Open Access repositories but also from commercial publishers). We are currently aware of a few such projects, two in Denmark (TING, developed by DBC http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting <http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting> , and the data being used in Summa, developed by the Aarhus State and University Library http://www.statsbiblioteket.dk/summa <http://www.statsbiblioteket.dk/summa>  - sorry for links in Danish!), and one in Sweden: Electronic Library Information Navigator, ELIN (http://elin.lub.lu.se/elinInfo <http://elin.lub.lu.se/elinInfo> ), developed by the Lund University Library. Does anyone know about other such initiatives from around the world? Note that we are not talking about providing search services but about aggregating data. If you are aware of any such initiative, please drop me an email or send it to the list.

Regards

Marja 

Marja Haapalainen

Avdelningen för nationell samverkan/Dept. of National Cooperation

Kungl. biblioteket / National library of Sweden

Email: [email-masked]

---

From: Till Kinstler <[email-masked]>
Date: Tue, 16 Feb 2010 10:57:36 +0100

Marja Haapalainen schrieb:

> We are therefore looking for other, library or consortia driven, initiatives regarding the collection and aggregation of metadata 

In Germany there is the DFG Nationallizenzen funding programme (some 
information in English is on 
http://www.dfg.de/en/research_funding/programmes/infrastructure/lis/digital_information/library_licenses/index.html).
Though the primary goal of this programme is the acquisition of digital 
scientific content, the Nationallizenzen license agreement requires 
publishers to deliver metadata about the purchased content as well.
We (Verbundzentrale des GBV, a German library consortium) collect that 
data, analyze, convert and aggregate it. So far we have about 22 million 
records.
This data is available through some interfaces (SRU, Z39.50, unAPI) and 
as download for entitled users (institutions participating in the DFG 
Nationallizenzen). Some libraries use it in their metasearch portals, 
few actually do something with the data downloads (eg. index it in their 
own search interfaces, like ELIB Bremen: 
http://elib.suub.uni-bremen.de/). And we use it ourselves in the 
Nationallizenzen discovery interface 
(http://finden.nationallizenzen.de/) we just started.

The whole data aggregation workflow really is a pain. Much effort goes 
into analyzation and conversion of all the funny things you get from 
publishers (few deliver bibliographic record formats like MARC21, but 
most provide some homebrewn XMLish formats, Excel sheets, formatted text 
files... worst thing was a >10000 pages Word document containing 
"records" as formatted paragraphs, weird...).

Till

-- 
Till Kinstler
Verbundzentrale des Gemeinsamen Bibliotheksverbundes (VZG)
Platz der Göttinger Sieben 1, D 37073 Göttingen
[email-masked], +49 (0) 551 39-13431, http://www.gbv.de

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 16 Feb 2010 10:38:10 -0500

The Ontario Scholars Portal is or was doing some data aggregation for 
themselves.

Ex Libris has announced but not yet launched a "Primo-Central" data 
aggregation that appears meant to compete with Summon.

Jonathan

Marja Haapalainen wrote:
> We are therefore looking for other, library or consortia driven, initiatives regarding the collection and aggregation of metadata (not only Open Access repositories but also from commercial publishers). We are currently aware of a few such projects, two in Denmark (TING, developed by DBC http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting <http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting> , and the data being used in Summa, developed by the Aarhus State and University Library http://www.statsbiblioteket.dk/summa <http://www.statsbiblioteket.dk/summa>  - sorry for links in Danish!), and one in Sweden: Electronic Library Information Navigator, ELIN (http://elin.lub.lu.se/elinInfo <http://elin.lub.lu.se/elinInfo> ), developed by the Lund University Library. Does anyone know about other such initiatives from around the world? Note that we are not talking about providing search services but about aggregating data. If you are aware of any such initiati!
 ve, please drop me an email or send it to the list.

---

From: Alejandro Garza Gonzalez <[email-masked]>
Date: Tue, 16 Feb 2010 12:37:31 -0600

I'm curious for the metadata aggregation cases as to how the data is 
stored/used:

a) Data imported from external sources into local stores?
b) Data queried realtime at record display time?
c) a combination of both?

_alejandro

Marja Haapalainen said the following on 16/02/2010 01:53 a.m.:
> We are therefore looking for other, library or consortia driven, initiatives regarding the collection and aggregation of metadata (not only Open Access repositories but also from commercial publishers). We are currently aware of a few such projects, two in Denmark (TING, developed by DBC http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting<http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting>  , and the data being used in Summa, developed by the Aarhus State and University Library http://www.statsbiblioteket.dk/summa<http://www.statsbiblioteket.dk/summa>   - sorry for links in Danish!), and one in Sweden: Electronic Library Information Navigator, ELIN (http://elin.lub.lu.se/elinInfo<http://elin.lub.lu.se/elinInfo>  ), developed by the Lund University Library. Does anyone know about other such initiatives from around the world? Note that we are not talking about providing search services but about aggregating data. If you are aware of any such initiative, please drop me an email or send it to the list.
>
>
>
> Regards
>
> Marja
>
>
>
>
>
> Marja Haapalainen
>

-- 
_________________ ___ _ _ _ _ _ _ _
*Ing. Alejandro Garza González*
Coordinación de proyectos y desarrollo de sistemas
Centro Innov@TE, Centro para la Innovación en Tecnología y Educación
Tecnológico de Monterrey

Tel. +52 [81] 8358.2000, Ext. 6751
Enlace intercampus: 80.689.6751, 80.788.6106
http://www.itesm.mx/innovate/

El contenido de este mensaje de datos no se considera oferta, propuesta 
o acuerdo, sino hasta que sea confirmado en documento por escrito que 
contenga la firma autógrafa del apoderado legal del ITESM. El contenido 
de este mensaje de datos es confidencial y se entiende dirigido y para 
uso exclusivo del destinatario, por lo que no podrá distribuirse y/o 
difundirse por ningún medio sin la previa autorización del emisor 
original. Si usted no es el destinatario, se le prohíbe su utilización 
total o parcial para cualquier fin.

The content of this data transmission must not be considered an offer, 
proposal, understanding or agreement unless it is confirmed in a 
document signed by a legal representative of ITESM. The content of this 
data transmission is confidential and is intended to be delivered only 
to the addressees. Therefore, it shall not be distributed and/or 
disclosed through any means without the authorization of the original 
sender. If you are not the addressee, you are forbidden from using it, 
either totally or partially, for any purpose.

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 16 Feb 2010 14:22:15 -0500

To me, the "metadata aggregation" cateogry covers 'a'.  'b' is something 
else. I think that's what Marja meant anyway, all her examples I was 
familiar with fit into 'a'.

Alejandro Garza Gonzalez wrote:
>> We are therefore looking for other, library or consortia driven, initiatives regarding the collection and aggregation of metadata (not only Open Access repositories but also from commercial publishers). We are currently aware of a few such projects, two in Denmark (TING, developed by DBC http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting<http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting>  , and the data being used in Summa, developed by the Aarhus State and University Library http://www.statsbiblioteket.dk/summa<http://www.statsbiblioteket.dk/summa>   - sorry for links in Danish!), and one in Sweden: Electronic Library Information Navigator, ELIN (http://elin.lub.lu.se/elinInfo<http://elin.lub.lu.se/elinInfo>  ), developed by the Lund University Library. Does anyone know about other such initiatives from around the world? Note that we are not talking about providing search services but about aggregating data. If you are aware of any such initiat!
 ive, please drop me an email or send it to the list.

---

From: Dean James <[email-masked]>
Date: Tue, 16 Feb 2010 13:25:44 -0600

Have a look at the Search10 feature offered by my library. Go to http://www.library.tmc.edu. You can search using the feature but will be unable to link to full-text. We built this using Vivisimo's Velocity software.

Dean James, Ph.D., M.S.L.S.
Associate Director, Collection Development
Houston Academy of Medicine-
Texas Medical Center Library
1133 John Freeman Blvd.
Houston, TX  77030
(P) 713-799-7122; (F) 713-799-7180
Email: [email-masked]
http://resource.library.tmc.edu

---

From: Eric Lease Morgan <[email-masked]>
Date: Tue, 16 Feb 2010 14:26:11 -0500

Marja Haapalainen wrote:

> ...We are currently looking at different ways of aggregating
> metadata (mainly about e-articles, e-journals and e-books) from
> different sources/publishers...

Till Kinstler wrote:

> The whole data aggregation workflow really is a pain....

I think the idea of aggregating metadata is/was long time in coming. Kudos. And yes, one of the bigger challenges, besides the licensing/purchasing issues, will be the workflow and data normalization processes.

On the other hand, the benefits can be huge. I see libraries as institutions who do collection, preservation, organization, and dissemination of data, information, and knowledge for specific audiences. By aggregating the metadata -- as opposed to licensing access to it -- libraries can fulfill their goals and provide useful services at the same time.

Actually having possession of the data/metadata opens quite up a number of possibilities. The creation of a unified search interface is just one of them. No information silos. If the data/metadata is all in the same index, then relevancy ranking algorithms and statistical analysis will be much more valid. (Federated searching. Ick.) Once the data is housed locally, it can be more easily integrated into the wider community of the library. It is easier to seamlessly insert it into course management systems. It is easier to create current awareness services. It is easier to distribute. Once the data/metadata is housed locally, then it is easier to apply "digital humanities" computing techniques against it. Create concordances. Extract statistically significant n-grams (one, two, or n-length phrases). There are things called "champion lists" -- sets of words denoting a theme -- that can then be applied a corpus and used to calculate which items are more relevant. With direct acc!
 ess to the data/metadata it is easier to literally chart and graph texts thus illustrating similarities between them. Access to the full text allows you to find similar phrases or paragraphs between texts, and this leads to the ability to trace ideas and authors through time.

Finding information is much less of a problem compared to ten or fifteen years ago. Everybody can find. What is needed now are tools that allow you to USE. This, I believe, is a big opportunity for the profession. Having the data/metadata makes such a thing much more possible.

-- 
Eric Lease Morgan
Head, Digital Access and Information Architecture Department
University of Notre Dame

---

From: "Diane I. Hillmann" <[email-masked]>
Date: Tue, 16 Feb 2010 14:49:42 -0500

On 2/16/10 4:57 AM, Till Kinstler wrote:
> most provide some homebrewn XMLish formats, Excel sheets, formatted
> text files... worst thing was a >10000 pages Word document containing
I think it's pretty common to underestimate the effort, skill and time 
it takes to aggregate disparate batches of metadata into a cohesive 
whole.  The challenges go far beyond the mapping problems (I, too, have 
parsed Word documents into metadata records!), and doing it well really 
requires some fairly careful data management strategies to be cost 
effective over time.  When I was working with the National Science 
Digital Library (NSDL) project some years ago, we spent quite a bit of 
time working on these challenges and published a clutch of papers about 
our work, which may be of help for those of you working in the 
aggregation area.  They can be found on the DCMI papers site 
(http://dcpapers.dublincore.org/ojs/pubs) with a search under author 
name "Hillmann" or on the our consulting site: 
http://managemetadata.org/ under the "about" tab.

Diane Hillmann

---

From: Warwick Cathro <[email-masked]>
Date: Wed, 17 Feb 2010 08:24:53 +1100

Marja,

The National Library of Australia's Trove service relies on aggregating metadata.  It is described here:
https://wiki.nla.gov.au/display/LABS/2.+Trove 
(the last line of that page describes the plans for including e-resources in Trove).

The underlying metadata harvester developed by the NLA has been released as open source:
https://code.nla.gov.au/redmine/projects/show/nlaharvester

Warwick

Warwick Cathro
Assistant Director-General, Resource Sharing and Innovation
National Library of Australia
Ph: 02 6262 1403
Fax: 02 6273 1133
Mob: 0411 868 411

-----Original Message-----
From: Next generation catalogs for libraries [mailto:[email-masked]] On Behalf Of Marja Haapalainen
Sent: Tuesday, 16 February 2010 6:53 PM
To: [email-masked]
Subject: [NGC4LIB] Aggregation of metadata

We are therefore looking for other, library or consortia driven, initiatives regarding the collection and aggregation of metadata (not only Open Access repositories but also from commercial publishers). We are currently aware of a few such projects, two in Denmark (TING, developed by DBC http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting <http://www.dbc.dk/nyheder/openlibrarystrategy/genereltbrondting> , and the data being used in Summa, developed by the Aarhus State and University Library http://www.statsbiblioteket.dk/summa <http://www.statsbiblioteket.dk/summa>  - sorry for links in Danish!), and one in Sweden: Electronic Library Information Navigator, ELIN (http://elin.lub.lu.se/elinInfo <http://elin.lub.lu.se/elinInfo> ), developed by the Lund University Library. Does anyone know about other such initiatives from around the world? Note that we are not talking about providing search services but about aggregating data. If you are aware of any such initiative!
 , please drop me an email or send it to the list.

Regards

Marja 

Marja Haapalainen

---

From: Thomas Krichel <[email-masked]>
Date: Tue, 16 Feb 2010 22:37:13 +0100

Diane I. Hillmann writes

> I think it's pretty common to underestimate the effort, skill and
> time it takes to aggregate disparate batches of metadata into a
> cohesive whole. 

  Tell me about it. ;-(

  FWIW, I make the datasets used to build AuthorClaim available as
  much as I can. Documentation is not perfect, but it's there

http://3lib.org
 
  Bug reports are welcome. 

  Cheers,

  Thomas Krichel                    http://openlib.org/home/krichel
                                http://authorclaim.org/profile/pkr1
                                               skype: thomaskrichel

---

From: Ross Singer <[email-masked]>
Date: Tue, 16 Feb 2010 22:15:51 -0500

On Tue, Feb 16, 2010 at 2:26 PM, Eric Lease Morgan <[email-masked]> wrote:

> Actually having possession of the data/metadata opens quite up a number of possibilities. The creation of a unified search interface is just one of them. No information silos. If the data/metadata is all in the same index, then relevancy ranking algorithms and statistical analysis will be much more valid.

Actually, just to play devil's advocate here:

How would this (in and of itself) eliminate information silos?  On the
surface, it would initially create more.  Summon, after all, is just
another silo, albeit a really big one.

Secondly, doesn't this scenario set up the same problem we're
currently trying to struggle with (every library has the same copy of
the same MARC record, to manage and maintain in a suboptimal system to
managing and maintaining them) except at orders of magnitude greater
scale?

Let me just say that I don't necessarily subscribe to these arguments
I'm making (although I am definitely interested in seeing responses to
them) but at the same time, I'm also not convinced that aggregated
indexes are necessarily the "solution" either (although, yes, in
comparison to federated search, probably).

-Ross.

---

From: Marja Haapalainen <[email-masked]>
Date: Wed, 17 Feb 2010 10:29:20 +0100

Yes it's 'a' data imported from external sources into local stores, that
we mean by metadata aggregation. 
'b' is more federated search, which we probably can not completely live
without because there will always be sources that are not hostable or
maybe too big to host.

marja

---

From: Anders Söderbäck <[email-masked]>
Date: Thu, 18 Feb 2010 10:32:10 +0100

Ross, you make a good devil's advocate and all your arguments makes a lot of sense. Since I am part of the team that together with Marja is investing this issue with regards to the swedish LIBRIS union catalog, I will try my best to respond to them. I definitely think this is a discussion worth having.

It doesn't. Summon is nothing but a big silo. Libraries doing the aggregation themselves would run a very large risk of creating more silos. Since the commercial vendors usually hesitate to openly publish their data (Nature deserves credit for being forward thinking and publishing article level metadata through OAI-PMH) the creation of silos seem hard to avoid, unless we can convince the publishers that metadata shall be open and available to everyone. Open data, I believe, would benefit both libraries and vendors. However, this benefit will only be possible if libraries are willing to do some of the dirty work themselves, not paying the vendors for it.

Summon (or rather any of the current vendor silos, though Summon as of today seem to be the most successful) represents another solution to the problem of non-open data. In the world of silos, one big silo is better than many small silos. And some publishers might be more comfortable with aggregating data through a few big silos. I am afraid, though, that the one big silo world is a step in the wrong direction. One big silo (be it WorldCat, Summon or whatever) means one big obstacle on the road to the world of no silos. It is probably much more difficult to get Summon to publish open data, I would guess their data is bound by several licensing agreements.

Yes. I am afraid this scenario is even worsethan the problem with the current MARC-silos. In todays world, we have z39.50 (we might not like it but it's there and it's being used) and we have a rough consensus among libraries that record sharing is a good thing. When it comes to data aggregation the data seem to be less open (since it is almost always created by someone else than the in house catalogers, and making aggregated data available through OAI-PMH is more visible than making individual records accessible through z39.50). Also, we don't have the rough consensus on how to cooperate on aggregated data.

That being said, I would be very interested in discussing possible ways of cooperation on data aggregation. If the aim is a world of no silos (and I am naive enough to think that this is what we all want) we need to find ways of cooperating around this issue. Which is one of the reasons for Marjas question - if data is being aggregated somewhere outside of Scandinavia, there might also be interesting ideas about cooperation around this issue. If we, as LIBRIS, would do this it would at least be for the benefit of all Swedish academic libraries. Hopefully more...

You are probably correct in that aggregated issus is not the "solution". Or rather, it is not *the* solution, only a solution to the problems of federated search. (But wasn't federated search a few years back being marketed as the solution to the problems of data aggregation?) I would definitely say that actually having possession of the data (as Eric says), actually having it in your own database, gives a lot of possibilites and a lot of flexibility that you don't get when accessing a vendor owned silo through an API. However, getting the data into your own database is, as have been stated by for exampel Till and Diane, painful. Libraries to me seem to be faced with the coice of "pain and flexibility" vs. "less pain and less flexibility". For most libraries, the pain of doing their own aggregation is probably unbearable. For other libraries, it might just be too expensive. However, if we as libraries want flexibility, we need to take this into consideration when signing up !
 for vendor aggregated indexes. What is the future for libraries when the vendors control the means of aggregation (marxist pun intended)?

(Note that I am not complaing about vendors or vendor solutions in general. Not even about vendor aggregated indexes. Though I personally don't like the one big silo scenario, it can be argued that this is the only realistic solution while the no silo solution is too utopic to be useful. I also know that there are vendors working towards the no silo world, and some libraries that seem very keen on keeping their silos. And I can definetely see a future where I myself sign up for a vendor aggregated silo. I just want to know what my options are...

Also, the long term solution I see as regards to both aggregation/federation and cooperation around aggregation is, in three words, Linked Open Data. However, the aggregated silos are here today, LOD might still be the hope of a better tomorrow.)

Best regards
Anders Söderbäck

-----Ursprungligt meddelande-----
Från: Next generation catalogs for libraries [mailto:[email-masked]] För Ross Singer
Skickat: den 17 februari 2010 04:16
Till: [email-masked]
Ämne: Re: [NGC4LIB] Aggregation of metadata

-Ross.

Libraries have to make a choce less pain/less flexibility vs. more pain/more flexibility

---

From: Anders Söderbäck <[email-masked]>
Date: Thu, 18 Feb 2010 10:32:10 +0100

Best regards
Anders Söderbäck

-----Ursprungligt meddelande-----
Från: Next generation catalogs for libraries [mailto:[email-masked]] För Ross Singer
Skickat: den 17 februari 2010 04:16
Till: [email-masked]
Ämne: Re: [NGC4LIB] Aggregation of metadata

-Ross.

---

From: Alejandro Garza Gonzalez <[email-masked]>
Date: Fri, 19 Feb 2010 17:04:56 -0600

I kind of thought that "aggregating metadata" kind of meant enriching an 
existing description (say, adding cover images, tables of contents, 
links to the author's picture, etc. to a basic MARC record) using 
external sources instead of adding more descriptions of more (different) 
things.

This posed (to me) the additional problem of deciding what the best 
source for the same kind of enrichment data comes from (say, you found 
tables of contents in 3 places... which do you want?) as well as 
converting each data to something *usable* =)

_alejandro