Subject: [CODE4LIB] Serials Solutions Summon

From: Yitzchak Schaffer <[email-masked]>
Date: Fri, 17 Apr 2009 12:03:33 -0400

Hello all:

I see that there was an Andrew Nagy-led breakout on Summon at the con. 
Summon is a NGC product with the distinction of using a local copy of 
indexes of licensed content (by agreement with Elsevier, JSTOR, et alia) 
for federated search - rather than the traditional Z39.50 or API calls 
to vendor servers.

Can anyone offer a brief summary of what was discussed?  I am 
particularly interested in the feasibility of obtaining local indexes 
for use in an OSS product.

Best,

-- 
Yitzchak Schaffer
Systems Manager
Touro College Libraries
33 West 23rd Street
New York, NY 10010
Tel (212) 463-0400 x5230
Fax (212) 627-3197
Email [email-masked]
Twitter /torahsyslib

---

From: Andrew Nagy <[email-masked]>
Date: Sat, 18 Apr 2009 14:50:32 -0400

Yitzchak - I'd be more than happy to answer any questions you have about
Summon.  I will give a brief description to answer your questions - but for
any other questions you might have we can discuss offline as to not spam the
mailing list with lots of propaganda for Summon - thought it is really
awesome and everyone should purchase a subscription :)

Summon is really more than an NGC as we are selling it as a service - a
unified discovery service.  This means that it is a single repository of the
library's content ( subscription content, catalog records, IR data, etc.).
Federated search is not apart of Summon ( thought federated search could be
used along side of Summon), all of your library's content is indexed in a
single repository - no need for broadcast searching.  We have an API for
Summon that allows you to access the service with all of the features that
we offer through the Summon User Interface.  This allows you to "plug"
Summon searching into an NGC such as VuFind or Blacklight (I've done the
development for Summon integration in VuFind already).  Our company is also
working on the Summon integration for AquaBrowser.

I'd be more than happy to give a demonstration for your institution on
Summon so you can see it in action and get a better understanding.

Please email me directly for any other questions - or if you would like to
schedule a demonstation for your library.

Cheers
Andrew

On Fri, Apr 17, 2009 at 12:03 PM, Yitzchak Schaffer <[email-masked]>wrote:

> indexes of licensed content (by agreement with Elsevier, JSTOR, et alia) for
> federated search - rather than the traditional Z39.50 or API calls to vendor
> servers.
>
> Can anyone offer a brief summary of what was discussed?  I am particularly
> interested in the feasibility of obtaining local indexes for use in an OSS
> product.
>
> Best,
>
> --
> Yitzchak Schaffer
> Systems Manager
> Touro College Libraries
> 33 West 23rd Street
> New York, NY 10010
> Tel (212) 463-0400 x5230
> Fax (212) 627-3197
> Email [email-masked]
> Twitter /torahsyslib
>

---

From: Yitzchak Schaffer <[email-masked]>
Date: Tue, 21 Apr 2009 10:27:06 -0400

Andrew Nagy wrote:
> Federated search is not apart of Summon

Well, if we understand "federated" to mean "bringing stuff together by 
searching all of it at once," then it is, as opposed to "broadcast 
searching," a term you used later in this sentence.  As in, "Origin:
1665–75; < L foederātus leagued together, allied, equiv. to foeder- 
(nom. s. foedus) league" from
http://dictionary.reference.com/search?q=federated

It is, though, a great *breakthrough* in the area of federated search, 
which is why we ordered an onsite demo immediately on hearing about this 
product.

But I don't think I was clear with my question in any case; it occurs to 
me now that my true question wasn't code-related, but seeing Summon on 
the conf agenda prompted me to bring it up here.  Namely: has anyone 
investigated whether the arrangements SerSol has with content vendors 
are easily duplicable by institutions for home-baked/potential OSS products

-- 
Yitzchak Schaffer
Systems Manager
Touro College Libraries
33 West 23rd Street
New York, NY 10010
Tel (212) 463-0400 x5230
Fax (212) 627-3197
Email [email-masked]
Twitter /torahsyslib

---

From: Mike Taylor <[email-masked]>
Date: Tue, 21 Apr 2009 15:40:39 +0100

I, and most of the people I've worked with, have been using the terms
"metasearch", "federated search", "broadcast search" and "distributed
search" synonymously for years.  Have they now settled down into
having distinct meanings?  If anyone could summarise, I'd be grateful.

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "Sorry if my English didn't good enough to well explain my idea"
	 -- Ronachai Pratanaphon.

Yitzchak Schaffer writes:
 > 1665?75; < L foeder?tus leagued together, allied, equiv. to foeder-

---

From: "Dr R. Sanderson" <[email-masked]>
Date: Tue, 21 Apr 2009 15:45:18 +0100

Sorry, but, "me too!"

Rob

---

From: Eric Lease Morgan <[email-masked]>
Date: Tue, 21 Apr 2009 10:48:55 -0400

> having distinct meanings?  If anyone could summarise, I'd be grateful.

Yes, to me, the quoted phases above are synonymous.

But I believe we are also seeing a new type of index manifesting  
itself, and this new index has yet to be named. Specifically, I'm  
thinking of the index where various types of content is aggregated  
into a single index and then queried. For example, instead of  
providing a federated search against one or more library catalogs, a  
Z39.50 accessible journal article index, a local cache of harvested  
OAI content, etc., I think we are beginning to see all of these  
content silos (and others) brought together into a single (Solr/ 
Lucene) index and searched simultaneously. I'm not sure, but I think  
this is how Summon works.

-- 
Eric Lease Morgan
Head, Digital Access and Information Architecture Department
Hesburgh Libraries, University of Notre Dame

---

From: Cathryn Bowie <[email-masked]>
Date: Tue, 21 Apr 2009 07:49:44 -0700

me 3

Cathryn Bowie
Electronic Services Librarian
State of Oregon Law Library
503-986-5921
[email-masked]

"Dr R. Sanderson" <[email-masked]> 
Sent by: Code for Libraries <[email-masked]>
04/21/2009 07:45 AM
Please respond to
Code for Libraries <[email-masked]>

To
[email-masked]
cc

Subject
Re: [CODE4LIB] Serials Solutions Summon

Sorry, but, "me too!"

Rob

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 10:50:07 -0400

Both the terms "federated searching" and "meta-searching" are often used 
ambiguously to refer to both of these techniques.

I've been trying to use "broadcast search" and "local index" to be clear 
about which technique I'm talking about. (I used to say 'cross-search' 
for 'broadcast search', but I think 'broadcast search' is more clear).

Jonathan

---

From: Karen Schneider <[email-masked]>
Date: Tue, 21 Apr 2009 10:50:32 -0400

> But I don't think I was clear with my question in any case; it occurs to me
> now that my true question wasn't code-related, but seeing Summon on the conf
> agenda prompted me to bring it up here.  Namely: has anyone investigated
> whether the arrangements SerSol has with content vendors are easily
> duplicable by institutions for home-baked/potential OSS products
>

It's a relationship, so anything is possible. But "easily"? That would
depend on who was doing it.

I also agree that "federated search" doesn't suit Summon well. The
natively-indexed content is key.

-- 
-- 
| Karen G. Schneider
| Community Librarian
| Equinox Software Inc. "The Evergreen Experts"
| Toll-free: 1.877.Open.ILS (1.877.673.6457) x712
| [email-masked]
| Web: http://www.esilibrary.com
| Be a part of the Evergreen International Conference, May 20-22, 2009!
| http://www.lyrasis.org/evergreen

---

From: Mike Taylor <[email-masked]>
Date: Tue, 21 Apr 2009 15:55:56 +0100

Eric Lease Morgan writes:
 > Lucene) index and searched simultaneously.

Ah yes.  You won't be surprised to hear that we are doing the same
thing with Zebra.  For this, we use the less than exciting term "local
indexes".

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "I have an ugly tendency to blame all my failings on others.
	 It's something I picked up from my parents" -- Matt Wedel.

---

From: "Dr R. Sanderson" <[email-masked]>
Date: Tue, 21 Apr 2009 15:55:30 +0100

Eric,

How is this 'new type' of index any different from an index of OAI-PMH 
harvested material?  Which in turn is no different from any other local 
search, just a different method of ingesting the data?

Sounds like good PR to me, rather than a revolution ;)

Rob

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 11:02:20 -0400

> depend on who was doing it.

I think in point of fact SerialSolution is doing a LOT of work to keep 
the feeds from the vendors flowing, keep the data updated, deal with 
errors, normalize it to some extent so it can all live in an index 
together.

It's good SerialSolutions has set the precedent, so it may be possible 
for other entities to duplicate those relationships. (I am VERY curious 
about whether SerSol is paying publishers for metadata or getting it for 
free).  But unless you are a large consortium, I think it's going to be 
more cost effective to pay SerSol (or another vendor) to wrangle all 
that metadata, than to try and do it yourself.

It _would_ be great if SerSol would actually give you (if you were 
subscribed) a feed of their harvested and normalized metadata, so you 
could still pay them to collect and normalize it, but then use it for 
your own purposes outside of Summon. I hope SerSol will consider this 
down the line, if Summon is succesful.

Jonathan

---

From: Eric Lease Morgan <[email-masked]>
Date: Tue, 21 Apr 2009 11:08:11 -0400

> search, just a different method of ingesting the data?

This "new type" of index is not any different in functionality from a  
well-implemented OAI service provider with the exception of the type  
of content it contains.

While OAI is/was popular, it was never really embraced by the  
commercial journal article indexers (Cambridge Scientific Abstracts,  
Web of Science, etc.) nor the commercial journal article publishers  
(Springer, Elsevier, etc.). Consequently their scholarly and in high  
demand content was not indexable. I think, but I'm not sure, Serials  
Solutions has made deals with such publishers to aggregate their  
content into a single whole and provide a more unified search  
interface it. We can see similar things in regards to OCLC and WorldCat.

Maybe these "new indexes" could be called "mega indexes" against  
traditional library content. Personally, such a thing is part of what  
I have been advocating as the content of "next generation" library  
catalogs. [1]

[1] http://www.library.nd.edu/daiad/morgan/musings/ngc-in-15-minutes/

-- 
Eric Lease Morgan
University of Notre Dame

---

From: "Frumkin, Jeremy" <[email-masked]>
Date: Tue, 21 Apr 2009 08:10:51 -0700

Well, I'm pretty sure we haven't settled at all on the terminology, but
here's how I use the terms:

Federated Search - I use this to mean a search where the query is sent
against a number of disparate resources; it doesn't matter if they are
local or not.

I don't use broadcast or distributed search as terms at all.

With LibraryFind, we generally describe it as a cross-collection
discovery tool; I like this description because it differentiates
between a broad discovery (i.e. across collections) approach and a
focused discovery (i.e. deeply within a collection) approach, which is a
useful distinction both from a systems side and from a user side.

Ok, I'm sure that now just added more complexity to things. 

-- jaf

==================================
Jeremy Frumkin
Assistant Dean / Chief Technology Strategist
University of Arizona Libraries

[email-masked]
+1 520.307.4548
==================================

-----Original Message-----
From: Code for Libraries [mailto:[email-masked]] On Behalf Of
Mike Taylor
Sent: Tuesday, April 21, 2009 7:41 AM
To: [email-masked]
Subject: Re: [CODE4LIB] Serials Solutions Summon

 _/|_
___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>
http://www.miketaylor.org.uk
)_v__/\  "Sorry if my English didn't good enough to well explain my
idea"
	 -- Ronachai Pratanaphon.

---

From: "Dr R. Sanderson" <[email-masked]>
Date: Tue, 21 Apr 2009 16:14:31 +0100

On Tue, 21 Apr 2009, Eric Lease Morgan wrote:
>> harvested material?  Which in turn is no different from any other
>> local search, just a different method of ingesting the data?

> well-implemented OAI service provider with the exception of the type

Not even the type of content, just the source of the content.
Eg SS have come to an agreement with the publishers to use their 
content, and they've stuffed it all in one big index with a nice 
interface.

NTSH, Move Along...

Rob

---

From: Thomas Dowling <[email-masked]>
Date: Tue, 21 Apr 2009 11:14:52 -0400

> having distinct meanings?  If anyone could summarise, I'd be grateful.

We've occasionally tried to disambiguate those terms for some purposes around
here and realized that, if most people use them synonymously, they're synonyms.
 You can define differences between meta-, federated, and broadcast search, but
every discussion on the topic will be punctuated by people asking, "Wait,
what's the difference again?"

We've started using "unified index" or "local index" for "stuff we get in
advance, load here, and search in place"; and good old "metasearch" for "stuff
scattered in umpteen different locations to which we send searches on demand
and collate results".

-- 
Thomas Dowling
[email-masked]

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 21 Apr 2009 11:28:07 -0400

From: "Thomas Dowling" <[email-masked]>
> what's the difference again?"

Leaving aside metasearch and broadcast search (terms invented more recently) 
it  is a shame if "federated" has really lost its distinction 
from"distributed".  Historically, a federated database is one that 
integrates multiple (autonomous) databases so it is in effect a virtual 
distributed database, though a single database.    I don't think that's a 
hard concept and I don't think it is a trivial distinction.

--Ray

---

From: "Walker, David" <[email-masked]>
Date: Tue, 21 Apr 2009 08:33:09 -0700

Even though Summon is marketed as a Serial Solutions system, I tend to think of it more as coming from Proquest (the parent company, of course).

Summon goes a bit beyond what Proquest and CSA have done in the past, loading outside publisher data, your local catalog records, and some other nice data (no small thing, mind you).  But, like Rob and Mike, I tend to see this as an evolutionary step for a database aggregator like Proquest rather than a revolutionary one.

Obviously, database aggregators like Proquest, OCLC, and Ebsco are well positioned to do this kind of work.  The problem, though, is that they are also competitors.  At some point, if you want to have a truly unified local index of _all_ of your database, you're going to have to cross aggregator lines.  What happens then?

--Dave

==================
David Walker
Library Web Services Manager
California State University
http://xerxes.calstate.edu
________________________________________
From: Code for Libraries [[email-masked]] On Behalf Of Dr R. Sanderson [[email-masked]]
Sent: Tuesday, April 21, 2009 8:14 AM
To: [email-masked]
Subject: Re: [CODE4LIB] Serials Solutions Summon

NTSH, Move Along...

Rob

---

From: Mike Taylor <[email-masked]>
Date: Tue, 21 Apr 2009 16:37:31 +0100

Ray Denenberg, Library of Congress writes:
 > > You can define differences between meta-, federated, and
 > > broadcast search, but every discussion on the topic will be
 > > punctuated by people asking, "Wait, what's the difference again?"
 > 
 > Leaving aside metasearch and broadcast search (terms invented more
 > recently) it is a shame if "federated" has really lost its
 > distinction from"distributed".  Historically, a federated database
 > is one that integrates multiple (autonomous) databases so it is in
 > effect a virtual distributed database, though a single database.  I
 > don't think that's a hard concept and I don't think it is a trivial
 > distinction.

Either it IS a hard concept, or you didn't describe it properly --
because I still don't get exactly what is going on here.  Are you
using "federated" to mean which Thomas and I (independently) are
calling "local indexes"?

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "... about as similar as two completely dissimilar things in a
	 pod" -- Black Adder.

---

From: "Walker, David" <[email-masked]>
Date: Tue, 21 Apr 2009 08:33:21 -0700

I've noticed that reference and instructional librarians (at least in published literature) tend to use the term "federated search" more often than others.  And by that they mean a broadcast search, not what Ray and many others mean by that term.  

Library technology folk tend to use the other terms more often.

--Dave

==================
David Walker
Library Web Services Manager
California State University
http://xerxes.calstate.edu
________________________________________
From: Code for Libraries [[email-masked]] On Behalf Of Ray Denenberg, Library of Congress [[email-masked]]
Sent: Tuesday, April 21, 2009 8:28 AM
To: [email-masked]
Subject: Re: [CODE4LIB] Serials Solutions Summon

--Ray

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 11:54:35 -0400

> Sounds like good PR to me, rather than a revolution ;)
I don't know about "revolution". But your right that this _approach_ is 
similar to an OAI-PMH approach sure. But getting this approach to 
actually work reliably and succesfully with published scholarly articles 
-- is NOT easy.  Most of these publishers and aggregators do not have 
OAI-PMH feeds. (And even if they did, the typical OAI-PMH feed supplying 
only OAI-DC data does NOT provide sufficient structured metadata for a 
search of scholarly content).

Most of them do not share their metadata freely.  You need to have an 
individual relationship with each one, and you need workflow in place on 
your end to make sure you have updated metadata for each one, and you 
need to deal with the inevitable bugs and bad data from the publishers 
and vendors, and you need to do some normalizing so that they can all 
actually live together in an index that works for the user.

This is not an easy thing to do. I believe that some library consortiums 
have long histories of trying to do this (OhioLink?); some have given up 
and no longer try to do it (CDL I think?), because it's very expensive 
and difficult to do right.

If SerialSolutions has succeeded in doing it so it works well, and can 
provide it an affordable cost -- this will, in my opinion be huge.

It's not the basic technology of having a local index that's 
"revolutionary".  It's, potentially, applying that technology to this 
particular case, and actually having it produce something that works 
well for end-users, and being able to do it at an affordable cost. 

But certainly you don't have to be SerialSolutions to do this. I hope 
that the product succeeds, and I hope that it gets 'competitors' 
(library consortial and other vendors) so we have options.

But if it was so easy to do affordably, we'd have it already, right?

Jonathan

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 11:57:22 -0400

> what's the difference again?"

That's why I like "broadcast search" though.  "Broadcast" is very hard 
to use to mean a local index, and once explained once will probably 
"stick".  'Broadcast' inherently means 'going out' to do your search, 
right?  Similarly, "local index".

I avoid using "federated" or "meta" to refer to one of these techniques 
specifically, because they are subject to so much confusion. But since 
these techniques are different in some crucial ways, we DO need ways to 
talk about them. I suggest "broadcast search" and "local index" (or 
"local meta-index"). 

Jonathan

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 11:59:17 -0400

> hard concept and I don't think it is a trivial distinction.

For at least 10 years vendors in the library market have been selling us 
products called "federated search" which are in fact 
distributed/broadcast search products.

If you want to reclaim the term "federated" to mean a local index, I 
think you have a losing battle in front of you.

So I'm sticking with "broadcast search" and "local index".  Sometimes 
you need to use terms invented more recently when the older terms have 
been used ambiguously or contradictorily.  To me, understanding the two 
different techniques and their differences is more important than the 
terminology -- it's just important that the terminology be understood.

---

From: Carl Grant <[email-masked]>
Date: Tue, 21 Apr 2009 11:12:28 -0500

There was some discussion along these lines over on the  
FederatedSearchBlog, which if you didn't see you might want to peruse...

http://federatedsearchblog.com/2009/03/19/beyond-federated-search/

http://federatedsearchblog.com/2009/03/20/beyond-federated-search-the-conversation-continues/

http://federatedsearchblog.com/2009/03/30/beyond-federated-search-%E2%80%93-winning-the-battle-and-losing-the-war/

Carl

Carl Grant
President
Ex Libris North America
1350 E Touhy Avenue, Suite 200 E
Des Plaines, IL 60018
T: 847-227-2615 (Toll Free: 800-762-6300)
F: 847-296-5636
M: 540-449-2418
W: www.exlibrisgroup.com
E: [email-masked]
Skype:     carl_grant

On Apr 21, 2009, at 10:33 AM, Walker, David wrote:

> Even though Summon is marketed as a Serial Solutions system, I tend  
> to think of it more as coming from Proquest (the parent company, of  
> course).
>
> Summon goes a bit beyond what Proquest and CSA have done in the  
> past, loading outside publisher data, your local catalog records,  
> and some other nice data (no small thing, mind you).  But, like Rob  
> and Mike, I tend to see this as an evolutionary step for a database  
> aggregator like Proquest rather than a revolutionary one.
>
> Obviously, database aggregators like Proquest, OCLC, and Ebsco are  
> well positioned to do this kind of work.  The problem, though, is  
> that they are also competitors.  At some point, if you want to have  
> a truly unified local index of _all_ of your database, you're going  
> to have to cross aggregator lines.  What happens then?
>
> --Dave
>
> ==================
> David Walker

---

From: Joe Hourcle <[email-masked]>
Date: Tue, 21 Apr 2009 12:35:39 -0400

On Tue, 21 Apr 2009, Eric Lease Morgan wrote:

> But I believe we are also seeing a new type of index manifesting itself, and 
> this new index has yet to be named. Specifically, I'm thinking of the index 
> where various types of content is aggregated into a single index and then 
> queried.

[trimmed]

Wouldn't this just a "union catalog"?

Now, you might want to differentiate between what's being joined -- 
multiple types of content all at one location vs. the same type of content 
at multiple locations.  (or both varying).  I don't think any of the terms 
mentioned really show that distinction, however.

We normally use 'heterogeneous' in the science to refer to more than one 
type of data object, but it's really superflous in our field, as there are 
very few forms of federated search where you'd have two repositories of 
similar data -- maybe for browse objects (movies, pre-rendered images), 
but those are typically viewed as supplementary metadata, and not the 
object of primary importance.

-----
Joe Hourcle
Principal Software Engineer
Solar Data Analysis Center
Goddard Space Flight Center

---

From: Peter Noerr <[email-masked]>
Date: Tue, 21 Apr 2009 17:01:47 +0000

From one of the Federated Search vendor's perspective... 

It seems in the broader web world we in the library world have lost "metasearch". That has become the province of those systems (mamma, dogpile, etc.) which search the big web search engines (G,Y,M, etc.) primarily for shoppers and travelers (kayak, mobissimo, etc.) and so on. One of the original differences between these engines and the library/information world ones was that they presented results by Source - not combined. This is still evident in a fashion in the travel sites where you can start multiple search sessions on the individual sites.

We use "Federated Search" for what we do in the library/information space. It equates directly to Jonathan's Broadcast Search which was the original term I used when talking about it about 10 years ago. Broadcast is more descriptive, and I prefer it, but it seems an uphill struggle to get it accepted.

Fed Search has the problem of Ray's definition of Federated, to mean "a bunch of things brought together". It can be broadcast search (real time searching of remote Sources and aggregation of a virtual result set), or searching of a local (to the searcher) index which is composed of material federated from multiple Sources at some previous time. We tend to use the term "Aggregate Index" for this (and for the Summon-type index) Mixed content is almost a given, so that is not an issue. And Federated Search systems have to undertake in real time the normalization and other tasks that Summon will be (presumably) putting into its aggregate index.

A problem in terminology we come across is the use of "local" (notice my careful caveat in its use above). It is used to mean local to the searcher (as in the aggregate/meta index above), or it is used to mean local to the original documents (i.e. at the native Source).

I can't imagine this has done more than confirm that there is no agreed terminology - which we sort of all knew. So we just do a lot of explaining - with pictures - to people.

Peter Noerr

Dr Peter Noerr
CTO, MuseGlobal, Inc.

+1 415 896 6873 (office)
+1 415 793 6547 (mobile)
www.museglobal.com

> Jonathan Rochkind
> Sent: Tuesday, April 21, 2009 08:59
> recently)

---

From: "Diane I. Hillmann" <[email-masked]>
Date: Tue, 21 Apr 2009 13:36:30 -0400

Jonathan:

I think you've cut to the chase on this one and seen the potential.  I
went to one of the roll out presentations at Midwinter on Summon, and
was quite impressed.  As someone who *was* an aggregator of metadata in
the recent past (NSDL in the early part of this decade), I can attest to
the fact that making disparate metadata play well together is not an
easy task.  The benefit of doing it the way Summon does it is threefold:

1. They are dealing directly with each provider, and saving the
libraries from having to do this (this is not a small thing, trust me)
2. They recognize the need and the potential for normalizing and
improving the metadata in aggregate (which is generally cheaper, and
vastly improves the behavior and possibilities for the data)
3. Because they also have data on what journals any particular library
customer has subscribed to, they can customize the product for each
library, ensuring that the library's users aren't served a bunch of
results that they ultimately can't access.

This very much the kind of thing that my colleagues at NSDL were trying
to do, but for a lot of reasons (largely political, unfortunately) were
never able to realize.

There will be lots of challenges for them as they move forward with
this, and I for one will be watching closely.  I did speak to Peter
McCracken after the Midwinter presentation, and pointed out that they
might find dealing with personal names a challenge they might want to
take up sooner, rather than later--remember they have both article
metadata and library metadata being presented together, with vastly
different conventions for names ...

Diane

Jonathan Rochkind wrote:
> Dr R. Sanderson wrote:
>> Eric,
>>
>> How is this 'new type' of index any different from an index of 
>> OAI-PMH harvested material?  Which in turn is no different from any 
>> other local search, just a different method of ingesting the data?
>>
> I don't know about "revolution". But your right that this _approach_ 
> is similar to an OAI-PMH approach sure. But getting this approach to 
> actually work reliably and succesfully with published scholarly 
> articles -- is NOT easy.  Most of these publishers and aggregators do 
> not have OAI-PMH feeds. (And even if they did, the typical OAI-PMH 
> feed supplying only OAI-DC data does NOT provide sufficient structured 
> metadata for a search of scholarly content).
>
> individual relationship with each one, and you need workflow in place 
> on your end to make sure you have updated metadata for each one, and 
> you need to deal with the inevitable bugs and bad data from the 
> publishers and vendors, and you need to do some normalizing so that 
> they can all actually live together in an index that works for the user.
>
> This is not an easy thing to do. I believe that some library 
> consortiums have long histories of trying to do this (OhioLink?); some 
> have given up and no longer try to do it (CDL I think?), because it's 
> very expensive and difficult to do right.
>

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 13:56:35 -0400

I think I like your term "aggregated index" even better than "local 
index", thanks Peter. You're right that "local" can be confusing as far 
as "local to WHAT".

So that's my new choice of terminology with the highest chance of being 
understood and least chance of being misconstrued: "broadcast search" 
vs. "aggregated index".

As we've discovered in this thread, if you say "federated search" 
without qualification, different people _will_ have different ideas of 
what you're talking about, as apparently the phrase has been 
historically used differently by different people/communities.

I think "broadcast search" and "aggregated index" are specific enough 
that it would be harder for reasonable people to misconstrue -- and 
don't (yet?) have a history of being used to refer to different things 
by different people. So it's what I'm going to use.

Jonathan

Peter Noerr wrote:
>>> integrates multiple (autonomous) databases so it is in effect a
>>>       
>> virtual
>>     
>>> distributed database, though a single database.    I don't think
>>>       
>> that's a
>>

---

From: Peter Murray <[email-masked]>
Date: Tue, 21 Apr 2009 14:09:21 -0400

On Apr 21, 2009, at 11:02 AM, Jonathan Rochkind wrote:
> subscribed) a feed of their harvested and normalized metadata, so  
> you could still pay them to collect and normalize it, but then use  
> it for your own purposes outside of Summon. I hope SerSol will  
> consider this down the line, if Summon is succesful.

I don't think it is part of SerSol's business model to offer a feed of  
the full metadata it aggregates, but it does seem to be part of the  
business model to offer an API upon which you could put your own  
interface to the underlying aggregated data.

> Wouldn't this just a "union catalog"?

"Catalog" is such a loaded term that I wouldn't want to touch it... ;-)

Peter
- --
Peter Murray                            http://www.pandc.org/peter/work/
Assistant Director, New Service Development  tel:+1-614-728-3600;ext=338
OhioLINK: the Ohio Library and Information Network        Columbus, Ohio
The Disruptive Library Technology Jester                http://dltj.org/
Attrib-Noncomm-Share   http://creativecommons.org/licenses/by-nc-sa/2.5/

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 21 Apr 2009 14:09:19 -0400

From: "Jonathan Rochkind" <[email-masked]>
> If you want to reclaim the term "federated" to mean a local index, I think
> you have a losing battle in front of you.

It's not a battle I plan to pursue, I don't fight battles anymore. I just 
feel obligated to observe that when vocabulary is tinkered with in this 
fashion -- and I did notice, probably more than ten years ago, that 
"federated" was being manipulated --  it  makes it difficult to express 
modeling concepts when definitions are a moving target.  In the future, 
vendors should be more careful about messing around with established 
definitions.  If you don't like the federated model (the old one), don't 
redefine the term, find a new term.  The old term could someday come in 
handy for expressing why you don't like that model, but it's useless if 
nobody agrees what it means.   --Ray

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 21 Apr 2009 14:13:12 -0400

> interface to the underlying aggregated data.

Yep, it's not presently, but I'm hoping that in the future they expand 
to that business model as well.  I think it's feasible.

An API on which you can act on their index is great.  But actually 
having the data to re-index yourself in exactly the way you wanted would 
give you even more power (if you wanted to do the more work to get it).  
And would still be worth paying SerSol for, for the work of aggregating 
and normalizing the data.

Jonathan

---

From: Alan Darnell <[email-masked]>
Date: Tue, 21 Apr 2009 14:25:13 -0400

It is possible for a consortium to build the same sort of service as Serials Solutions.  Besides the OhioLink example, we've been doing that in Ontario for the last 7 years or so - aggregating ejournal content (15 million articles), abstract and index databases (over 100 now in partnership with Proquest), ebooks (about 50,000 commercial ebooks and 170,000 plus digitized ebooks from the Open Content Alliance).  It is a significant effort to deal with all the data feeds but as publishers migrate their production processes to XML we're finding that it is getting a little easier each year.  We aggregate everything in a single XML database from a company called MarkLogic.  The biggest issues we struggle with are currency - it's never as fast as the publisher site though it isn't far behind when things are working well - and quality control - the publisher production processes are shifting to XML but the quality of the data varies.  But hey, it's a library, and these are age-old issues present even in the print world.

Alan

On 4/21/09 2:13 PM, "Jonathan Rochkind" <[email-masked]> wrote:

Jonathan

---

From: Jason Stirnaman <[email-masked]>
Date: Tue, 21 Apr 2009 15:36:08 -0500

Agree. When you step outside libraryland and into corporate/enterprise
IT (thinking Autonomy, FAST, etc.) then "federated search" is often used
to refer to aggregated local indexing of distinct databases.

Jason
-- 

Jason Stirnaman
Digital Projects Librarian/School of Medicine Support
A.R. Dykes Library, University of Kansas Medical Center
[email-masked]
913-588-7319

>>> On 4/21/2009 at 12:56 PM, in message <[email-masked]>,
Jonathan

>> It seems in the broader web world we in the library world have lost

> "metasearch". That has become the province of those systems (mamma,
dogpile, 
> etc.) which search the big web search engines (G,Y,M, etc.) primarily
for 
> shoppers and travelers (kayak, mobissimo, etc.) and so on. One of the

> original differences between these engines and the
library/information world 
> ones was that they presented results by Source - not combined. This
is still 
> evident in a fashion in the travel sites where you can start multiple
search 
> sessions on the individual sites.
>>
>> We use "Federated Search" for what we do in the library/information
space. 
> It equates directly to Jonathan's Broadcast Search which was the
original 
> term I used when talking about it about 10 years ago. Broadcast is
more 
> descriptive, and I prefer it, but it seems an uphill struggle to get
it 
> accepted.
>>
>> Fed Search has the problem of Ray's definition of Federated, to mean
"a 
> bunch of things brought together". It can be broadcast search (real
time 
> searching of remote Sources and aggregation of a virtual result set),
or 
> searching of a local (to the searcher) index which is composed of
material 
> federated from multiple Sources at some previous time. We tend to use
the 
> term "Aggregate Index" for this (and for the Summon-type index) Mixed
content 
> is almost a given, so that is not an issue. And Federated Search
systems have 
> to undertake in real time the normalization and other tasks that
Summon will 
> be (presumably) putting into its aggregate index.
>>
>> A problem in terminology we come across is the use of "local"
(notice my 
> careful caveat in its use above). It is used to mean local to the
searcher 
> (as in the aggregate/meta index above), or it is used to mean local
to the 
> original documents (i.e. at the native Source).
>>
>> I can't imagine this has done more than confirm that there is no
agreed 
> terminology - which we sort of all knew. So we just do a lot of
explaining - 
> with pictures - to people.
>>
>> Peter Noerr
>>
>>
>> Dr Peter Noerr
>> CTO, MuseGlobal, Inc.
>>
>> +1 415 896 6873 (office)
>> +1 415 793 6547 (mobile)
>> www.museglobal.com 
>>
>>
>>
>>
>>   
>>> For at least 10 years vendors in the library market have been
selling
>>> us

---

From: Yitzchak Schaffer <[email-masked]>
Date: Wed, 22 Apr 2009 13:21:24 -0400

> down the line, if Summon is succesful.

This is available as a dump for their traditional holdings product, 
which makes it possible to do just this (i.e. use SerSol cleaned 
holdings/access info in a local system).  My working with SerSol has 
brought me to see them as essentially a great data aggregation service 
with some OK software bundled in.  We are looking ahead to possibly 
using this technique by loading their data into a local ERMS.

Agreed that such an availability of data would be a great service with 
the Summon metadata as well.

-- 
Yitzchak Schaffer
Systems Manager
Touro College Libraries
33 West 23rd Street
New York, NY 10010
Tel (212) 463-0400 x5230
Fax (212) 627-3197
Email [email-masked]
Twitter /torahsyslib

---

From: Andrew Nagy <[email-masked]>
Date: Mon, 4 May 2009 12:15:52 -0400

David - Keep in mind that aggregators are not the original publishers of
content - so even if an aggregator is not yet participating in Summon, the
content in their aggregated databases most often **is** indexed by the
service. To date there are already over 80 individual content providers
participating **in addition to** competing aggregators ProQuest and Gale,
bringing together content from over four thousand publishers.
Regardless of the competitive landscape among aggregators, publishers are
participating in Summon in order to increase discovery of their content.
It's a win-win.

Andrew

On Tue, Apr 21, 2009 at 11:33 AM, Walker, David <[email-masked]>wrote:

> Even though Summon is marketed as a Serial Solutions system, I tend to
> think of it more as coming from Proquest (the parent company, of course).
>
> Summon goes a bit beyond what Proquest and CSA have done in the past,
> loading outside publisher data, your local catalog records, and some other
> nice data (no small thing, mind you).  But, like Rob and Mike, I tend to see
> this as an evolutionary step for a database aggregator like Proquest rather
> than a revolutionary one.
>
> Obviously, database aggregators like Proquest, OCLC, and Ebsco are well
> positioned to do this kind of work.  The problem, though, is that they are
> also competitors.  At some point, if you want to have a truly unified local
> index of _all_ of your database, you're going to have to cross aggregator
> lines.  What happens then?
>
> --Dave
>
> ==================
> David Walker