Subject: [CODE4LIB] exact title searches with z39.50

From: Eric Lease Morgan <[email-masked]>
Date: Mon, 27 Apr 2009 17:13:41 -0400

What are the ways to accomplish exact title searches with z39.50?

I'm looping through a list of MARC records trying to determine whether  
or not we own multiple copies of an item. After reading MARC field  
245, subfield a I am creating the following z39.50 query:

   @attr 1=4 "foo bar"

Unfortunately my local implementation seems to interpret this in a  
rather regular expression sort of way -- "* foo bar *". Does anybody  
out there know how to create a more exact query? I only want to find  
titles exactly equalling "foo bar".

-- 
Eric Lease Morgan
University of Notre Dame

---

From: "Binkley, Peter" <[email-masked]>
Date: Mon, 27 Apr 2009 15:24:15 -0600

You could start with the exact title search as expressed in the Bath
Profile: http://www.collectionscanada.gc.ca/bath/tp-bath2.9-e.htm#a .
But you may well have to tinker to discover the combination that your
server will accept and interpret the way you want it to.

All the best,

Peter

> -----Original Message-----
> From: Code for Libraries [mailto:[email-masked]] On 
> Behalf Of Eric Lease Morgan
> Sent: Monday, April 27, 2009 3:14 PM
> To: [email-masked]
> Subject: [CODE4LIB] exact title searches with z39.50
> 
> I'm looping through a list of MARC records trying to 
> determine whether or not we own multiple copies of an item. 
> After reading MARC field 245, subfield a I am creating the 
> following z39.50 query:
> 
>    @attr 1=4 "foo bar"
> 
> Unfortunately my local implementation seems to interpret this 
> in a rather regular expression sort of way -- "* foo bar *". 
> Does anybody out there know how to create a more exact query? 
> I only want to find titles exactly equalling "foo bar".
> 
> --
> Eric Lease Morgan
> University of Notre Dame
> 
>

---

From: Bill Dueber <[email-masked]>
Date: Mon, 27 Apr 2009 19:02:50 -0400

Like so many library "standards," z30.50 is a syntax and a set of rough
guidelines. You have no idea what's actually happening on the other end,
because it's not specified, and you just have to either find someone you can
ask at the target machine or reverse engineer it.

On Mon, Apr 27, 2009 at 5:13 PM, Eric Lease Morgan <[email-masked]> wrote:

> I'm looping through a list of MARC records trying to determine whether or
> not we own multiple copies of an item. After reading MARC field 245,
> subfield a I am creating the following z39.50 query:
>
>  @attr 1=4 "foo bar"
>
> Unfortunately my local implementation seems to interpret this in a rather
> regular expression sort of way -- "* foo bar *". Does anybody out there know
> how to create a more exact query? I only want to find titles exactly
> equalling "foo bar".
>
> --
> Eric Lease Morgan
> University of Notre Dame
>

-- 
Bill Dueber
Library Systems Programmer
University of Michigan Library

---

From: Ross Singer <[email-masked]>
Date: Mon, 27 Apr 2009 23:36:31 -0400

I think in order to accomplish this you'd have to send a completeness
or truncation attribute:

@attr 1=4 6=3 "foo bar" # search for 'foo bar' as the complete field
@attr 1=4 6=2 "foo bar" # search for 'foo bar' as the complete subfield
@attr 1=4 5=100 "foo bar" # do not truncate - although this is
probably not exactly right.

The full list is here:
http://www.loc.gov/z3950/agency/defns/bib1.html

Although I will bet that Aleph (which I assume you're querying)
doesn't support any of this.

I actually just wrote about this exact thing tonight:
http://dilettantes.code4lib.org/2009/04/commoditizing-the-stack/

-Ross.

---

From: Mike Taylor <[email-masked]>
Date: Tue, 28 Apr 2009 09:32:58 +0100

Bill Dueber writes:
 > > I'm looping through a list of MARC records trying to determine
 > > whether or not we own multiple copies of an item. After reading
 > > MARC field 245, subfield a I am creating the following z39.50
 > > query:
 > >
 > >  @attr 1=4 "foo bar"
 > >
 > > Unfortunately my local implementation seems to interpret this in
 > > a rather regular expression sort of way -- "* foo bar *". Does
 > > anybody out there know how to create a more exact query? I only
 > > want to find titles exactly equalling "foo bar".
 >
 > Like so many library "standards," z30.50 is a syntax and a set of
 > rough guidelines. You have no idea what's actually happening on the
 > other end, because it's not specified, and you just have to either
 > find someone you can ask at the target machine or reverse engineer
 > it.

The irony is that Z39.50 actually make _much_ more effort to specify
semantics than most other standards -- and yet still finds itself in
the situation where many implementations do not respond correctly to
the BIB-1 attribute 6=3 (completeness=complete field) which is how
Eric should be able to do what he wants here.

Not that I have any good answers to this problem ... but I DO know
that inventing more and more replacement standards it NOT the answer.
Everything that's come along since Z39.50 has suffered from exactly
the same problem but more so.

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "Not raw -- cooked" -- Monty Python's Flying Circus.

---

From: Eric Lease Morgan <[email-masked]>
Date: Tue, 28 Apr 2009 08:27:08 -0400

> What are the ways to accomplish exact title searches with z39.50?

Thank you for all the prompt and helpful replies. The most precise and  
complete "magic incantation" came from Larry Dixon of the Library of  
Congress:

   Exact match in Z39.50 is accomplished by using additional
   attributes. See the attribute table in the Bath Profile. [1]
   Below is a real-world example -- user is attempting to locate the
   journal "Canadian Poetry" and doesn't know any of the unique
   identifiers associated with it. A keyword search for "Canadian
   poetry" gets 90 hits -- exact match gets 3.

   Z> f @attr 1=4 "canadian poetry"
   Number of hits: 90

   Z> f @attr 1=4 @attr 2=3 @attr 3=1 @attr 4=1 @attr 5=100
        @attr 6=3 "canadian poetry"
   Number of hits: 3

This query takes advantage of many additional attribute types/names as  
alluded to by Tim Shearer: use/title, relation/equal, position/first  
in field, structure/phrase, truncation/do not truncate, and  
completeness/complete field. It would have taken me a long, long time  
to figure this out, and luckily my server supports it.

Wow, isn't the Internet cool, and /me wonders, "Did the Bath Profile  
come from... Bath? [2]"

[1] Bath Profile - http://www.collectionscanada.gc.ca/bath/tp-bath2.9-e.htm#a
[2] Bath, London, and ancient stone circles - http://infomotions.com/gallery/bath/

-- 
Eric Lease Morgan
Hesburgh Libraries, University of Notre Dame

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 28 Apr 2009 10:03:09 -0400

From: "Mike Taylor" <[email-masked]>
> the same problem but more so.

I think this remains to be seen for SRU/CQL, in particular for the example 
at hand, how to search for exact title.  There are two related issues: one, 
how arcane the standard is, and two, how closely implementations conform to 
the intended semantics. And clearly the first has a bearing on the second.

And even I would say that Z39.50 is a bit on the arcance side when it comes 
to formulating a query for exact title. With  SRU/CQL there is an "exact" 
relation ('exact' in 1.1,  '=='  in 1.2).  So I would think there is less 
excuse for a server to apply a creative interpretation. If it cannot support 
"exact title" it should fail the search. With Z39.50 there is more perceived 
latitude for a server to pretend it supports something it doesn't.

--Ray

---

From: David Fiander <[email-masked]>
Date: Tue, 28 Apr 2009 10:11:22 -0400

On Tue, Apr 28, 2009 at 8:27 AM, Eric Lease Morgan <[email-masked]> wrote:
>
> Wow, isn't the Internet cool, and /me wonders, "Did the Bath Profile come
> from... Bath? [2]"
>

Yes.

http://www.collectionscanada.gc.ca/bath/tp-bath2.1-e.htm#c

---

From: Mike Taylor <[email-masked]>
Date: Tue, 28 Apr 2009 15:43:49 +0100

Ray Denenberg, Library of Congress writes:
 > > The irony is that Z39.50 actually make _much_ more effort to
 > > specify semantics than most other standards -- and yet still
 > > finds itself in the situation where many implementations do not
 > > respond correctly to the BIB-1 attribute 6=3
 > > (completeness=complete field) which is how Eric should be able to
 > > do what he wants here.
 > >
 > > Not that I have any good answers to this problem ... but I DO
 > > know that inventing more and more replacement standards it NOT
 > > the answer.  Everything that's come along since Z39.50 has
 > > suffered from exactly the same problem but more so.
 > 
 > I think this remains to be seen for SRU/CQL, in particular for the
 > example at hand, how to search for exact title.  There are two
 > related issues: one, how arcane the standard is, and two, how
 > closely implementations conform to the intended semantics. And
 > clearly the first has a bearing on the second.
 > 
 > And even I would say that Z39.50 is a bit on the arcance side when
 > it comes to formulating a query for exact title. With SRU/CQL there
 > is an "exact" relation ('exact' in 1.1, '==' in 1.2).  So I would
 > think there is less excuse for a server to apply a creative
 > interpretation. If it cannot support "exact title" it should fail
 > the search.

IMHO, this is where it breaks down 90% of the time.  Servers that
can't do what they're asked should say "I can't do that", but -- for
reasons that seem good at the time -- nearly no server fails requests
that it can "sort of" fulfil.  Nine out of ten Z39.50 servers asked to
do a whole-field search and which can't do it will instead do a word
search, because "it's better to give the user SOMETHING".  I bet the
same is true of SRU servers.  (I am as guilty as anyone else, I've
written servers like that.)

The idea that "it's better to give the user SOMETHING" might -- might
-- have been true when we mostly used Z39.50 servers for interactive
sessions.  Now that they are mostly used as targets in metasearching,
that approach is disastrous.

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "I try to take one day at a time, but sometimes several days
	 attack me at once" -- Ashleigh Brilliant.

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 28 Apr 2009 11:32:53 -0400

Right, Mike. There is a long and rich history of the debate between loose 
and strict interpretation, in the world at large, and in particular, within 
Z39.50, this debate raged from the late 1980s throughout the 90s.  The 
faction that said "If you can't give the client what is asks for, at least 
give them something; make them happy" was almost religious in its zeal. 
Those who said "If you can't give the client what it asks for, be honest 
about it; give them good diagnostic information, tell them a better way to 
formulate the request, etc. But don't pretend the transaction was a success 
if it wasn't" was shouted down most every time.   I can't predict, but I'm 
just hoping that lessons have been learned from the mess that that mentality 
got us into.

--Ray

---

From: "Walker, David" <[email-masked]>
Date: Tue, 28 Apr 2009 08:37:50 -0700

I'm not sure it's a _big_ mess, though, at least for metasearching.

I was just looking at our metasearch logs this morning, so did a quick count: 93% of the searches were keyword searches.  Not a lot of exactness required there.  It's mostly in the 7% who are doing more specific searches (author, title, subject) where the bulk if the problems lie, I suspect.

--Dave

==================
David Walker
Library Web Services Manager
California State University
http://xerxes.calstate.edu
________________________________________
From: Code for Libraries [[email-masked]] On Behalf Of Ray Denenberg, Library of Congress [[email-masked]]
Sent: Tuesday, April 28, 2009 8:32 AM
To: [email-masked]
Subject: Re: [CODE4LIB] exact title searches with z39.50

--Ray

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 28 Apr 2009 11:43:17 -0400

It can be a chicken-egg thing too.  Maybe more users would be doing more 
sophisticated searches if they actually _worked_. 

Plus I know that I could write systems to use federated search to embed 
certain functionality in certain places, if more sophisticated searches 
worked more reliably.

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 28 Apr 2009 11:53:11 -0400

From: "Walker, David" <[email-masked]>
> I'm not sure it's a _big_ mess, though, at least for metasearching.

I wasn't thinking  specifically about metasearch, but rather,  bad decisions 
getting replicated and you end up with an installed base of bad 
implementations. The best illustration would be the huge mess that HTML is.

--Ray

---

From: Jonathan Rochkind <[email-masked]>
Date: Tue, 28 Apr 2009 11:57:06 -0400

HTML works out pretty well. If our biggest failures were 'failures' like 
HTML, we'd be doing pretty well.

---

From: Mike Taylor <[email-masked]>
Date: Tue, 28 Apr 2009 17:31:33 +0100

Jonathan Rochkind writes:
 > > I wasn't thinking specifically about metasearch, but rather, bad
 > > decisions getting replicated and you end up with an installed
 > > base of bad implementations. The best illustration would be the
 > > huge mess that HTML is.
 >
 > HTML works out pretty well. If our biggest failures were 'failures'
 > like HTML, we'd be doing pretty well.

Got to agree there (even though it undermines the point I was making
before) -- HTML is not a good example of a system that's undermined
its utility by trying too hard to be helpful.

That Clay Shirky observation again: "You cannot simultaneously have
mass adoption and rigor".  It seems pretty clear that it applies to
something like HTML, where you want to have literally millions of
people writing it.  Not so much in implementing search standards,
where the number of implementers is likely in double figures.

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor    <[email-masked]>    http://www.miketaylor.org.uk
)_v__/\  "Diagnosing: it is OK." -- wonderful diagnostic from _something_
	 in my AUTOEXEC.BAT

---

From: "Ray Denenberg, Library of Congress" <[email-masked]>
Date: Tue, 28 Apr 2009 14:12:13 -0400

From: "Jonathan Rochkind" <[email-masked]>
> HTML, we'd be doing pretty well.

HTML is a wonderful standard.

And I don't mean to take the discussion off-course.   My point was simply 
that because early browsers did not insist on clean html, the proliferation 
of unlean html has reached the point where, well, whether you consider it a 
"mess"  or not depends on how much importance you place on clean html.  It's 
important to me.

--Ray

---

From: Peter Noerr <[email-masked]>
Date: Wed, 29 Apr 2009 17:15:57 +0000

To sidestep the issue of strict/relaxed and face the real world of spotty implementation of standards (and it seems to apply however non/arcane they are) we provide a configurable "strictness" flag and the ability to have non-supported indexes and some functions mapped to supported ones on a Source by Source basis. Admins can allow users to have this strict/relaxed switch or not. And users can apply it or not. For both the majority case is "not" (i.e. relaxed is used).

Peter

Dr Peter Noerr
CTO, MuseGlobal, Inc.

+1 415 896 6873 (office)
+1 415 793 6547 (mobile)
www.museglobal.com

> From: Code for Libraries [mailto:[email-masked]] On Behalf Of
> Jonathan Rochkind
> Sent: Tuesday, April 28, 2009 08:43
> > I was just looking at our metasearch logs this morning, so did a quick
> count: 93% of the searches were keyword searches.  Not a lot of exactness
> required there.  It's mostly in the 7% who are doing more specific searches
> (author, title, subject) where the bulk if the problems lie, I suspect.
> >
> > --Dave
> >
> > ==================
> > David Walker
> > From: Code for Libraries [[email-masked]] On Behalf Of Ray
> Denenberg, Library of Congress [[email-masked]]