Subject: [CODE4LIB] Greenstone: tweaking Lucene indexing

From: Yitzchak Schaffer <[email-masked]>
Date: Tue, 29 Sep 2009 10:01:47 -0400

Hello,

Sorry for any cross-posting annoyance.  I have a request for a 
Greenstone collection I'm working on, to add context snippets to search 
results; for example a search for "yak culture" might return this in the 
list of results:

... addressing the fine points of <strong>yak culture</strong>, the 
zoosociologists took into account ...

Sounds like a pretty basic feature, say our sponsors, and I agree.  (Ah, 
it's also an old Trac ticket at http://trac.greenstone.org/ticket/444)

I see that GS out-of-the-box is set *not* to store the fulltext in the 
index, which seems to be a prerequisite for this kind of thing, as in 
http://bit.ly/ljNkL .  Has anyone modified the Lucene indexing wrapper 
locally to do this?

Given that we don't have any Java coders on staff, I've started porting 
the Lucene wrapper to PHP for use with a custombuilder.pl and 
Zend_Search_Lucene.  I already have a PHP frontend, so adjusting that to 
display the results shouldn't be a problem; OTOH because the frontend is 
PHP, I'm restricted to using buildtype lucene, or something else with 
good PHP support.

Many thanks,

-- 
Yitzchak Schaffer
Systems Manager
Touro College Libraries
33 West 23rd Street
New York, NY 10010
Tel (212) 463-0400 x5230
Fax (212) 627-3197
Email [email-masked]

---

From: Erik Hatcher <[email-masked]>
Date: Tue, 29 Sep 2009 07:21:09 -0700

The Lucene Highlighter doesn't require that the text you want  
highlighted be stored.  In fact, you can pass in any arbitrary text to  
the Highlighter.

See the various getBestFragments from the Highlighter class:
   <http://lucene.apache.org/java/2_4_1/api/org/apache/lucene/search/highlight/Highlighter.html 

	Erik

On Sep 29, 2009, at 7:01 AM, Yitzchak Schaffer wrote:

> Greenstone collection I'm working on, to add context snippets to  
> search results; for example a search for "yak culture" might return  
> this in the list of results:
>
> Sounds like a pretty basic feature, say our sponsors, and I agree.   
> (Ah, it's also an old Trac ticket at http://trac.greenstone.org/ticket/444)
>
> I see that GS out-of-the-box is set *not* to store the fulltext in  
> the index, which seems to be a prerequisite for this kind of thing,  
> as in http://bit.ly/ljNkL .  Has anyone modified the Lucene indexing  
> wrapper locally to do this?
>
> Given that we don't have any Java coders on staff, I've started  
> porting the Lucene wrapper to PHP for use with a custombuilder.pl  
> and Zend_Search_Lucene.  I already have a PHP frontend, so adjusting  
> that to display the results shouldn't be a problem; OTOH because the  
> frontend is PHP, I'm restricted to using buildtype lucene, or  
> something else with good PHP support.
>
> Many thanks,
>
> -- 
> Yitzchak Schaffer
> Systems Manager
> Touro College Libraries
> 33 West 23rd Street
> New York, NY 10010
> Tel (212) 463-0400 x5230
> Fax (212) 627-3197

---

From: Yitzchak Schaffer <[email-masked]>
Date: Tue, 29 Sep 2009 10:33:44 -0400

> highlighted be stored.  In fact, you can pass in any arbitrary text to 

Thanks Erik,

What I'm looking for is to return the context of the search result, not 
just the ID of the containing document - e.g. when all I input is "yak 
culture," I get back the context from the document as a search result, 
without having to retrieve the doc itself:

GS out of the box does not appear to support this, as it does not store 
the fulltext in the index.  So yes, I can highlight stuff, but as it 
stands, I don't have the text to work with.  IANA Lucene guru, so 
correct me if I misunderstand.

---

From: Erik Hatcher <[email-masked]>
Date: Tue, 29 Sep 2009 07:45:58 -0700

On Sep 29, 2009, at 7:33 AM, Yitzchak Schaffer wrote:

>> highlighted be stored.  In fact, you can pass in any arbitrary text  
>> to the Highlighter.
>
> Thanks Erik,
>
> What I'm looking for is to return the context of the search result,  
> not just the ID of the containing document - e.g. when all I input  
> is "yak culture," I get back the context from the document as a  
> search result, without having to retrieve the doc itself:
>
> GS out of the box does not appear to support this, as it does not  
> store the fulltext in the index.  So yes, I can highlight stuff, but  
> as it stands, I don't have the text to work with.  IANA Lucene guru,  
> so correct me if I misunderstand.

I'm a bit confused then.  You mentioned that somehow Zend Lucene was  
going to help, but if you don't have the text to highlight anywhere  
then the Highlighter isn't going to be of any use.  Again, you don't  
need the full text in the Lucene index, but you do need it get it from  
somewhere in order to be able to highlight it.

	Erik

---

From: Yitzchak Schaffer <[email-masked]>
Date: Tue, 29 Sep 2009 11:32:28 -0400

Erik Hatcher wrote:
> going to help, but if you don't have the text to highlight anywhere then 
> the Highlighter isn't going to be of any use.  Again, you don't need the 
> full text in the Lucene index, but you do need it get it from somewhere 
> in order to be able to highlight it.

Erik,

I started to port the native Greenstone Java Lucene wrapper to PHP, so I 
could then modify it to add this feature, as I don't know Java.  This 
would mean using Zend Lucene for the actual indexing implementation.  My 
question is whether anyone's already done it, in Java or otherwise.

Thanks for the clarification,

---

From: Ross Singer <[email-masked]>
Date: Tue, 29 Sep 2009 12:52:37 -0400

Yitzchak, are you interested in actually searching the fulltext?  Or just
highlighting the terms?

If you're only interested in highlighting it, it might be a whole lot easier
to implement this in javascript through something like jQuery:

http://johannburkard.de/blog/programming/javascript/highlight-javascript-text-higlighting-jquery-plugin.html

That way you're not juggling mostly redundant Lucene indexes and trying to
keep them synced.

How are you getting your search results?  Does Greenstone have some sort of
search API that returns the highlighted results?  Would it make a difference
if you could add a field to the Lucene document (meaning would you have
access to it through your PHP API to Greenstone)?  If so, you could probably
do this pretty easily via one of the JVM scripting languages (Groovy, JRuby,
Jython, Quercus -- PHP in the JVM) so you just have the single Lucene index
instead of multiple.

Another approach might be to serve the Lucene index via Solr [1] or
Lucene-WS (http://lucene-ws.net/) which would allow you to skip Greenstone
altogether for searching.

Basically, I would try to avoid going the Zend_Lucene route if at all
possible.

-Ross.

1.
http://www.google.com/search?q=solr+on+an+existing+lucene+index&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a

On Tue, Sep 29, 2009 at 11:32 AM, Yitzchak Schaffer <
[email-masked]> wrote:

> Erik Hatcher wrote:
>
>> I'm a bit confused then.  You mentioned that somehow Zend Lucene was going
>> to help, but if you don't have the text to highlight anywhere then the
>> Highlighter isn't going to be of any use.  Again, you don't need the full
>> text in the Lucene index, but you do need it get it from somewhere in order
>> to be able to highlight it.
>>
>
> Erik,
>
> could then modify it to add this feature, as I don't know Java.  This would
> mean using Zend Lucene for the actual indexing implementation.  My question
> is whether anyone's already done it, in Java or otherwise.
>

---

From: Yitzchak Schaffer <[email-masked]>
Date: Tue, 29 Sep 2009 13:58:08 -0400

> Yitzchak, are you interested in actually searching the fulltext?  Or just

Sorry this wasn't clearer.  Let me re-summarize, and report on a new 
development:

- Greenstone allows for Lucene as one of the indexing plugins

- I took advantage of this for use in our PHP frontend, EmeraldView 
(http://emeraldview.tourolib.org/)

- Greenstone includes a Java wrapper class for Lucene which indexes 
documents as the collection is built

- This wrapper class indexes but does not store the document full text; 
thus a search only returns document IDs of hits.  This means that, in 
order to place search terms in context, we have to load the actual 
documents.  I want the search API itself to return the surrounding text.

New info:

I was in fact able to hack the Java to include the full text in the 
index.  Just a matter of adding a line of code and an "if" statement, 
once I'd been immersed in the code long enough.  Trying to port it to 
PHP (i.e. rewrite it) was instrumental in figuring out why in the world 
the Greenstone indexing code is structured the way it is.

---

From: Yitzchak Schaffer <[email-masked]>
Date: Tue, 29 Sep 2009 14:46:31 -0400

> Yitzchak, are you interested in actually searching the fulltext?  Or just

Just in case my earlier response didn't make it crystal clear: we're 
trying to search the fulltext, and put the search string in context 
within the document which includes it.