Subject: [CODE4LIB] solr | StopFilterFactory - stopwords.txt

From: Eric James <[email-masked]>
Date: Wed, 11 Nov 2009 15:41:02 -0500

Has anyone already given some thought into refining the solr stopwords.txt for library collections, particularly finding aids? The words included in the out of the box stopwords.txt are of very questionable unimportance:

<an and are as at be but by for if in into is it not of on or s such t that the their then there these they this to was will with>

We were indexing a field id with "no." as one of its tokens (for number), but wanted a query with "no" (where the person did not add the period) to find the doc, but in actuality the "no" would get stripped by the StopFilterFactory. And thus we stumbled upon this list, and was a bit suprised by some of the inclusions (ex:"will"), and exclusions( ex:"a").

Thanks,

Eric James

Yale University Libraries

---

From: Erik Hatcher <[email-masked]>
Date: Thu, 12 Nov 2009 09:06:46 -0500

I often recommend against stop word removal altogether.  Is there any  
reason you need to remove them?

The primary reason stop words get removed is to increase performance  
of queries with very common terms.  If you are encountering that,  
using Solr's CommonGramsFilter(Factory) is a good solution to keep  
your stop words and alleviate the performance degradation potential.   
The HathiTrust folks have had success with the common grams capability.

	Erik

On Nov 11, 2009, at 3:41 PM, Eric James wrote:

> Has anyone already given some thought into refining the solr  
> stopwords.txt for library collections, particularly finding aids?  
> The words included in the out of the box stopwords.txt are of very  
> questionable unimportance:
>
> <an and are as at be but by for if in into is it not of on or s such  
> t that the their then there these they this to was will with>
>
>
>
> We were indexing a field id with "no." as one of its tokens (for  
> number), but wanted a query with "no" (where the person did not add  
> the period) to find the doc, but in actuality the "no" would get  
> stripped by the StopFilterFactory. And thus we stumbled upon this  
> list, and was a bit suprised by some of the inclusions (ex:"will"),  
> and exclusions( ex:"a").
>
>
>
> Thanks,
>
> Eric James
>

---

From: Eric James <[email-masked]>
Date: Thu, 12 Nov 2009 12:33:02 -0500

Thanks, Erik, there is no specific reason for their removal, I think this was just that the StopFilterFactory is preconfigured in the analyzer chain for fieldType=text.  We will do some performance testing with this filter removed.

BTW, a useful tool in deciding appropriate stopwords is the schema browser, which can be found on the /solr/admin page.  Here you can see term frequencies for each of the fields sorted from highest frequency to help weed out the terms of little querying value.

Eric  
 
> Date: Thu, 12 Nov 2009 09:06:46 -0500
> From: [email-masked]
> Subject: Re: [CODE4LIB] solr | StopFilterFactory - stopwords.txt
> To: [email-masked]
>