Subject: [CODE4LIB] Pinyin (Romanized Chinese) searching

From: Bess Sadler <[email-masked]>
Date: Thu, 29 Oct 2009 09:35:30 -0400

By sending this here I hope I'm going to hit everyone on the  
blacklight, vufind, and solrmarc mailing lists, and maybe some other  
interested parties.

Our East Asian Languages Librarian has approached me with a problem he  
wants to see solved. According to him, the typical North American  
library cataloging rules for constructing Pinyin transliterations are  
different from the rules that are used in China. What this means is  
that native Chinese speakers have a lot of trouble searching our  
catalog (it is "practically unusable" was his exact quote). His  
proposal, and I think it's a good one, is that since we're re-indexing  
our records into solr anyway, we could apply at index time an  
algorithm to convert North American Pinyin to Chinese rules Pinyin,  
index both values, and thus make the catalog much more useful to an  
under-served population. This seems like a great suggestion to me, but  
before I start devoting development cycles to it I wanted to poll the  
community... is there a more obvious answer that I'm not seeing? Has  
anyone solved this already?

What's the right place for such a piece of code? Solrmarc seems the  
obvious place to me. As it has been described to me so far, this  
doesn't seem like an issue affecting people outside the library realm,  
which makes it seem too niche and community-specific to get it built  
into the lucene codebase, but I could be wrong about that. Maybe it  
would be better as a lucene contrib library?

So, thoughts? Anyone know more about this than I do and want to speak  
up?

Thanks!

Bess

---

From: Demian Katz <[email-masked]>
Date: Thu, 29 Oct 2009 10:17:07 -0400

> would be better as a lucene contrib library?

I actually think the most logical place for this to be implemented would be at the Solr level, as a filter.  I say this for a two main reasons:

1.) One of the great strengths of Solr is the fact that you have the ability to apply the same text analysis logic to input at index time and at query time.  The SolrMarc approach gives you only one solution: expand each term into multiple entries in the Solr index.  Using a Solr filter allows this approach, but it also adds some other options -- i.e. normalize all the text to a particular format in the index, and then normalize queries to match it using the same algorithm, thus reducing index size.  I know very little about Chinese Romanization, so I don't really know which strategy would make the most sense... but Solr filters will give you more flexibility.

2.) One of the issues the VuFind community will be tackling before too long is dealing with records from formats other than MARC.  Presumably Romanization problems are not limited strictly to MARC.  It seems to make the most sense to address the problem at the index level rather than in a tool tied to a particular input format.

- Demian

---

From: Jonathan Rochkind <[email-masked]>
Date: Thu, 29 Oct 2009 10:56:09 -0400

I wonder if you could do it in Solr itself, instead of SolrMarc?  As an 
input filter (I forget the exact technical language for this, but 
there's something you can set to do this task in Solr); which you don't 
need to get it built into the Solr distro to do, Solr allows you to have 
your own custom plugins like this in the lib directory, and specify in 
them in the schema.xml and such. I think. Just a brainstorming idea, 
don't know enough details about Solr to know if it would work well for 
sure. But if it would, I wonder if putting it configuration in Solr 
itself is preferable to SolrMarc, for wider sharing/utility of code with 
anyone using Solr but not neccesarily using SolrMarc.

Bess Sadler wrote:
> So, thoughts? Anyone know more about this than I do and want to speak up?
>
> Thanks!
>
> Bess

---

From: Till Kinstler <[email-masked]>
Date: Thu, 29 Oct 2009 16:34:15 +0100

> So, thoughts? Anyone know more about this than I do and want to speak up?

I'd second Demian's and Jonathan's statements: Do that in Solr by using 
a Filter (either at indexing or search time).
You want to solve that using an algorithm that translates american 
transcription into chinese, correct? If you have that algorithm (is 
there one?), it's a perfect job for a filter and I guess there are use 
cases outside libraryland as well. It's not only us dealing with 
transcription of chinese...
If I misunderstood your approach and you want to use a dictionary to map 
the different transcriptions, solr.SynonymFilterFactory could provide a 
solution. 
(http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory) 

Till

-- 
Till Kinstler
Verbundzentrale des Gemeinsamen Bibliotheksverbundes (VZG)
Platz der Göttinger Sieben 1, D 37073 Göttingen
[email-masked], +49 (0) 551 39-13431, http://www.gbv.de

---

From: "Ya'aqov Ziso" <[email-masked]>
Date: Thu, 29 Oct 2009 12:21:59 -0400

Till, Bess, Ralph,

>> assuming the algorithm for enriching the spellings of a word (from PinYin to
Chinese) exists, will the result include both forms, PinYin AND Non-PinYin
Chinese transliteration and BOTH forms will be indexed?
>> the principle of indexing different forms for spelling a vertain word exists
in name authority records, where a name (for ex. Pushkin) has over 30 forms of
different spellings. The string of different names can be expended (Ralph¹s work
with VIAF and with fuzzy logic in WorldCat/identities is definitely relevant
here).
maybe something is already underway (?)
>> how large will the resulting index be? managable for medium-small
installations of vuFIND?

Ya¹aqov Ziso, Electronic Resource Management Librarian, Rowan University 856
256 4804