Retrieval Experiments at Morpho Challenge 2008

Paul McNamee

Johns Hopkins University
Human Language Technology Center of Excellence

Morpho Challenge 2008 hosted an extrinsic evaluation of morphological
analysis that explored whether unsupervised morphology induction could
benefit information retrieval.  This paper presents results in
alternative methods for word normalization using test sets in 13
languages from the Cross-Language Evaluation Forum (CLEF) ad-hoc
evaluations between 2002 and 2007.  Preliminary results for the Morpho
Challenge 2008 evaluation are available in just English, Finnish and
German.  These results appear to be consistent with the larger set of
CLEF experiments we conducted.  We found that: (1) rule-based stemming
is effective in less morphologically complicated languages; (2)
alternative methods for stemming such as unsupervised learning of
morphemes and least common n-gram stemming are helpful; and, (3) full
character n-gram indexing is the most effective form of tokenization
in more morphologically complex languages.

We examined a variety of methods for lexical normalization, including
no transfomation, a rule-based stemmer (Snowball), segments produced
by the Morfessor algorithm, least common n-grams from input words (of
lengths 4 and 5), and regular character n-grams (of lengths 4 and 5).
The most effective technique was character n-gram indexing which
achieved a relative gain of 18% in mean average precision over
unlemmatized words.  In Czech, Bulgarian, Finnish, and Hungarian gains
of over 40% were observed.  While rule-based stemming can be quite
effective, such tools are not available in every language and even
when present, require additional work to integrate with an IR system.
When language-neutral methods are able to achieve the same, or better
performance, their use should be seriously considered.