Morpho Challenge

The objective of the Morpho Challenge is to design a statistical machine learning algorithm that discovers which morphemes (smallest individually meaningful units of language) words consist of. Ideally, these are basic vocabulary units suitable for different tasks, such as text understanding, machine translation, information retrieval, and statistical language modeling.

Morpho Challenges and workshops

• Morpho Challenge 2010: [data] [results] [workshop]
• Morpho Challenge 2009: [data] [results] [workshop]
• Morpho Challenge 2008: [data] [results] [workshop]
• Morpho Challenge 2007: [data] [results] [workshop]
• Morpho Challenge 2005: [data] [results] [workshop]

Complete results 2007-2010

Information retrieval (IR) and statistical machine translation (SMT) evaluations correspond to those of Challenge 2010. All older submissions have been re-evaluated.

• English
• Finnish
• German
• Turkish

Descriptions of the evaluations and the evaluated algorithms are found in the following article:

• Sami Virpioja, Ville T. Turunen, Sebastian Spiegler, Oskar Kohonen, and Mikko Kurimo. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45-90, 2011. [PDF]

(Click on the title for suggested BibTeX entry.)

Evaluation methods & software

• Morpho Challenge 2009-2010 evaluation (MC)
  • Instructions and software
  • Reference: Mikko Kurimo, Sami Virpioja, Ville T. Turunen, Graeme W. Blackwood, and William Byrne. Overview and results of Morpho Challenge 2009. In Multilingual Information Access Evaluation I. Text Retrieval Experiments, volume 6241 of Lecture Notes in Computer Science, pages 578-597. Springer, 2010.
• EMMA
  • Instructions and software
  • Reference: Sebastian Spiegler and Christian Monson. EMMA: A Novel Evaluation Metric for Morphological Analysis. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING). August 2010.
• EMMA-2
  • Python script, license
  • Reference: Sami Virpioja, Ville T. Turunen, Sebastian Spiegler, Oskar Kohonen, and Mikko Kurimo. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45-90, 2011.
• CoMMA
  • Python script, license
  • Reference: Sami Virpioja, Ville T. Turunen, Sebastian Spiegler, Oskar Kohonen, and Mikko Kurimo. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45-90, 2011.
• Boundary precision and recall (BPR)
  • Python script, license
  • Reference: Sami Virpioja, Ville T. Turunen, Sebastian Spiegler, Oskar Kohonen, and Mikko Kurimo. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45-90, 2011.

The MC evaluation scripts are implemented in Perl. EMMA, EMMA-2, CoMMA, and BPR are implemented in Python. In addition, EMMA and EMMA-2 require the lpsolve library and CoMMA and BPR require the Munkres module for Python.