Sorry for the word order reversal -- should have been:
After the workshop we ran some no-stemmer/stemmer experiments. The results
were remarkable (I did not test statistical significance): for
Title-Description (shorter queries) average precision went from 0. 259 to
0.334 (29% improvement), for Title-Description-Narrative (longer queries),
average precision went from 0.236 to 0.367 (56% improvement).
Fred
-----Original Message-----
From: Martin Porter [mailto:martin.porter@grapeshot.co.uk]
Sent: Friday, December 10, 2004 2:26 PM
To: gey@berkeley.edu; 'Diana Maynard'
Cc: snowball-discuss@lists.tartarus.org
Subject: RE: [Snowball-discuss] evaluation of Snowball stemmers
Fred,
Do you mean you got a 29%/56% average precision improvement when you
switched stemming off? Anything is possible, but this does surprise me: I
would have expected Russian, with its highly (and regularly) inflected
vocabulary to do quite well under stemming.
If you look at the paper at
http://clef.isti.cnr.it/2004/working_notes/WorkingNotes2004/16.pdf
(Mono- and Crosslingual Retrieval Experiments at the University of
Hildesheim - René Hackl, Thomas Mandl and Christa Womser-Hacker) the
evidence, for Finnish, points the other way ("the snowball stemmer works
very well"). Their Russian experiments were not unfortunately taken to
conclusion, but I feel much more confidence myself in the snowball Russian
stemmer than the snowball Finnish stemmer.
On the other hand I have had verbal notice (which I did not entirely trust!)
of the Finnish stemmer doing badly in some other tests.
I should point out that although the version of the stemmer you picked up
works for KOI-8, Snowball is designed to make switching to other character
codes as easy as possible. See the notes at
http://snowball.tartarus.org/codesets/guide.html
Martin
This archive was generated by hypermail 2.1.3 : Thu Sep 20 2007 - 12:02:46 BST