Character codes

Links

Snowball main page
Snowball header files for: ISO Latin 1 (hex form); ISO Latin 1 (decimal form); MS-DOS Latin 1

The Snowball scripts on this site define the codings of accented letters and other non-ASCII forms in a series of explicit declarations. For example, the German stemmer includes the lines

    /* special characters (in ISO Latin I) */

    stringdef a"   hex 'E4'
    stringdef o"   hex 'F6'
    stringdef u"   hex 'FC'
    stringdef ss   hex 'DF'

In the ISO Latin I code set, hex E4, F6, FC and DF are the numeric values of characters ä, ö, ü and ß respectively. These codings in the stemmer scripts then correspond to the codings used in the sample data.

For a more general approach, you may wish to replace the set of stringdefs by a get directive of the form,

    get 'ISO-Latin-1'

possibly compiling with an -include option that declares the directory where this and other files are held,

    Snowball gstem.sbl -o gstem ... -include /home/shazzer/snowball/codesets

Appropriate code sets for ISO Latin I and MS-DOS Latin I are provided via the links above, and others will be added on demand or if submitted to us.

For Russian, two sets of stringdefs are given in the script — KOI8-R, and (commented out) Unicode. For the other stemmers currently on offer the Unicode placings correspond to the ISO-Latin I placings, so no extra headers for Unicode need, at present, be given.

If you wish to describe other Latin-alphabet based codesets for use in Snowball headers, you should adhere to the following conventions:

accent	ASCII form	example
acute	single quote	`e'`
grave	grave	a`
umlaut	double quote	`u"`
circumflex	circumflex	`i^`
cedilla	comma	`c,`
tilde	tilde	`n~`
ring	letter o	`ao`
line through	solidus	`o/`

And, should they ever arise, use b for breve (as in Rumanian), l and r for left and right hook (as in Polish), q for double acute (as in Hungarian) and v for hacek (as in Czech).

The ‘line-through’ accent covers a numbers of miscellaneous cases: the Scandinavian o/, Icelandic d/ and Polish l/.

Use ae and ss for æ ligature and the German ß, with upper case forms AE and SS. Use th for Icelandic thorn.