<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=utf-8">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@Arial Unicode MS";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
code
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-NZ link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#1F497D'>HI Vanessa,<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>You’re correct. Dspace uses
Apache Lucene to index and analyse full text, and one of the things a Lucene
analyser (note: all docs and code refer to the American spelling, analyzer)
does is perform “stemming” on indexed tokens, so common suffixes like “ing”, “ed”,
“es”, “ly” are chopped off the tokens, and off your search terms as well.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>It’s possible to write your own
Lucene analyser or extend your own, and replace the default DSAnalyzer that Dspace
uses with your version. I’ve done this in my installations, not for stemming,
but to properly tokenise macronised vowels (ā ē ī ō ū) that are used in New
Zealand but aren’t in supported ISO character sets.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>This page might help explain the
concepts better than myself: <a
href="http://wiki.apache.org/lucene-java/ConceptsAndDefinitions?highlight=(stemmer)">http://wiki.apache.org/lucene-java/ConceptsAndDefinitions?highlight=(stemmer)</a><o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>This is a quote from the Dspace system
docs:<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p>The Lucene analyzer used in searching and indexing can be configured by
setting the <code><span style='font-size:10.0pt'>search.analyzer</span></code>
configuration item in <code><span style='font-size:10.0pt'>dspace.cfg</span></code>
to the class of the desired analzyer. If this item is not present/commented
out, the default Lucene analyzer <code><span style='font-size:10.0pt'>org.dspace.search.DSAnalyzer</span></code>
is used.<o:p></o:p></p>
<p>As well as those analyzers included in the Lucene distribution (see <code><span
style='font-size:10.0pt'>lucene.jar</span></code>), a Chinese analyzer from the
Lucene sandbox is included in <code><span style='font-size:10.0pt'>lucene-sandbox.jar</span></code>.
This analyzer is yet to be included in the core Lucene distribution but can be
configured by setting <code><span style='font-size:10.0pt'>search.analyzer =
org.apache.lucene.analysis.cn.ChineseAnalyzer</span></code> in <code><span
style='font-size:10.0pt'>dspace.cfg</span></code>.<o:p></o:p></p>
<p class=MsoNormal><span style='color:#1F497D'>Cheers,<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Kim<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> dspace-general-bounces@mit.edu
[mailto:dspace-general-bounces@mit.edu] <b>On Behalf Of </b>Vanessa Barrett<br>
<b>Sent:</b> Friday, 18 September 2009 4:30 p.m.<br>
<b>To:</b> dspace-general@mit.edu<br>
<b>Subject:</b> [Dspace-general] Indexing - automatic mapping of plurals
andalternate endings?<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=EN-AU>Can anyone confirm my understanding of how
DSpace performs keyword indexing/searching? I suspect that it is doing
automatic mapping of singular and plural forms of words.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU>How I came to this understanding was as
follows.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU>I was searching for an item authored by
Alys (alternate spelling of Alice) Clark.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU>I retrieved three items none of which had
the word alys in the metadata or bitstream. If I searched for alys on its
own I got 168 hits and a cursory glance at the results list showed that they
all had an author with some part of their name being ali.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU>I then tried searching for each of the
following forms</span><span lang=EN-AU style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-family:"Arial","sans-serif"'>aly,
ali, alis, alys alies <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU>All of these as single search terms
retrieved exactly the same number of records – 168. Results included
items with the following strings in Abstract<o:p></o:p></span></p>
<p class=MsoNormal><u><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black;background:yellow;mso-highlight:yellow'>- ALIS</span></u><span
lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'> (Advanced Landmine Imaging System), which is a novel landmine
detection sensor system<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>- Current ventilatory practices for the management of <u><span
style='background:yellow;mso-highlight:yellow'>ALI</span></u> favor low tidal
volumes<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>- Current Trends in Periodontal Diagnosis & Disease
Recognition in Malaysia / T.B. Taiyeb <u><span style='background:yellow;
mso-highlight:yellow'>Ali</span></u><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>- Radiology in the acute abdomen / P.G. Devitt, A. <u><span
style='background:yellow;mso-highlight:yellow'>Aly</span></u>, M. Thomas<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>My conclusion is that DSpace is doing some process of mapping
plural to singular forms of words including allowing for alternate endings.
If it is doing this it is very clever but just a little annoying as Alys is not
the plural of Ali.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>Also if clever enough to do this why can</span><span lang=ZH-CN
style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";color:black'>’</span><span
lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>t it map fiber to fibre and color to colour which would have much
greater benefits in searching a database that includes North American and
European data.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-AU style='font-size:9.5pt;font-family:"Arial Unicode MS","sans-serif";
color:black'>Cheers, </span><span lang=EN-AU><o:p></o:p></span></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
line-height:12.0pt'><span lang=EN-AU style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Vanessa
Barrett<br>
Digital Services Librarian<br>
The University of Adelaide, AUSTRALIA 5005<br>
Ph : +61 8 8303 4625<br>
e-mail: vanessa.barrett@adelaide.edu.au<br>
<br>
</span><span lang=EN-AU style='font-size:9.0pt;font-family:"Verdana","sans-serif"'>CRICOS
Provider Number 00123M<br>
-----------------------------------------------------------<br>
IMPORTANT: This message may contain confidential or legally privileged
information. If you think it was sent to you by mistake, please delete all
copies and advise the sender. For the purposes of the SPAM Act 2003, this email
is authorised by The University of Adelaide. <o:p></o:p></span></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
line-height:12.0pt'><span lang=EN-AU style='font-size:9.0pt;font-family:"Verdana","sans-serif"'>Think
green: read on the screen.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-AU><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>