https://sourceforge.net/p/python-ngram/bugs/2009-06-11T14:46:11ZRecent changes to bugs2009-06-11T14:46:11Z2009-06-11T14:46:11ZAnonymoushttps://sourceforge.net/u/userid-None/https://sourceforge.netd20dd275e8c3817e8d56c632caea888d61444f5c

<div class="markdown_content"><p>This module should become Python 3.0 compliant. I patch was submitted to the patches category.</p></div>

2007-11-21T14:43:05Z2007-11-21T14:43:05ZMichel Alberthttps://sourceforge.net/u/exhuma/https://sourceforge.netaf9c4d514cd4ccfeb7dd0f9dc06ce7b0ed6d8972

<div class="markdown_content"><p>If supplying a multibyte string as either "haystack" or "needle", results are unpredictable:</p> <p>&gt;&gt;&gt; ngram.compare('dfsédfsdf', 'dfsédfsdf')<br /> XXd: 'XXd'<br /> Xdf: 'Xdf'<br /> dfs: 'dfs'<br /> fs�: 'fs\xc3'<br /> sé: 's\xc3\xa9'<br /> éd: '\xc3\xa9d'<br /> �df: '\xa9df'<br /> dfs: 'dfs'<br /> fsd: 'fsd'<br /> sdf: 'sdf'<br /> dfX: 'dfX'<br /> fXX: 'fXX'<br /> 1.0</p> <p>Note that the trigrams in the middle are not trigrams at all, but di-grams because the multibyte character is recognized as two characters. Essentially, in this case, as everything get's treated the same, the end-result is correct. However, when supplying the ctrings as unicode objects, it all works as expected:</p> <p>&gt;&gt;&gt; ngram.compare(u'dfsédfsdf', u'dfsédfsdf')<br /> XXd: u'XXd'<br /> Xdf: u'Xdf'<br /> dfs: u'dfs'<br /> fsé: u'fs\xe9'<br /> séd: u's\xe9d'<br /> édf: u'\xe9df'<br /> dfs: u'dfs'<br /> fsd: u'fsd'<br /> sdf: u'sdf'<br /> dfX: u'dfX'<br /> fXX: u'fXX'<br /> 1.0</p> <p>For this reason, I will add a type-check that will only allow unicode objects to be passed down into the module. This may also reveal possible encoding trouble, beacuse the unicode conversion will most likely fail in that case. So the module will not accept data that is obviously wrong.</p></div>

2007-11-21T14:11:16Z2007-11-21T14:11:16ZMichel Alberthttps://sourceforge.net/u/exhuma/https://sourceforge.net41447c70cef250ea5c9ce528cf91ff66a4123e9b

<div class="markdown_content"><p>Internally the algorithm pads the supplied strings with "X" characters. This causes problems when the string itself begins or ends with an X. The similarity score will be smaller than expected as one trigram "disappears".</p> <p>A solution is in the works ;)</p></div>

2006-07-04T20:53:33Z2006-07-04T20:53:33ZAnonymoushttps://sourceforge.net/u/userid-None/https://sourceforge.net76d309fe8caedec3ba558132937aa357064fb7e1

<div class="markdown_content"><p>The source package is empty!</p></div>