Recent changes to bugs

Implement Python 3.0 compliance

Anonymous — Thu, 11 Jun 2009 14:46:11 -0000

This module should become Python 3.0 compliant. I patch was submitted to the patches category.

Character encoding trouble

Michel Albert — Wed, 21 Nov 2007 14:43:05 -0000

If supplying a multibyte string as either "haystack" or "needle", results are unpredictable:

>>> ngram.compare('dfsédfsdf', 'dfsédfsdf')
XXd: 'XXd'
Xdf: 'Xdf'
dfs: 'dfs'
fs�: 'fs\xc3'
sé: 's\xc3\xa9'
éd: '\xc3\xa9d'
�df: '\xa9df'
dfs: 'dfs'
fsd: 'fsd'
sdf: 'sdf'
dfX: 'dfX'
fXX: 'fXX'
1.0

Note that the trigrams in the middle are not trigrams at all, but di-grams because the multibyte character is recognized as two characters. Essentially, in this case, as everything get's treated the same, the end-result is correct. However, when supplying the ctrings as unicode objects, it all works as expected:

>>> ngram.compare(u'dfsédfsdf', u'dfsédfsdf')
XXd: u'XXd'
Xdf: u'Xdf'
dfs: u'dfs'
fsé: u'fs\xe9'
séd: u's\xe9d'
édf: u'\xe9df'
dfs: u'dfs'
fsd: u'fsd'
sdf: u'sdf'
dfX: u'dfX'
fXX: u'fXX'
1.0

For this reason, I will add a type-check that will only allow unicode objects to be passed down into the module. This may also reveal possible encoding trouble, beacuse the unicode conversion will most likely fail in that case. So the module will not accept data that is obviously wrong.

Wrong results due to padding with "X"

Michel Albert — Wed, 21 Nov 2007 14:11:16 -0000

Internally the algorithm pads the supplied strings with "X" characters. This causes problems when the string itself begins or ends with an X. The similarity score will be smaller than expected as one trigram "disappears".

A solution is in the works ;)

Sources not available

Anonymous — Tue, 04 Jul 2006 20:53:33 -0000

The source package is empty!