normalize.py

No __version__ declared · Source: modules/normalize.py

Unicode normalization, visual and phonetic substitutions, and configurable string matching.

Setup and relationship to toolbox

Use import normalize with this file on the import path. Its text functions use the standard library. If pmblue_update is already importable, the module calls self_update(primary_mod=True) at import time; if it is absent, normalization still loads without bootstrapping it. The current file does not declare __version__.

toolbox.Matching contains a related, newer copy of this functionality. The standalone module does not include toolbox's Matching.Methods.levenshtein_local, and behavior is not identical in every option combination. Import the intended module explicitly.

Examples

import normalize

print(normalize.convert_to_normal("Héllô"))  # hello
print(normalize.convert_to_single_letters("heeellooo"))  # helo
print(normalize.Matching.match("Héllô", "hello"))  # True

settings = normalize.Matching.Settings(ignore_spaces=True)
print(normalize.Matching.match("Main St", "mainst", settings))  # True
print(normalize.Matching.within("Main St", "10 Main Street"))  # True
print(normalize.Matching.fuzzy_match("12345", "12344", threshold=.8))  # True

Normalization functions

Normalization is lossy. Its visual substitutions deliberately merge some letters, digits, and symbols; preserve original input separately when displaying or storing user names.

Matching settings

settings = normalize.Matching.Settings(
    normalize=True,
    extraneous_characters=False,
    ignore_spaces=False,
    match_asterisk=False,
    fuzzy_threshold=1,
    phonetic=False,
)

Matching.match(a, b, settings=None) compares complete transformed strings; Matching.within(needle, text, settings=None) checks containment. Both return booleans. The shared Matching.default_settings is used when no settings object is passed.

1. Apply convert_to_normal when normalization is enabled.
2. Remove ordinary spaces if ignore_spaces=True.
3. Lowercase and apply the fixed substitutions 0→o, @→a, $→s, 1→i, z→2, +→t, even when normalize=False.
4. Collapse consecutive duplicates when extraneous_characters=True; punctuation is otherwise retained unless normalization removes it.
5. Apply phonetic substitutions when enabled.
6. Compare exactly at fuzzy_threshold=1, or use positional fuzzy comparison at another threshold.

Fuzzy and wildcard behavior

Matching.fuzzy_match(a, b, threshold=.8) compares corresponding lowercase characters and returns whether the fraction that match reaches the threshold. It skips normalization. fuzzy_within(needle, text, threshold=.8) applies that comparison to each same-length window in the second string.

An asterisk is a single-position wildcard for exact match(). It is not a glob spanning arbitrary text. In this standalone version, match_asterisk=True with a fuzzy threshold below 1 still calls ordinary fuzzy_match() without wildcard masking. For within(), the wildcard branch builds paired strings with zip(), which truncates the longer input; it does not search all possible wildcard offsets.

Input limits

• Pass strings. Inputs and thresholds are not validated or automatically coerced.
• Low-level fuzzy matching divides by compared length; an empty input can raise ZeroDivisionError.
• Fuzzy and wildcard comparisons stop at the shorter input, so unmatched trailing characters can be ignored.
• These fuzzy functions count matching positions rather than edit distance; an insertion can shift every subsequent character.
• Unsupported Unicode names may disappear, and unusual number-name inputs can trigger errors in the legacy mapping rules.
• Use separate Settings instances instead of mutating the shared default for unrelated callers.

Settings and function contracts

Matching is a namespace: call its functions on the class, for example Matching.match(a, b), rather than constructing Matching(). Matching.Settings is the configurable data object. Its six public attributes have the same names as the constructor arguments, and the matching functions read them when called. Creating a Settings object has no file or network side effects.

match(txt1, txt2, settings=None) interprets both inputs as complete values. within(txt1, txt2, settings=None) interprets the first as the search term and the second as the containing text. fuzzy_match and fuzzy_within accept a numeric threshold, not a Settings object. Every comparison returns a boolean; these APIs do not return a similarity score or a list of positions.

Worked example: find candidates while preserving originals

This builds a candidate list using normalized containment and keeps the original strings for display. The settings object is dedicated to this search.

from normalize import Matching

names = ["Héloise Martin", "Ada Lovelace", "HELOISE DUPONT"]
search_settings = Matching.Settings(normalize=True, ignore_spaces=True)
query = "heloise"

matches = [name for name in names if Matching.within(query, name, search_settings)]
print(matches)  # ['Héloise Martin', 'HELOISE DUPONT']

exact_settings = Matching.Settings(normalize=True, ignore_spaces=False)
print(Matching.match("Héloise", "HELOISE", exact_settings))  # True
print(Matching.match("Héloise", "Héloise Martin", exact_settings))  # False

Worked example: constrain positional fuzzy comparison

Check nonempty equal lengths before calling the low-level fuzzy function when trailing suffixes must count as different values. This guard handles two important limitations without changing the module.

from normalize import Matching

def similar_code(first, second, threshold=0.8):
    if not first or not second or len(first) != len(second):
        return False
    return Matching.fuzzy_match(first, second, threshold=threshold)

print(similar_code("AB123", "AB124"))  # True: 4 of 5 positions match
print(similar_code("AB123", "AB1234"))  # False: lengths differ
print(similar_code("", ""))  # False: no division by zero

masked = Matching.Settings(normalize=False, match_asterisk=True)
print(Matching.match("AB*23", "AB123", masked))  # True

Inspecting a Unicode mapping

Pass the Unicode name rather than the character to name_to_normal. For example, name_to_normal(unicodedata.name("€")) returns "e". The full convert_to_normal function also decomposes composed accents before invoking its custom name rules. A successful mapping is a comparison aid rather than a reversible encoding.