UTS #39: Unicode Security Mechanisms

What it is

UTS #39 is the Unicode Consortium's normative spec for identifier spoofing defenses. The current data files are for Unicode 18.0.0 (https://www.unicode.org/Public/18.0.0/security/). Before Unicode 17.0 they sat in versioned directories under https://www.unicode.org/Public/security/. Implementers can claim conformance per feature (General, IDN and Email profiles, confusables, mixed-script, Restriction Levels, mixed numbers). A change is allowed only if the added or removed characters or mappings are listed precisely.

How it works

Confusability is a string comparison: skeleton(X) = bidiSkeleton(LTR, X), and X and Y are confusable if their skeletons match. The data maps characters to prototypes, and mappings are idempotent, so no recursion is needed. The data was closed under transitivity and substring operations, so the test can over-flag.

Mixed-script detection intersects each character's augmented Script_Extensions. An empty result means mixed-script. "Single-script" means at least one shared script, not exactly one.

Confusables fall into three classes:

• Single-script: for example "ljeto" and "ljeto".
• Mixed-script: for example "paypal" and "pаypаl" with Cyrillic а.
• Whole-script: for example Latin "scope" and Cyrillic "ѕсоре". The whole-script case passes a single-script check, so it needs its own test.

Mixed numbers are detected by computing each Nd digit's zero character (cp minus numeric value). More than one distinct zero means mixed systems. The page gives ICU Java sample code, and Nl and No characters are rejected.

Joiners are handled by limited contexts. Since Unicode 15.0, ZWJ and ZWNJ are Identifier_Status=Restricted. An implementation that keeps them must declare a modified profile and allow them only where rules A1, A2 or B hold. A1 allows ZWNJ breaking a cursive join (/$LJ $T* ZWNJ $RJ/). A2 allows ZWNJ in a conjunct (/$L $M* $V $M₁* ZWNJ $M₁* $L/). B allows ZWJ after a virama, not before a dependent vowel (/$L $M* $V $M₁* ZWJ (?!$D)/). Anything passing these also passes IDNA2008 CONTEXTJ, but not the reverse.

Email profile:

• The domain-part must meet the IDN profile and UTS #46.
• The local-part allows ASCII atext characters. Non-ASCII characters must be Identifier_Status=Allowed, and a leading one must be XID_Start.
• Bidi controls are disallowed in the local-part.
• Display should wrap each part in bidi isolates.

What to do

• Apply the profile as C′ = (C ∩ {Identifier_Status=Allowed}) ∪ Exception. Exception holds syntax characters such as "$", "-" and ".". Keep U+00B7 for Catalan.
• Use LTR, RTL or FS bidiSkeleton where strings cannot be forced single-script, such as programming identifiers. When display direction is unknown, use skeleton (LTR).
• Fast path: with d=LTR and no R or AL characters, bidiSkeleton equals internalSkeleton.
• Judge whole-script confusables against the TLD's script. Latin "circle" is fine in circle.com and Cyrillic "сігсӀе" is fine in сігсӀе.рф, but each is suspect under the other TLD.
• Never display or normalize with skeletons. Treat them like a hashcode.
• Recompute stored skeletons and migrate database indexes whenever the Unicode version changes. Skeletons are not stable across versions and are valid only between systems on the same version.
• Compare Identifier_Status and Identifier_Type case-insensitively, ignoring hyphens and underbars. Since Unicode 13.0 they are written with underbars, which can break naive parsers.
• For registration, treat flagged characters as a soft no with an appeal path. For lookup, treat them as a warning signal.
• IDN operators can express restrictions as RFC 7940 Label Generation Rules.

Caveats

• Identifier_Status has no stability guarantee. Characters can move to Restricted, and implementations can override to keep old behavior.
• The confusables data covers Latin and Cyrillic heavily, so whole-script and mixed-script checks flag many legitimate strings.
• Restricting joiner contexts limits but does not remove confusability. In Tamil most allowed sequences are visually confusable.
• In-script confusability is user-dependent and not fully captured.
• Confusables from before Unicode 7.0 were not idempotent, and the SL, SA and ML tables were removed in 8.0. Version 8.0 or later is strongly advised.
• In Unicode 9.0, xidmodifications.txt was split into IdentifierStatus.txt and IdentifierType.txt.
• Identifier_Type=Aspirational has been empty since Unicode 10.0.
• Email servers' equivalence rules (case or dot sensitivity) cannot be queried.