This library offers a compact data structure for "generalized"1 UTF-8 encoded codepoints. The design is based on an implicit data structure2, which uses @popCount and bit masking to check membership quickly, with minimal branching, and without having to decode the UTF-8 into another format (for instance, a codepoint). This design is original, in the sense that I invented it. There may be prior art, it's remarkably difficult to search for "UTF-8 character sets" and find papers on set data structures, so I can't say with high confidence that it's truly novel; in a sense, it's an obvious extension of the widespread practice of using a pair of u64 bitmasks to detect a set of ASCII values. What I can say is: it's effective. The RuneSet struct is just a slice of u64, offering a few variations on set membership tests, depending on how confident you are that the string it's testing is valid UTF-8, and what you would like the test to do in the event that it isn't.
Features
- Fast utf8 codepoint sets for Zig
- Provides tools for working with sets of Unicode codepoints, also known as characters or runes
- Sets of UTF-8 Characters
- Documentation available
- The RuneSet is immutable by design
- The data structure is very fast