PlanetSquires Forums

USTRING: Universal String
A dynamic UTF-16 string type for FreeBASIC (Win/Linux, 32/64 bit)

Information and source found at the github repository:  https://github.com/PaulSquires/USTRING

Full compiled FBC compiler toolchains for 32/64 Windows and Linux are included in the repository.
\fbc-win
\fbc-linux

USTRING is true intrinsic type, not a library.

USTRING is not a UDT, a class, a macro, or a header you #include. It is a data type inside the compiler, registered exactly the way ZSTRING and WSTRING are.

It replaces the need to use José's DWSTRING class in the AfxNova library.

The FB 1.20.0 source was directly modified so that USTRING becomes the 4th string data type in FB (along with STRING, ZSTRING, and WSTRING).

USTRING passes all FBC tests. To see the string in action, take a look at the ustring_usage_test.bas file located in the \tests subfolder. It showcases USTRING used every way the FB language allows. Every construct is checked, not just compiled, so it doubles as documentation you can trust.

Covered: every declaration form, dynamic and fixed, arrays (static, dynamic, multi-dimensional, initialised), UDTs, inheritance and virtual methods, properties, operator overloads, every parameter mode (BYVAL / BYREF / BYREF AS CONST, dynamic and fixed), pointers, and every return form (BYVAL, BYREF, fixed-length).

I hope to introduce USTRING to the FB maintainers for inclusion into the core FB compiler. It solves one of the most longstanding omissions in the FB core language - a dynamic unicode enabled string data type.

The core engine
Three things: a descriptor, an allocator, and hand-rolled primitives.
typedef struct _FBUSTRING {
    FB_UCHAR *data;   /* uint16_t, on every target */
    ssize_t   len;    /* CODE UNITS, not bytes */
    ssize_t   size;   /* allocated units, excl. terminator */
} FBUSTRING;
12 bytes on 32-bit, 24 on 64-bit — asserted at compile time against what symb-data.bas hardcodes, because if those disagree every ustring variable is the wrong size on the stack.

The pieces that matter:

• Length is authoritative, the terminator is a courtesy. Every operation uses len, so embedded NULs survive and LEN is O(1). That's also why appending is O(1) amortised while WSTRING's is O(n²) — no walk to find the end.
  
• Size-discriminator ABI. Every entry point takes a (ptr, size) pair: -1 = var-len descriptor, size & FB_STRISFIXED = USTRING * N, 0 = raw units. Same scheme FBSTRING uses, so one function body serves all three forms.
  
• Temp-descriptor pool — 256 slots, identified by pointer range, with the temp flag in len's sign bit. Plus descriptor stealing, which is what keeps a = b + c + d linear instead of allocating per term.
  
• Geometric growth — (units + 15) & ~15 then +12.5% headroom (ustr_core.c:77).
  
• No libc wide functions. All primitives hand-written in ustr_prim.c.
  

ushort vs. extending wstring
WSTRING has two problems, and they're independent:
(1) it can't grow;
(2) its element width varies by platform — 2 bytes on Windows, 4 on Linux, 1 on DOS.

Extending WSTRING to be dynamic fixes (1) and leaves (2) — and (2) is the one that actually corrupts things. You'd end up with a dynamic string that's still UTF-16 on one target and UTF-32 on another: same source, different bytes on disk, different offsets, different LEN. It would also be a breaking change to a type whose sizeof and ABI programs already depend on.

So the element is uint16_t explicitly, never wchar_t — with a compile-time assert enforcing it. That's what makes the representation identical everywhere.

The subtle part is the no-libc rule, and it's the reason for spelling it uint16_t rather than "wchar_t where it happens to be 16 bits". On Windows they are the same width, so wcslen, wcsstr, towlower would link and appear to work perfectly. Using them there and hand-rolled code elsewhere is exactly how two platforms drift apart — which is the failure this type exists to remove. Same reasoning drove using a generated case table instead of towupper, and a fixed UTF-8 conversion instead of the C locale.

Why UTF-16 internally, not UTF-8
Deliberate trade:

• Bought: O(1) LEN and []; free Win32 ...W interop (a pointer reinterpret, no copy); semantics familiar from WSTRING-on-Windows.
• Paid: 2 bytes per character even for ASCII, and a surrogate pair counts as 2 for LEN — so MID can split one, exactly as on a WSTRING.

Note this only concerns the in-memory form. Files are UTF-8 regardless, and STRING ↔ USTRING is UTF-8 on every target — so the UTF-16 choice never leaks into anything portable.


NEXT UP:  Generics
Once Generics become a core FB language feature, it will be easy to have:
(1) Iterator protocol
(2) For Each:   for each x in c
(3) Containers:   Vector, Dictionary, Set, List