Paul, could you please test the compilation with UTF-16 BOM encoding?
I'm not sure if it's just my setup, because I've noticed that many of the issues I report don't seem to be encountered by others—which makes me feel like I'm being difficult...
In Tiko, I've been unable to get UTF-16 BOM to compile properly. I can't reliably reproduce the error. Sometimes it compiles successfully, but when I run it again, parts of the code mysteriously go missing, causing compilation failures. In short, it's completely unusable for me.
If these problems are due to my own beginner-level mistakes, I hope you'll bear with me, Paul.
Thank you."
No problem at all. This is one of those areas that needs to be 100% solid. Compiling any type of file format should never result in code corruption. José mentioned in a different post that he has seen code corruption when switching back and forth among different file encodings, eg. ANSI -> UTF-8 -> Unicode
Although you've tested it and it works fine on your end, I'd still like to report my specific situation:
I am using the "Compile & Run" command to execute my programs, not "Quick Run". When my source file is saved in UTF‑16 with BOM format, even very simple code triggers errors. To be more specific:
Dim As Integer a = 100 : Print a — this runs fine, though it disappears immediately.
But when I add Sleep at the end, like this:
Dim As Integer a = 100 : Print a : Sleep
and then run it again, I get the following error:
error 14: Expected identifier, found 'integ'
At this point, the code displayed on the screen is exactly what I wrote above.
However, after I close the file and reopen it, the source code has become something like:
Dim As i
— with the rest of the characters completely missing.
I also tested this in FBIDE. Unlike Tiko, FBIDE only offers UTF‑16LE and UTF‑16BE (no BOM option), and the same code runs perfectly fine there.
Is it possible for you to post a sample file here? Change the file extension to *.txt if a .bas won't attach to a post here. I'd like to test your file directly rather than trying to create one of my own.
Hi Paul,
We'll try to solve these problems one by one, as I observe how Tiko behaves:
Case 1: Ansi file -> codification changed to utf8 (without BOM) by choosing UTF8 in the editor.
When a file is originally saved in ANSI and contains accented characters — for example "José" — those characters are stored using single‑byte extended ASCII codes. In ANSI (Windows‑1252), "é" is stored as byte 0xE9.
Scintilla, however, does not automatically recode the existing buffer when you tell it "now the encoding is UTF‑8". It simply interprets the existing bytes as UTF‑8, even though they are still ANSI bytes.
So the ANSI byte 0xE9 is not valid UTF‑8. Scintilla displays it as the literal escape sequence "xE9", producing JosxE9.
This is why the text becomes corrupted.
When the user changes the encoding from ANSI to UTF‑8, Scintilla cannot magically convert the buffer. Tiko must do the conversion manually:
1. Read the current ANSI buffer from Scintilla.
2. Convert that buffer to UTF‑8 .
3. Clear the Scintilla buffer.
4. Insert the newly converted UTF‑8 text into Scintilla.
5. Tell Scintilla that the buffer is now UTF‑8, so it interprets the bytes correctly.
6. Be aware that UTF‑8 uses more bytes, so the buffer size changes. If you use the same size, some characters at the end of the file will disappear.
Case 2 — ANSI file with accented characters → switch to UTF‑8 with BOM or to UTF-16 with BOM
When the file is ANSI and contains accented characters (e.g., "áéíóúöü"), switching the encoding to UTF‑8 with BOM triggers an internal conversion inside Tiko. This time, unlike the UTF‑8‑without‑BOM case, Tiko actually converts the buffer to UTF‑8 before giving it to Scintilla, and Scintilla displays the text correctly.
However, there is a serious problem:
UTF‑8 uses more bytes than ANSI.
For example:
ANSI "á" = 1 byte (0xE1)
UTF‑8 "á" = 2 bytes (0xC3 0xA1)
So the converted buffer is larger than the original ANSI buffer.
Tiko does not resize the Scintilla buffer before inserting the UTF‑8 text.
As a result:
The converted UTF‑8 text is copied into a buffer that is too small.
The end of the file is truncated.
Several characters disappear.
This is not a Scintilla issue — it is a buffer‑management issue in Tiko.
Reverting back to ANSI makes things even worse
When switching back to ANSI:
The buffer now contains UTF‑8 bytes.
Tiko tries to interpret those UTF‑8 bytes as ANSI.
UTF‑8 sequences like C3 B3 (ó) become ó in ANSI.
Because the buffer was already truncated, the corruption is multiplied.
The result is something like:
áéíóúöüóúöü
This is expected when UTF‑8 bytes are misinterpreted as ANSI.
Case 3 — File originally saved as UTF‑8 or UTF‑16 → switch to ANSI
When the file is originally encoded in UTF‑8 or UTF‑16 and contains accented characters, switching the encoding to ANSI produces the same corruption seen in previous cases.
The result looks like:
áéíóúöü³ÃºÃ¶Ã¼
Why this happens
Tiko does not convert the buffer before changing the encoding.
When switching to ANSI, Tiko should:
* read the UTF‑8/UTF‑16 buffer,
* convert it to ANSI,
* resize the buffer,
* insert the converted text into Scintilla.
But instead, it simply tells Scintilla "this is ANSI now" without changing the bytes.
UTF‑8 or UTF‑16 bytes are interpreted as ANSI.
For example:
UTF‑8 "ó" = C3 B3
ANSI interprets those bytes as "ó"
The buffer is truncated.
UTF‑8 and UTF‑16 require more bytes than ANSI.
Since Tiko does not resize the buffer before inserting the converted text, the end of the file is lost.
Changing the encoding is not the same as converting the text.
If you change the encoding without converting the buffer, Scintilla will reinterpret the bytes incorrectly.
And if you convert the text but don't resize the buffer, you will truncate the file.
Fantastic José, Wow! Really appreciate the time and effort you've put into this investigation. I am testing all your assertions today and am going to modify Tiko's internals to correctly handle the ansi->utf8->unicode encodings. I will post back her with my results.
This bug was very eye opening for me. I had made some fundamental wrong assumptions about how the text conversions worked when applied to the scintilla code buffer.
Tiko's file‑encoding conversion — switching a document between ANSI, UTF‑8, UTF‑8 (BOM), and UTF‑16 (BOM) — had two core defects that corrupted or destroyed text. First, changing the encoding relabelled the document without reliably converting its bytes: choosing "UTF‑8" ran no conversion at all, and a status‑bar refresh (which fires on every caret move) would then flip Scintilla's codepage over the unconverted bytes — turning José into Josxe9 on screen. Second, and worse, every UTF‑16 save wrote only half the file: the code passed a count of characters where the file API expected a count of bytes, so the document was silently truncated mid‑character on disk. Reads compounded this by stopping at the first embedded NUL. The fix routes all conversions through a single, length‑explicit encoder that decodes to UTF‑16 and re‑encodes correctly (BOMs and byte counts included), makes the status bar display‑only, applies the codepage change in the safe order, restores the caret by line/column, and warns before any lossy conversion to ANSI.
I have fixed all three reported scenarios: (1) ANSI → UTF‑8 (no BOM) no longer produces xE9‑style garbage — accented characters convert cleanly; (2) ANSI → UTF‑8 (BOM) / UTF‑16 (BOM) no longer truncates the end of the file, and round‑tripping back to ANSI no longer yields é‑style mojibake; and (3) UTF‑8 / UTF‑16 → ANSI now converts the bytes properly. I have also added a warning popup messagebox when a text conversion contains characters ANSI can't represent (CJK, Cyrillic, emoji). That change is lossy and can not be undone so the user is given the option to cancel proceeding.
I put these changes in a local git where I am working on a large refactor of the Tiko settings dialog and, more notably, a huge refactor of the themeing system (color changes to themes are now interactive and their effects seen in live real time within the Tiko display). The git will be merged to the "development" branch as soon as the features are stable enough for inclusion.