- Welcome to PlanetSquires Forums.
PlanetSquires Forums
"It’s not a bug; it’s an undocumented feature" - Anonymous
Recent posts
#71
PlanetSquires Software / Re: Regarding UTF-16 encoding ...
Last post by José Roca - July 25, 2026, 09:29:50 PMCase 3 — File originally saved as UTF‑8 or UTF‑16 → switch to ANSI
When the file is originally encoded in UTF‑8 or UTF‑16 and contains accented characters, switching the encoding to ANSI produces the same corruption seen in previous cases.
The result looks like:
áéíóúöü³ÃºÃ¶Ã¼
Why this happens
Tiko does not convert the buffer before changing the encoding.
When switching to ANSI, Tiko should:
* read the UTF‑8/UTF‑16 buffer,
* convert it to ANSI,
* resize the buffer,
* insert the converted text into Scintilla.
But instead, it simply tells Scintilla "this is ANSI now" without changing the bytes.
UTF‑8 or UTF‑16 bytes are interpreted as ANSI.
For example:
UTF‑8 "ó" = C3 B3
ANSI interprets those bytes as "ó"
The buffer is truncated.
UTF‑8 and UTF‑16 require more bytes than ANSI.
Since Tiko does not resize the buffer before inserting the converted text, the end of the file is lost.
When the file is originally encoded in UTF‑8 or UTF‑16 and contains accented characters, switching the encoding to ANSI produces the same corruption seen in previous cases.
The result looks like:
áéíóúöü³ÃºÃ¶Ã¼
Why this happens
Tiko does not convert the buffer before changing the encoding.
When switching to ANSI, Tiko should:
* read the UTF‑8/UTF‑16 buffer,
* convert it to ANSI,
* resize the buffer,
* insert the converted text into Scintilla.
But instead, it simply tells Scintilla "this is ANSI now" without changing the bytes.
UTF‑8 or UTF‑16 bytes are interpreted as ANSI.
For example:
UTF‑8 "ó" = C3 B3
ANSI interprets those bytes as "ó"
The buffer is truncated.
UTF‑8 and UTF‑16 require more bytes than ANSI.
Since Tiko does not resize the buffer before inserting the converted text, the end of the file is lost.
#72
PlanetSquires Software / Re: Regarding UTF-16 encoding ...
Last post by José Roca - July 25, 2026, 08:58:20 PMCase 2 — ANSI file with accented characters → switch to UTF‑8 with BOM or to UTF-16 with BOM
When the file is ANSI and contains accented characters (e.g., "áéíóúöü"), switching the encoding to UTF‑8 with BOM triggers an internal conversion inside Tiko. This time, unlike the UTF‑8‑without‑BOM case, Tiko actually converts the buffer to UTF‑8 before giving it to Scintilla, and Scintilla displays the text correctly.
However, there is a serious problem:
UTF‑8 uses more bytes than ANSI.
For example:
ANSI "á" = 1 byte (0xE1)
UTF‑8 "á" = 2 bytes (0xC3 0xA1)
So the converted buffer is larger than the original ANSI buffer.
Tiko does not resize the Scintilla buffer before inserting the UTF‑8 text.
As a result:
The converted UTF‑8 text is copied into a buffer that is too small.
The end of the file is truncated.
Several characters disappear.
This is not a Scintilla issue — it is a buffer‑management issue in Tiko.
Reverting back to ANSI makes things even worse
When switching back to ANSI:
The buffer now contains UTF‑8 bytes.
Tiko tries to interpret those UTF‑8 bytes as ANSI.
UTF‑8 sequences like C3 B3 (ó) become ó in ANSI.
Because the buffer was already truncated, the corruption is multiplied.
The result is something like:
áéíóúöüóúöü
This is expected when UTF‑8 bytes are misinterpreted as ANSI.
When the file is ANSI and contains accented characters (e.g., "áéíóúöü"), switching the encoding to UTF‑8 with BOM triggers an internal conversion inside Tiko. This time, unlike the UTF‑8‑without‑BOM case, Tiko actually converts the buffer to UTF‑8 before giving it to Scintilla, and Scintilla displays the text correctly.
However, there is a serious problem:
UTF‑8 uses more bytes than ANSI.
For example:
ANSI "á" = 1 byte (0xE1)
UTF‑8 "á" = 2 bytes (0xC3 0xA1)
So the converted buffer is larger than the original ANSI buffer.
Tiko does not resize the Scintilla buffer before inserting the UTF‑8 text.
As a result:
The converted UTF‑8 text is copied into a buffer that is too small.
The end of the file is truncated.
Several characters disappear.
This is not a Scintilla issue — it is a buffer‑management issue in Tiko.
Reverting back to ANSI makes things even worse
When switching back to ANSI:
The buffer now contains UTF‑8 bytes.
Tiko tries to interpret those UTF‑8 bytes as ANSI.
UTF‑8 sequences like C3 B3 (ó) become ó in ANSI.
Because the buffer was already truncated, the corruption is multiplied.
The result is something like:
áéíóúöüóúöü
This is expected when UTF‑8 bytes are misinterpreted as ANSI.
#73
PlanetSquires Software / Re: Regarding UTF-16 encoding ...
Last post by José Roca - July 25, 2026, 06:19:01 PMHi Paul,
We'll try to solve these problems one by one, as I observe how Tiko behaves:
Case 1: Ansi file -> codification changed to utf8 (without BOM) by choosing UTF8 in the editor.
When a file is originally saved in ANSI and contains accented characters — for example "José" — those characters are stored using single‑byte extended ASCII codes. In ANSI (Windows‑1252), "é" is stored as byte 0xE9.
Scintilla, however, does not automatically recode the existing buffer when you tell it "now the encoding is UTF‑8". It simply interprets the existing bytes as UTF‑8, even though they are still ANSI bytes.
So the ANSI byte 0xE9 is not valid UTF‑8. Scintilla displays it as the literal escape sequence "xE9", producing JosxE9.
This is why the text becomes corrupted.
When the user changes the encoding from ANSI to UTF‑8, Scintilla cannot magically convert the buffer. Tiko must do the conversion manually:
1. Read the current ANSI buffer from Scintilla.
2. Convert that buffer to UTF‑8 .
3. Clear the Scintilla buffer.
4. Insert the newly converted UTF‑8 text into Scintilla.
5. Tell Scintilla that the buffer is now UTF‑8, so it interprets the bytes correctly.
6. Be aware that UTF‑8 uses more bytes, so the buffer size changes. If you use the same size, some characters at the end of the file will disappear.
We'll try to solve these problems one by one, as I observe how Tiko behaves:
Case 1: Ansi file -> codification changed to utf8 (without BOM) by choosing UTF8 in the editor.
When a file is originally saved in ANSI and contains accented characters — for example "José" — those characters are stored using single‑byte extended ASCII codes. In ANSI (Windows‑1252), "é" is stored as byte 0xE9.
Scintilla, however, does not automatically recode the existing buffer when you tell it "now the encoding is UTF‑8". It simply interprets the existing bytes as UTF‑8, even though they are still ANSI bytes.
So the ANSI byte 0xE9 is not valid UTF‑8. Scintilla displays it as the literal escape sequence "xE9", producing JosxE9.
This is why the text becomes corrupted.
When the user changes the encoding from ANSI to UTF‑8, Scintilla cannot magically convert the buffer. Tiko must do the conversion manually:
1. Read the current ANSI buffer from Scintilla.
2. Convert that buffer to UTF‑8 .
3. Clear the Scintilla buffer.
4. Insert the newly converted UTF‑8 text into Scintilla.
5. Tell Scintilla that the buffer is now UTF‑8, so it interprets the bytes correctly.
6. Be aware that UTF‑8 uses more bytes, so the buffer size changes. If you use the same size, some characters at the end of the file will disappear.
#74
PlanetSquires Software / Re: AfxNova / FreeBasic / Plan...
Last post by hajubu - July 25, 2026, 01:52:49 PMhi Paul ,
Congratulation that your joy is back again.
Thanks for the long awaited HelpCenter and the new Base Website
www.planetsquires.com (base)
www.planetsquires/docs docs)
That will surely help to boost the Tiko and the modernized Freebasic.
Thanks again to you and José.
b.r. Hajubu -Hans
P.S.:
I just played a little with a simple implementation in Tiko .
---
# Local HelpCenter
The Help Center is a static site with
a 2.4 MB search index and no server behind it,
which is why it also works offline from
a local folder and inside tiko's own F1 pane.
My test implementation for the moment is:
a) using wget
b) clean (docs/*js) and (assets/*css,*js) files from @v=(sha1-id)
b) using Tiko User Tools
=> Alt-1 => opens local => HelpCenter (same as online)
HelpCenter Alt+1
ASCII Chart Ctrl+1
GUID Generator Ctrl+2
```
structural scheme : Index in D:\Users\docs\helpcenter\docs\
...screenshots: (temporarily 30d)
Congratulation that your joy is back again.
Thanks for the long awaited HelpCenter and the new Base Website
www.planetsquires.com (base)
www.planetsquires/docs docs)
That will surely help to boost the Tiko and the modernized Freebasic.
Thanks again to you and José.
b.r. Hajubu -Hans
P.S.:
I just played a little with a simple implementation in Tiko .
---
# Local HelpCenter
The Help Center is a static site with
a 2.4 MB search index and no server behind it,
which is why it also works offline from
a local folder and inside tiko's own F1 pane.
My test implementation for the moment is:
a) using wget
b) clean (docs/*js) and (assets/*css,*js) files from @v=(sha1-id)
b) using Tiko User Tools
=> Alt-1 => opens local => HelpCenter (same as online)
HelpCenter Alt+1
ASCII Chart Ctrl+1
GUID Generator Ctrl+2
```
structural scheme : Index in D:\Users\docs\helpcenter\docs\
...screenshots: (temporarily 30d)
#75
PlanetSquires Software / Re: PsControls Updates
Last post by Paul Squires - July 25, 2026, 11:23:06 AMHelp Center documentation updated for the new controls. PsProgressBar and PsOptionButton
https://www.planetsquires.com/docs/
https://www.planetsquires.com/docs/
#76
PlanetSquires Software / Re: PsControls Updates
Last post by Paul Squires - July 25, 2026, 11:14:21 AMNew control: PsOptionButton
Similar to PsCheckButton but allows for mutual exclusive selection with a group of PsOptionButtons.
Similar to PsCheckButton but allows for mutual exclusive selection with a group of PsOptionButtons.
#77
PlanetSquires Software / Re: PsControls Updates
Last post by Paul Squires - July 25, 2026, 10:49:47 AMNew control: PsProgressBar
Easier one because it is all visual.
Easier one because it is all visual.
#78
PlanetSquires Software / Re: Regarding UTF-16 encoding ...
Last post by Paul Squires - July 25, 2026, 10:48:05 AMIs it possible for you to post a sample file here? Change the file extension to *.txt if a .bas won't attach to a post here. I'd like to test your file directly rather than trying to create one of my own.
#79
PlanetSquires Software / Re: Regarding UTF-16 encoding ...
Last post by fbfans - July 25, 2026, 08:53:59 AMAlthough you've tested it and it works fine on your end, I'd still like to report my specific situation:
I am using the "Compile & Run" command to execute my programs, not "Quick Run". When my source file is saved in UTF‑16 with BOM format, even very simple code triggers errors. To be more specific:
Dim As Integer a = 100 : Print a — this runs fine, though it disappears immediately.
But when I add Sleep at the end, like this:
Dim As Integer a = 100 : Print a : Sleep
and then run it again, I get the following error:
error 14: Expected identifier, found 'integ'
At this point, the code displayed on the screen is exactly what I wrote above.
However, after I close the file and reopen it, the source code has become something like:
Dim As i
— with the rest of the characters completely missing.
I also tested this in FBIDE. Unlike Tiko, FBIDE only offers UTF‑16LE and UTF‑16BE (no BOM option), and the same code runs perfectly fine there.
I am using the "Compile & Run" command to execute my programs, not "Quick Run". When my source file is saved in UTF‑16 with BOM format, even very simple code triggers errors. To be more specific:
Dim As Integer a = 100 : Print a — this runs fine, though it disappears immediately.
But when I add Sleep at the end, like this:
Dim As Integer a = 100 : Print a : Sleep
and then run it again, I get the following error:
error 14: Expected identifier, found 'integ'
At this point, the code displayed on the screen is exactly what I wrote above.
However, after I close the file and reopen it, the source code has become something like:
Dim As i
— with the rest of the characters completely missing.
I also tested this in FBIDE. Unlike Tiko, FBIDE only offers UTF‑16LE and UTF‑16BE (no BOM option), and the same code runs perfectly fine there.
#80
PlanetSquires Software / Re: Regarding UTF-16 encoding ...
Last post by Paul Squires - July 25, 2026, 06:40:36 AMNo problem at all. This is one of those areas that needs to be 100% solid. Compiling any type of file format should never result in code corruption. José mentioned in a different post that he has seen code corruption when switching back and forth among different file encodings, eg. ANSI -> UTF-8 -> Unicode