Some questions about using Tiko 1.3

[José Roca]

I'm not planning to generate HTML, PDF or CHM documentation.
The whole point of using Markdown is precisely to avoid that kind of workflow. Markdown lets me update or correct the documentation instantly, without having to rebuild anything or maintain multiple output formats.

Anyone who wants HTML or PDF versions is, of course, free to generate them on their own using any of the many existing tools. But I won't be maintaining or producing those formats myself.

Best regards,
José

[Paul Squires]

Thanks José for the corrections. I am adding your changes to the code base. Appreciate it.

[Paul Squires]

1、Uncertain issue: The "Find in Files" shortcut Ctrl+Shift+F doesn't work when the Chinese input method is active, but it works fine after switching to English mode. Ctrl+Shift is used to switch between different Chinese input methods in Chinese Windows. However, I tested other functions that also use Ctrl+Shift combinations (such as Clear All Bookmarks, Uncomment Code Block), and they all work normally. I'm just reporting this.

You can map that shortcut to another key combination if you wish.
Just use File / Preferences / Keyboard Shortcuts  and Modify the Ctrl+Shift+F menu item.

<various encoding issues>

Let's discuss these issues again in the future once I release the next Tiko update. Maybe most will be fixed by the changes I have made.

3、I noticed that the tooltip text in the status bar keeps refreshing when hovering the mouse.

- Fixed: Added code to prevent main statusbar tooltip text from continuously refreshing.
Thanks

4、One suggestion: Regarding drag-and-drop copy, in Tiko you have to drag the selected text first and then hold Ctrl to copy, otherwise it won't work. In programs like Word, Notepad++, and EditPlus, you can either do it the Tiko way OR select text and hold Ctrl while dragging to the target location. Of course, this is just my habit—please ignore it if it's troublesome to implement.

Okay, I understand what you are trying to do, although I have not (yet) been able to implement it into Tiko. You say that it works in Notepad++? I could not get it to work there either. I tried using version 8.8.1 to so a Ctrl copy drag but it did not work. I'll keep trying to find a way to do this.

I just want Tiko to get better and better, with more complete Unicode support. Please don't mind me being so detailed. Thanks again.

Awesome, keep the suggestions coming.

Lastly, here is the Chinese translation file. I've tried my best to make the translations accurate, but my English is limited.

I have added it to the Tiko distribution and will be in the next release.

Note: hz-utf8-rom.bas, sub printhz cannot be displayed in the function list.

This problem is related to you using the "End" statement in your code. It screws up the internal parser. This problem has been reported before by another user but has not yet been fixed.
You probably should not be abruptly ending your program using "End".

[José Roca]

The use of both END and SYSTEM are strongly dircouraged.

Used to exit the program, and return to the operating system. An optional integer return value can be specified to indicate an error code to the system. If no return value is given, a value of 0 is automatically returned at the end of the program.

Usage of this statement does not cleanly close scope. Local variables will not have their destructors called automatically, because FreeBASIC does not do stack unwinding. Only the destructors of global variables will be called in this case.

For this reason, it is discouraged to use End simply to mark the end of a program; the program will come to an end automatically, and in a cleaner fashion, when the last line of module-level code has executed.

[José Roca]

Thanks José for the corrections. I am adding your changes to the code base. Appreciate it.

With the latest version of DWSTRING you can do:

Utf8ToAnsi

DIM dws AS DWSTRING = DWSTRING(utf8Str, CP_UTF8)
DIM ansiStr AS STRING = dws

UnicodeToUtf8

DIM strUtf8 AS STRING = dws.utf8

AnsiToUtf8

DIM dws AS DWSTRING = strAnsi
DIM strUtf8 AS STRING = dws.utf8

I also have added detection of utf8, so if you pass an ansi string to DIM dws AS DWSTRING = DWSTRING(utf8Str, CP_UTF8) it will be treated as ansi by changing CP_UTF8 to CP_ACP.

If you don't want/need a check, for speed reasons, then use:

DIM dws AS DWSTRING
dws.utf8 = strUtf8

[José Roca]

A little additional clarification.

If the utf-8 string is directly assigned to the DWSTRING's constructor with the code page CP_UTF8, it calls the method IsUtf8:

Code Select Expand

' ========================================================================================
' Detect if the codification is valid utf-8
' ========================================================================================
FUNCTION DWSTRING.IsUtf8 (BYREF s AS STRING) AS BOOLEAN
   IF LEN(s) = 0 THEN RETURN FALSE
   ' // Detect if it has a UTF-8 BOM
   IF LEN(s) >= 3 THEN
      IF ASC(s,1) = &hEF AND ASC(s,2) = &hBB AND ASC(s,3) = &hBF THEN RETURN TRUE
   END IF
   DIM wLen AS LONG = MultiByteToWideChar(CP_UTF8, MB_ERR_INVALID_CHARS, STRPTR(s), LEN(s), NULL, 0)
   RETURN (wLen <> 0)
END FUNCTION
' ========================================================================================

This method first checks if it has an utf-8 BOM; if not, it calls MultiByteToWideChar with the MB_ERR_INVALID_CHARS. This function fails both if it is not utf-8 and also if is malformed utf-8.

Assigning the string using the uft8 property of DWSTRING (dws.utf8 = string) assumes that you're passing a valid utff-8 string and it does not check the validity and calls WideCharToMultiByte without the MB_ERR_INVALID_CHARS flag. In Windows XP an earlier, the bad character(s) were dropped; in Vista and later, the bad character(s) are converted to U+FFFD (�).

For more information, see Raymond Chen's post "How does the MultiByteToWideChar function treat invalid characters?" at https://devblogs.microsoft.com/oldnewthing/20120504-00/?p=7703

[José Roca]

I have asked to CoPilot about porting DWSTRING to Linux to end with the UTF-8 headaches that the users of that platform have.

And it has replied:

🌟 Why a Linux version of DWSTRING is straightforward
1. FreeBasic already gives you UTF‑32 on Linux
This is the jackpot.

WSTRING on Linux = UCS‑4 = UTF‑32.

So your internal buffer becomes:

one code point per element

no surrogate pairs

no variable‑length sequences

no special cases

It's even simpler than UTF‑16.

2. Linux already provides the conversion functions you need
Windows:

MultiByteToWideChar

WideCharToMultiByte

Linux equivalents:

mbstowcs / wcstombs

or iconv for more control

These do the same job.

3. The rest of DWSTRING is platform‑neutral
Your class logic:

constructors

operators

buffer management

UTF‑8 validation

integration with FB string functions

...all of that stays exactly the same.

4. No external dependencies means no obstacles
Because DWSTRING is:

one file

no DLLs

no libraries

no Windows‑only baggage

...it ports cleanly.

🌱 Why it feels too easy
Because most developers approach UTF‑8 the hard way:

manipulating UTF‑8 directly

rewriting MID, LEFT, RIGHT, INSTR

building ULeft, URight, UInstr, UReplace, URemove, UInsert

getting stuck on UMid

fighting variable‑length encodings

You avoided all of that by stepping outside the UTF‑8 trap entirely.

Your model:

UTF‑8 → fixed‑width Unicode → process → fixed‑width Unicode → UTF‑8
...is the same one used by:

Windows

.NET

Java

Qt

Python (on Linux)

ICU

It's easy because it's correct.

🎯 And yes — with or without AI, it's absolutely doable
An AI tool could help generate the Linux version, but the truth is:

your design is already portable

the platform‑specific parts are tiny

the logic is universal

the class is dependency‑free

So whether you do it manually or with assistance, the path is clear.

[fbfans]

@Paul, José
Thanks.
I have received your reply. The issue with the function name not being recognized is actually quite simple—I can resolve it by using "system." I'm just not entirely sure why this became a problem in the first place.

I'm not sure why I seem to be encountering so many issues, while other users don't seem to have any problems with tiko. I hope this doesn't come across the wrong way, Paul.

1、Regarding the function list for files: Shouldn't clicking on it allow you to expand or collapse the list for each file? Next to the filename, there's an arrow, but clicking it jumps to the function list of the corresponding file instead of collapsing it. In contrast, the bookmark list follows the expected behavior—clicking expands or collapses it properly.

2、The scrollbar's drag activation area is too narrow. When dragging the scrollbar, if the mouse moves even slightly outside of it, it immediately loses focus and the scrollbar disappears from view, even though it can still be dragged.

3、Regarding bookmarks: Normally, when a bookmark is set, it should appear in the sidebar's bookmark tab. Currently, only bookmarks set via Ctrl+F2 appear automatically. Bookmarks set with the mouse do not—they only show up if I click the "..." next to the sidebar bookmarks and select "Expand All." In your version 1.3 update, you added the option to set breakpoints. If you intend for a mouse click to set a breakpoint, I suggest either disabling the option to set bookmarks with the mouse (by removing the relevant setting) or assigning mouse double-click or another key to set breakpoints to avoid conflicts. Otherwise, a logic for adding mouse-set bookmarks to the list should be implemented.
    Additionally, I revisited the translation and am unsure if "clear bookmarks" in the sidebar and menu bar are meant to have the same meaning. However, the menu bar option only clears bookmarks for the current file, while the sidebar option clears bookmarks for all files. The difference is quite significant—they are not performing the same action, so the translation should reflect that distinction.

I feel that José's dwstring is very powerful. The transcoding function he demonstrated is logically clear, concise, robust, and almost like a tutorial—definitely worth learning from. I suggest Paul consider adopting this approach, as it could greatly simplify Unicode-related issues. It's highly recommendable.

Regarding Chinese character encoding conversion, Paul may not have the conditions to test it. If you need me to assist with testing, feel free to reach out—I'm more than happy to help. Email: erliwuwei@gmail.com.

[Paul Squires]

I'm just not entirely sure why this became a problem in the first place.

It is a well known Tiko parsing problem. The "end" interferes with the parser determining where the Sub/Function ends because the parser expects an End Sub or an End Function.

1、Regarding the function list for files: Shouldn't clicking on it allow you to expand or collapse the list for each file? Next to the filename, there's an arrow, but clicking it jumps to the function list of the corresponding file instead of collapsing it. In contrast, the bookmark list follows the expected behavior—clicking expands or collapses it properly.

Yes, you're probably right about this one. I will look to see why I implemented it the current way and change it if necessary.

2、The scrollbar's drag activation area is too narrow. When dragging the scrollbar, if the mouse moves even slightly outside of it, it immediately loses focus and the scrollbar disappears from view, even though it can still be dragged.

This was actually fixed (finally) yesterday. See this post: https://www.freebasic.net/forum/viewtopic.php?p=310481#p310481

3、Regarding bookmarks: Normally, when a bookmark is set, it should appear in the sidebar's bookmark tab. Currently, only bookmarks set via Ctrl+F2 appear automatically. Bookmarks set with the mouse do not—they only show up if I click the "..." next to the sidebar bookmarks and select "Expand All."

Yes, there does seem to be something strange going on there.

In your version 1.3 update, you added the option to set breakpoints. If you intend for a mouse click to set a breakpoint, I suggest either disabling the option to set bookmarks with the mouse (by removing the relevant setting) or assigning mouse double-click or another key to set breakpoints to avoid conflicts. Otherwise, a logic for adding mouse-set bookmarks to the list should be implemented.

Check out the option:  Environment Options / Code Editor / "Margin click toggles Breakpoint rather than Bookmark".

Additionally, I revisited the translation and am unsure if "clear bookmarks" in the sidebar and menu bar are meant to have the same meaning. However, the menu bar option only clears bookmarks for the current file, while the sidebar option clears bookmarks for all files. The difference is quite significant—they are not performing the same action, so the translation should reflect that distinction.

Clicking the "..." and selecting "clear bookmarks" should remove all bookmarks from ALL files. Right clicking on a filename within the list and selecting "clear bookmarks" should remove bookmarks from that file only. Maybe I should rename them "Clear Bookmarks All Files", and " Clear Bookmarks Selected File", or something similar.

[fbfans]

Hi Paul, thanks for your reply.

Check this option: Environment Options / Code Editor / "Click margin to toggle breakpoints instead of bookmarks".

What I mean is, the current program logic is that clicking the line number area either sets a breakpoint or a bookmark—you can only choose one. Why not set breakpoints with a left click and bookmarks with a right click, or use a double click and other mouse actions instead?
Another point is about the real-time sync and refresh issue of the sidebar. I closed some files, but their entries still remain in the sidebar. Theoretically, the closed files should no longer be displayed there.
I downloaded the tiko.exe file from the development branch directory on GitHub at https://github.com/PaulSquires/tiko/tree/development, and it prompts an incompatibility with my system. I'm using 64-bit Windows 11.
Also, I just noticed there are so many discussions about Tiko on the FreeBASIC forum. Should I post my questions there from now on?