Some questions about using Tiko 1.3

[fbfans]

Dear Paul,

I'm a FreeBasic hobbyist from China, and I really like Tiko—its clean design and Chinese language support are great. I've been using it for a while and have a few questions:

Some functions and procedures aren't recognized. Are there specific things I should pay attention to?

The text encoding switch in the bottom-right corner – when I click it, characters get corrupted, both Chinese and English turn into gibberish, and the source code becomes unusable. Is this indicator only meant to show the file encoding, not to change it?

Under ANSI encoding, can the editor be improved to delete one entire Chinese character at a time?
Chinese characters are double‑byte in ANSI mode, and currently pressing delete removes only half, leaving the other half as garbage.

The TODO list in the Output window shows garbled Chinese in Unicode mode. It only displays correctly in ANSI mode.

I'm also working on localizing Tiko. What does "Search Results" in the Output window refer to? I'm not sure how to use it.

Apologies if anything I've said isn't clear—I'm still learning. Thank you for your time and for this wonderful IDE.

[Paul Squires]

I'm a FreeBasic hobbyist from China, and I really like Tiko—its clean design and Chinese language support are great.

Awesome, happy to hear that the editor is working okay for you! I have not dealt with many Chinese programmers and sadly I do not understand the language so it's hard for me to give 100% accurate advice.

Some functions and procedures aren't recognized. Are there specific things I should pay attention to?

Most likely this is the limitation of Tiko's built in parser. There are probably edge cases that Tiko is not parsing correctly to identify them as functions. If you wish, you can post an example of the Function / End Function signature that you see as not being recognized and I'll take a look to identify the underlying problem.

The text encoding switch in the bottom-right corner – when I click it, characters get corrupted, both Chinese and English turn into gibberish, and the source code becomes unusable. Is this indicator only meant to show the file encoding, not to change it?

It *should* try to change it.

Under ANSI encoding, can the editor be improved to delete one entire Chinese character at a time?
Chinese characters are double‑byte in ANSI mode, and currently pressing delete removes only half, leaving the other half as garbage.

I don't think that you should use ANSI mode with Chinese characters. You are better off using UTF-8 or UTF-16 (unicode) because ANSI is only designed for single character languages like English and other Latin based languages.

The TODO list in the Output window shows garbled Chinese in Unicode mode. It only displays correctly in ANSI mode.

Ahhh... I just checked Tiko's internal database and it looks like years ago that I must have used fixed ZSTRING elements instead of WSTRING !!! That could definitely be the reason why the TODO text is wrong. It could also be why the functions/procedures are not being displayed. I will change this and post an update.

I'm also working on localizing Tiko. What does "Search Results" in the Output window refer to? I'm not sure how to use it.

Search Results are those found when you perform a "Find in Files" found under the Edit menu (or Ctrl+Shift+F)

[Paul Squires]

I have made changes to Tiko's internal parser and database to use AfxNova's dynamic wstring DWSTRING variable type. This *should* help with storing the Chinese text for TODO, etc.

The changes (and tiko.exe) can be found on the Development branch. That branch will eventually be merged to the Master branch before the next Tiko update is released.

https://github.com/PaulSquires/tiko/tree/development
 

[fbfans]

Thanks, Paul. You've been really helpful. I truly appreciate your patience and detailed replies to a beginner like me—I feel really honored and flattered.I'm not familiar with GitHub, so I'll just wait for your next version update. Right now, there are only minor issues that affect the experience a little but don't prevent usage, and I can work around them.I'm really looking forward to the powerful code completion in the new version. I find it difficult to type AFXNova-related functions because they are too long and hard to remember (my English isn't very good).
I'll keep reporting small issues with Tiko, and I hope I don't bother you:
1、Uncertain issue: The "Find in Files" shortcut Ctrl+Shift+F doesn't work when the Chinese input method is active, but it works fine after switching to English mode. Ctrl+Shift is used to switch between different Chinese input methods in Chinese Windows. However, I tested other functions that also use Ctrl+Shift combinations (such as Clear All Bookmarks, Uncomment Code Block), and they all work normally. I'm just reporting this.
2、The following are all encoding issues with double-byte characters like Chinese. You might not have the environment to test them, so I took screenshots. Although they don't affect Latin-language users at all, I still hope you can add them to your to-do list and fix them when you have time.
(1) Find in Files cannot search for Chinese characters in Unicode-encoded files; it only works in ANSI format. As you can see from the filenames in my screenshot, several files starting with "hz" have identical content but different encodings (maybe your recent fixes have addressed this using DWSTRING, but I haven't been able to test it).
(2) When using split view, garbled characters appear in ANSI-encoded files. It seems that in horizontal split, the left pane shows garbled text; in vertical split, the top pane shows garbled text. The garbled characters disappear when restoring to a single window. No such issue with Unicode files.
3、I noticed that the tooltip text in the status bar keeps refreshing when hovering the mouse. Is this the default behavior of AFXNova functions? I've been studying the Limit function recently, so I'm quite sensitive to similar issues, haha.
4、One suggestion: Regarding drag-and-drop copy, in Tiko you have to drag the selected text first and then hold Ctrl to copy, otherwise it won't work. In programs like Word, Notepad++, and EditPlus, you can either do it the Tiko way OR select text and hold Ctrl while dragging to the target location. Of course, this is just my habit—please ignore it if it's troublesome to implement.
I just want Tiko to get better and better, with more complete Unicode support. Please don't mind me being so detailed. Thanks again.
Lastly, here is the Chinese translation file. I've tried my best to make the translations accurate, but my English is limited. Please forgive any inaccuracies.
Note: hz-utf8-rom.bas, sub printhz cannot be displayed in the function list.

[José Roca]

Welcome to the forum.

Never use ANSI with Chinese.

Code Select Expand

dim as string hz1,hz2
hz1 = "123微软专用操作系统安装工具456！"
hz2 = "abcgq这是显示汉字的子程序def"

The STRING data type is ansi. Therefore, the Chinese characters will be comverted to ?? or similar. Use WSTRING or AfxNova's DWSTRING.

The use of SYSTEM and END are discouraged. Please, read the manual.

[hajubu]

Hi Jose,

reading just the discussion here, I have stated some deviations in the 'AfxNova'-Files.

When introducing the new generic Info Header in the *.bi,*.inc you used files in Ansii-format.

Would it not better using UTF8  , or even UTF16-LE - as long "GitHub" does not interfere ,
to make the life easier for the presentation in "old" Editors.

b.r. Hans

P.S. Besides these small glitches,
I do really appreciate the effort and amount of hours , put in the work for the community-,
- especially here in the Tiko-Base .

Thanks so much ! Merci beaucoup ! Muchas gracias ! Vielen Dank !



'---------------------------------------------------------------------------
 FAQ for JOSE - AfxNova Files ( except doc-files (.md)
'---------------------------------------------------------------------------
1st) Why using Ansii-Text at all ,
2nd) with [left_quote/right_quote],[chr(147).. ..chr(148)]
3rd) Would it not better, using (at least) Utf8-text throughout all AfxNova-files
IMHO::
--> advantage - UTF8 depiction/presentation would be clearer for all cases (EN-us).
BUT ....
--> typically:: UTF16 for East Asian languages ,most: one 16 bit word, rare: two 16 bit words
--> and for Western languages  UTF-8 (ASCII is equivalent over the ASCII range (0-127) )
--> Processing: UTF-16 for user-mode applications is easier than processing UTF-8
--> Windows(today) :: UTF16  ,
--> (UTF32 rarely used , where codepoints ask for....)
::: Unicode .version....17...2025 AND goes on ...18.... t.b.d.
'-----------------------------------------------------------------------------
'
'*************************************
--> here my findings in AFxNova-files for the quotes around a 'tag'
'************************************
'RE(\sMIT\s) License
' in 99 files
'---
' i.e. ANSI TEXT {MIT} and  "AS IS" e.g. with  [Double_quotes], [chr(34).. ..chr(34)]
' Written in 2017...  by José Roca     -->'4a 6f 73 E9'::'José'
'' THE SOFTWARE IS PROVIDED "AS IS",   -->'22 41 53 20 49 53 22'::'"AS IS"'
'---
'...\AfxNova\AfxComplex.inc
'...\AfxNova\CComplex.inc
'---------------------------
'---------------------------
'RE (\sMIT\s)|("AS IS")
'in 97 files
---
' i.e. ANSI TEXT ... "AS IS"  with  [left_quote/right_quote],[chr(147).. ..chr(148)]
' Copyright (c) 2025 José Roca   --> '4a 6f 73 e9'::'José'
' License: Distributed under the MIT license.
' THE SOFTWARE IS PROVIDED "AS IS",   -->'93 41 53 20 49 53 94 2c'::'"AS IS",'
'
'***************************

more details in the attachment (list of the ANSII-AfxNova-Files)

[José Roca]

I have had problems with not ansi files uploaded to GitHub when someone downloads them as a zip file, specially if they are in utf-16. The problem seems to be caused by the BOM.

[José Roca]

I have made changes to Tiko's internal parser and database to use AfxNova's dynamic wstring DWSTRING variable type. This *should* help with storing the Chinese text for TODO, etc.

The changes (and tiko.exe) can be found on the Development branch. That branch will eventually be merged to the Master branch before the next Tiko update is released.

https://github.com/PaulSquires/tiko/tree/development
 

Paul, if you're using FreeBASIC's UTF‑8 conversion routines, I strongly suggest switching to MultiByteToWideChar and WideCharToMultiByte. FreeBASIC's routines are notoriously flaky and tend to corrupt characters.

[Paul Squires]

Thanks José,

Yes, I am using MultiByteToWideChar and WideCharToMultiByte for the conversions. I have posted the routines below that I am using for the conversions back and forth.

Code Select Expand

' ========================================================================================
' Maps UTF-8 string to Ansi string.
' ========================================================================================
function Utf8ToAnsi(byref strUtf8 as string) as string

   dim i as long                ' // Loop counter
   dim strAscii as string       ' // Ascii string
   dim idx as long              ' // Position in the string
   dim c as long                ' // ASCII code
   dim b2 as long               ' // Second byte
   dim fSkipChar as boolean     ' // Flag

   if len(strUtf8) = 0 then exit function
   
   ' // The maximum length of the translated string will be
   ' // the same as the length of the original string.
   ' // We are pre-allocating the buffer for faster operation
   ' // than concatenating each character one by one.
   strAscii = space(len(strUtf8))

   ' // Intialize index position in the string buffer
   ' // used to store the converted Ascii string
   idx = 1
   
   ' // Examine the contents of each character in the UTF-8 encoded string
   for i = 1 to len(strUtf8)
      ' // If fSkipChar is set we have to skip this character
      if fSkipChar then
         fSkipChar = 0
         continue for
      end if
      ' // Get the Ascii code of the character
      c = asc(mid(strUtf8, i, 1))
      ' // If it is betwen 0 and 127...
      if c < 128 then
         ' // ...we simply copy it to the string buffer...
         mid(strAscii, idx, 1) = mid(strUtf8, idx, 1)
         ' // ...and increase the position by 1.
         idx = idx + 1
      elseif c < 224 then
         ' // We need to join this byte and the next byte.
         b2 = asc(mid(strUtf8, i + 1, 1))
         if b2 > 127 then
            c = (c - 192) * 64 + (b2 - 128)
            mid(strAscii, idx, 1) = chr(c)
            ' // Set the flag to skip the next character
            fSkipChar = true
            ' // Increase the position by 1.
            idx = idx + 1
         end if
      end if
   next

   ' // Return the string
   function = left(strAscii, idx - 1)

end function


' ========================================================================================
' Maps Unicode character string to a UTF-8 string.
' ========================================================================================
function UnicodeToUtf8( byval wzUnicode as DWSTRING ) as string
    dim sUtf8 as string

    ' Maps Unicode character string to a UTF-8 string.
    sUtf8 = string(len(wzUnicode) * 2, 0)
   
    dim as long bytesWritten = _
        WideCharToMultiByte ( _
        CP_UTF8, _                        'Set to UTF-8
        0, _                              'Conversion type
        cast(LPCWSTR, wzUnicode.vptr), _  'Unicode string to convert
        len(wzUnicode), _                 'Length of Unicode string
        cast(LPSTR, strptr(sUtf8)), _     'UTF-8 string
        len(sUtf8), _                     'Length of UTF-8 buffer
        byval 0, _                        'Invalid character replacement
        byval 0)                          'Replacement was used flag

    function = left(sUtf8, bytesWritten)

end function


' ========================================================================================
' Maps Ansi character string to a UTF-8 string.
' ========================================================================================
function AnsiToUtf8( byref sAnsi as string ) as string
    dim sUnicode as string
    dim sUtf8    as string

    'Maps Ansi character string to a UTF-8 string.

    'Step one, convert to UNICODE
    sUnicode = string(len(sAnsi) * 2, 0)
    MultiByteToWideChar(CP_ACP, _                  'System default Windows ANSI code page
                     MB_PRECOMPOSED, _          'Conversion type
                     cast(LPCSTR, strptr(sAnsi)), _     'ANSI string to convert
                     len(sAnsi), _              'Lenght of ANSI string
                     cast(LPWSTR, strptr(sUnicode)), _  'Unicode string
                     len(sUnicode))             'Lenght of Unicode buffer

    'Step two, convert to UTF-8
    sUtf8 = string(len(sAnsi), 0)
    WideCharToMultiByte(CP_UTF8, _                 'Set to UTF-8
                     0, _                       'Conversion type
                     cast(LPCWSTR, strptr(sUnicode)), _  'Unicode string to convert
                     len(sUnicode) / 2, _       'Lenght of Unicode string
                     cast(LPSTR, strptr(sUtf8)), _     'UTF-8 string
                     len(sUtf8), _              'Length of UTF-8 buffer
                     byval 0, _                 'Invalid character replacement
                     byval 0)                   'Replacement was used flag
    function = sUtf8

end function


' ========================================================================================
' Determine if a string is likely UTF-8 encoded
' ========================================================================================
function isUTF8encoded(byref text as string) as boolean
    dim as ubyte ptr p = strptr(text)
    dim as integer nlen = len(text)
    dim as integer i = 0
    dim as boolean hasUTF8Sequences = false

    while i < nlen
        dim as ubyte c = p[i]

        if c < &h80 then
            ' ASCII character
            i += 1

        elseif (c and &hE0) = &hC0 then
            ' 2-byte UTF-8 sequence
            if i + 1 >= nlen then return false
            if (p[i + 1] and &hC0) <> &h80 then return false
            if c < &hC2 then return false
            hasUTF8Sequences = true
            i += 2

        elseif (c and &hF0) = &hE0 then
            ' 3-byte UTF-8 sequence
            if i + 2 >= nlen then return false
            if (p[i + 1] and &hC0) <> &h80 then return false
            if (p[i + 2] and &hC0) <> &h80 then return false
            hasUTF8Sequences = true
            i += 3

        elseif (c and &hF8) = &hF0 then
            ' 4-byte UTF-8 sequence
            if i + 3 >= nlen then return false
            if (p[i + 1] and &hC0) <> &h80 then return false
            if (p[i + 2] and &hC0) <> &h80 then return false
            if (p[i + 3] and &hC0) <> &h80 then return false
            hasUTF8Sequences = true
            i += 4

        else
            ' Invalid byte for UTF-8
            return false
        end if
    wend

    return hasUTF8Sequences
end function


[José Roca]

'*************************************
--> here my findings in AFxNova-files for the quotes around a 'tag'
'************************************
'RE(\sMIT\s) License
' in 99 files
'---
' i.e. ANSI TEXT {MIT} and  "AS IS" e.g. with  [Double_quotes], [chr(34).. ..chr(34)]
' Written in 2017...  by José Roca    -->'4a 6f 73 E9'::'José'
'' THE SOFTWARE IS PROVIDED "AS IS",  -->'22 41 53 20 49 53 22'::'"AS IS"'
'---
'...\AfxNova\AfxComplex.inc
'...\AfxNova\CComplex.inc
'---------------------------
'---------------------------
'RE (\sMIT\s)|("AS IS")
'in 97 files
---
' i.e. ANSI TEXT ... "AS IS"  with  [left_quote/right_quote],[chr(147).. ..chr(148)]
' Copyright (c) 2025 José Roca  --> '4a 6f 73 e9'::'José'
' License: Distributed under the MIT license.
' THE SOFTWARE IS PROVIDED "AS IS",  -->'93 41 53 20 49 53 94 2c'::'"AS IS",'
'

I don't know what you're doing. All the files are ANSI, and the text THE SOFTWARE IS PROVIDED "AS IS" is written exactly the same way in all of them (I literally copy and paste it). Maybe you're opening the file with an editor that replaces straight quotes with typographic ones?

[José Roca]

AnsiToUtf8

1. You allocate the UTF‑8 buffer as `len(sAnsi)`, but UTF‑8 may require up to 3 bytes per ANSI character. This means WideCharToMultiByte can truncate or produce invalid output.

2. You never call WideCharToMultiByte with a NULL buffer to obtain the correct output size. This is the recommended and safe way to use the API.

3. You don't check the return values of MultiByteToWideChar or WideCharToMultiByte, so conversion errors go unnoticed.

My code first calls MultiByteToWideChar/WideCharToMultiByte with a NULL buffer to get the exact size needed, allocates the correct amount of memory, and only then performs the conversion. That's why it doesn't corrupt characters.

Code Select Expand

function AnsiToUtf8( byref sAnsi as string ) as string
    dim as integer wlen, u8len
    dim as string sUnicode, sUtf8

    if len(sAnsi) = 0 then return ""

    ' Step 1: ANSI ? UTF-16 (size query)
    wlen = MultiByteToWideChar(CP_ACP, 0, strptr(sAnsi), len(sAnsi), NULL, 0)
    if wlen = 0 then return ""

    sUnicode = string(wlen * 2, 0)

    if MultiByteToWideChar(CP_ACP, 0, strptr(sAnsi), len(sAnsi), cast(LPWSTR, strptr(sUnicode)), wlen) = 0 then
        return ""
    end if

    ' Step 2: UTF-16 ? UTF-8 (size query)
    u8len = WideCharToMultiByte(CP_UTF8, 0, cast(LPCWSTR, strptr(sUnicode)), wlen, NULL, 0, NULL, NULL)
    if u8len = 0 then return ""

    sUtf8 = string(u8len, 0)

    if WideCharToMultiByte(CP_UTF8, 0, cast(LPCWSTR, strptr(sUnicode)), wlen, strptr(sUtf8), u8len, NULL, NULL) = 0 then
        return ""
    end if

    return sUtf8
end function


[hajubu]

Quote:
-------------------
I don't know what you're doing. All the files are ANSI, and the text THE SOFTWARE IS PROVIDED "AS IS" is written exactly the same way in all of them (I literally copy and paste it). Maybe you're opening the file with an editor that replaces straight quotes with typographic ones?
-------------------
In the discussion with transfering libraries to a better standard, it might a good idea to be an the safe side  for presentation the  character-depiction with a common base.

SO my first question was "why using Ansi" and not "UTF8" w.o. BOM

Nevertheless (.\AfxNova\*.* has overall app 1958 with 1759 text files '(1479*.bas, 114*.md, 166*(.bi,*.inc), 17*.xml, 12*.rc, 2*.txt)' whereof I stated are using the char(147/148) in Ascii (Ansi) .


All my Editor(s) are doing fine; therefore I could identify that case.
All depiction/presentation works in the Editors fine.

BUT not all tools showing for the char(147),char(148) inside an Ascii-text the correct left-double /right-double Quote : instead then presenting little black square ( as it above char(127) - Extended Ascii-Codes -128-255

Then the differences of  Windows-1252 using displayable characters , superset of  ISO 8859-1 / Latin-1, in terms of printable characters, but differs from the IANA's ISO-8859-1) in the range of 128-159.
Sometimes it is sufficient to select the correct code-page to be conform with wanted visualizing.

To avoid such things , I prefer using therefore the char(34) for the std. Text-Quote
to be on the safe side.

b.r. Hans

P.S. Jose : Do you have plans to adapt (convert) *.md to *.html and then to *.pdf for a common Help-System to be used outside or within Tiko?
If Yes, I can give you a hand, as I have already started with a prototype 'Windows-Procedures'( html, pdf) with TOC (table-of-content)


 




 

[José Roca]

UnicodeToUtf8 has the same sizing issue

Even though LEN(wzUnicode) returns the number of UTF‑16 characters, multiplying by 2 does not guarantee enough space for UTF‑8. UTF‑8 may need 1, 2, 3 or 4 bytes per character.

Code Select Expand

function UnicodeToUtf8( byref wzUnicode as DWSTRING ) as string
    dim as long u8len
    dim as string sUtf8

    ' Step 1: get required UTF‑8 size (including null terminator)
    u8len = WideCharToMultiByte( _
                CP_UTF8, _
                0, _
                cast(LPCWSTR, *wzUnicode), _
                -1, _
                NULL, _
                0, _
                NULL, _
                NULL)

    if u8len = 0 then return ""

    ' Step 2: allocate buffer
    sUtf8 = string(u8len, 0)

    ' Step 3: convert
    WideCharToMultiByte( _
        CP_UTF8, _
        0, _
        cast(LPCWSTR, *wzUnicode), _
        -1, _
        strptr(sUtf8), _
        u8len, _
        NULL, _
        NULL)

    ' Remove null terminator
    return left(sUtf8, u8len - 1)
end function


[José Roca]

Utf8ToAnsi

Code Select Expand

function Utf8ToAnsi( byref sUtf8 as string ) as string
    dim as long wlen, alen
    dim as string sUnicode, sAnsi

    if len(sUtf8) = 0 then return ""

    '-----------------------------------------
    ' Step 1: UTF‑8 → UTF‑16 (size query)
    '-----------------------------------------
    wlen = MultiByteToWideChar( _
                CP_UTF8, _
                0, _
                strptr(sUtf8), _
                len(sUtf8), _
                NULL, _
                0)

    if wlen = 0 then return ""

    sUnicode = string(wlen * 2, 0)

    MultiByteToWideChar( _
        CP_UTF8, _
        0, _
        strptr(sUtf8), _
        len(sUtf8), _
        cast(LPWSTR, strptr(sUnicode)), _
        wlen)

    '-----------------------------------------
    ' Step 2: UTF‑16 → ANSI (size query)
    '-----------------------------------------
    alen = WideCharToMultiByte( _
                CP_ACP, _
                0, _
                cast(LPCWSTR, strptr(sUnicode)), _
                wlen, _
                NULL, _
                0, _
                NULL, _
                NULL)

    if alen = 0 then return ""

    sAnsi = string(alen, 0)

    WideCharToMultiByte( _
        CP_ACP, _
        0, _
        cast(LPCWSTR, strptr(sUnicode)), _
        wlen, _
        strptr(sAnsi), _
        alen, _
        NULL, _
        NULL)

    return sAnsi
end function


[José Roca]

isUTF8encoded

Code Select Expand

function isUTF8encoded( byref s as string ) as boolean
    if len(s) = 0 then return false

    ' Optional: Detect UTF-8 BOM
    if len(s) >= 3 then
        if asc(s,1)=&hEF and asc(s,2)=&hBB and asc(s,3)=&hBF then return true
    end if

    dim as long wlen = MultiByteToWideChar( _
        CP_UTF8, _
        MB_ERR_INVALID_CHARS, _   ' strict validation
        strptr(s), _
        len(s), _
        NULL, _
        0)

    return (wlen <> 0)
end function