PlanetSquires Forums

Just uploaded a new release 1.3.0. Way too many internal changes to list so the change log is pretty generic.
https://github.com/PaulSquires/WinFBE/releases

Version 1.3.0 (March 24, 2017)
- Changed: Major internal changes. Project management; multiple projects. Too many changes to list.
- Changed: New Find/Replace functionality. Much more like that found in code editors like VSCode.

Hi Paul,

Thank you for the good news to all us FB coders: WinFBE. I’ve been using it for a couple of months already, and I’m very pleased indeed. But some changes in particular, from v1.2.4 to v1.3.0 made me slightly panicking (well, panicking).

Where’s the beautiful “Find in files”, the enormous “Replace with scope” (and let’s not forget the sweet “Find in all open documents”)? I’d be one happy man if you’d even consider putting back the “old” functions.

But, great work!
Fredrikk

Hi Fredrikk, the Find in Files is still there. It has not been removed. Replace is still there also but the new functionality for both Find and/or Replace is to act on the current active document or, if applicable, the current active selected area. I removed the options for all open documents. Not sure if I will add the All Open Documents back in there because I did find trouble/bugs with it previously.

I see. Shall take a closer look.

I downloaded and have been working with WinFBE 1.3.0.  The Function List seems to have 2 copies of the functions/subs.  Is this to be expected or is this a bug?  It is different from 1.24, which works as I expect it.  Thanks.

Hi Jim, yup, bug :)
Looking at it now.

Hi Jim, all fixed.   https://github.com/PaulSquires/WinFBE/releases
Version 1.3.1

Thanks

Thanks for WinFBE 1.31.  It fixed the Function List problem.  Something else has cropped up.  If I forget to close the Find window before I execute the project code, bad things happen. The editor goes into convolutions.  I usually restart WinFBE when this occurs.

Thanks
Jim

Hi Jim, I have having trouble recreating such behaviour here.
I did notice that if a word is highlighted in the code editor and you invoke Find dialog that the number of found matches is not displaying. that is now fixed and will be uploaded in next update.

I couldn't repeat my problem with the open Find dialog completely disabling WinFBE when executing code but the Find dialog does not redraw itself properly after quitting the execution of the code.  Also it appears that when adding new subs or functions, they will not appear on the Function List until the file is closed and then reopened.  A save command from the menu will properly align the existing Functions and line numbers but not add new subs or functions.

Thanks
Jim

Thanks Jim, you're exactly right. I have applied two changes. (1) To fix the repaint issue with the Find/Replace dialog, and (2) To fix the newly added sub/functions not displaying in Function List when file is saved.

Thanks for letting me know about these problems. If you find more then please let me know.

You simply need to download the EXE's from GitHub for the changes.

hi,Thanks for WinFBE 1.31。WinFBE when entering Chinese, continue to enter the ENTER key to wrap, the program will appear to stop running。

We use Windows Notepad.exe to create a new blank Unicode file and save it, open with WinFBE, and WinFBE does not appear to display Unicode files correctly.

winFBE 1.3.1+ win8.1 32bit

Thanks
ganlinlao

I tried to reproduce the problem. I used Notepad to create a file and saved it as "unicode". I then switched WinFBE option settings to rad/write files as utf-unicode. I could read the file but it was mostly ????? characters. I then created another text file using Notepad and saved it as UTF-8 in Notepad. I as then able to read and write to it with no problems in WinFBE.

You may need to ensure that you have selected the corrected codepage for displaying and entering your code. See attachment.

The Scintilla control does not understand Windows unicode (UTF-16) files. It deals with unicode using UTF-8.

thanks Jose Roca
When I install WinFBE in a directory containing Chinese characters, WinFBE cannot load file, if the code file is placed in a directory containing Chinese characters, the WinFBE can open code file, but cannot save edited code file, cannot compile code file

thanks
ganlinlao

Thanks ganlinlao,

I made a change to the handling of the filename for the Localization file name/path and have uploaded the new exe to GitHub at https://github.com/PaulSquires/WinFBE

You just need to download the WinFBE32.exe and/or WinFBE64.exe and overwrite your existing files.

Not sure if this will correct the problem but please let me know.

hi TechSupport
I downloaded the latest version of WinFBE from Github, but WinFBE could not load the file from the containing Chinese character directory. I found the reason for this is because the ExePath () function in the WinFBE.bas file could not return the directory name with Chinese characters correctly.

I tried to modify some of the code in WinFBE.bas and recompile it. WinFBE.exe can load a file with a Chinese character path。

code in winFBE.bas:   
'  dim wszLocalizationFile as WString * MAX_PATH
'   wszLocalizationFile = Exepath + wstr("\Languages\") + gConfig.LocalizationFile
'   If LoadLocalizationFile(@wszLocalizationFile) = False Then
'      MessageBoxW( 0, WStr("Localization file could not be loaded. Aborting application.") + vbcrlf + _
'                   wszLocalizationFile, _
'                   WStr("Error"), MB_OK Or MB_ICONWARNING Or MB_DEFBUTTON1 Or MB_APPLMODAL )

   dim wszLocalizationFile as cwstr
   wszLocalizationFile &=afxgetCurDir()
   wszLocalizationFile &= cwstr("\Languages\")
   wszLocalizationFile &= gConfig.LocalizationFile
   If LoadLocalizationFile(Cast(wstring ptr,wszLocalizationFile)) = False Then
      MessageBoxW( 0, WStr("Localization file could not be loaded. Aborting application.") + vbcrlf + _
                   wszLocalizationFile, _
                   WStr("Error"), MB_OK Or MB_ICONWARNING Or MB_DEFBUTTON1 Or MB_APPLMODAL )
      Return 1
   End If

Thanks
ganlinlao

Good catch, but better use AfxGetExePathName instead of AfxGetCurDir (this one may fail if the current directory is not the same that the path of the executable).

The instrinsic FB functions that deal with files and folders are not unicode aware. This is why I have provided replacements for them in my Afx framework. Wrapper functions in AfxWin.inc and AfxPath.inc, and methods in the classes CFileSys.inc and CFindFile.inc

Hi ganlinlao, thanks so much for finding th source of the problem. I will make the change to the source code and upload it to GitHub. I never realized that EXEPATH() was the source of the problem.

Thanks!
Paul

I am also going to change DIR() and CURDIR() functions to Afx versions,

Changes made and new EXE's uploaded to GitHub.

hi,TechSupport

WinFBE in dealing with Chinese characters, there will be some problems.

I also use notepad++ software to enter the same Chinese text and save it without this problem. Notepad++ is also dependent on Scilexer.dll

I switched to WinFBE in ANSI mode, new document, entered some English and Chinese. Enter the ENTER key at the end of the Chinese text to switch to a new line, and the program stops responding
(//)


thanks
ganlinlao

Thanks ganlinlao, I have a pretty good idea where I think this problem is coming from in the code. Give me a little while to rework some of the code and I will post some new EXE's for you to try. I will post here when the new code is ready.

I have changed the code in clsDocument that loads and saves the code text to honor the Ansi/Utf8 setting of the editor. It now uses the FreeBasic Open function with the appropriate file encoding setting based ont he Ansi/Utf8 setting in the editor. Please test to see if this corrects the Chinese character issue. Thanks!

hi，TechSupport
    I downloaded the latest version of WinFBE from Github，But it does not better solve the problem of Chinese characters, you use  gConfig.UnicodeEncoding  parameter, meaning that I edit the code containing the Chinese characters, can only use the UTF8 mode, can not use the ANSI mode, which is unfair, and will bring other problems。
My advice on this question is:
Source code file format, perhaps one of the Ansi,unicode,utf8+bom,utf8, we can first detect the source code file's first three bytes,  can determine that it is Unicode, utf8+Bom text file, and use the FielEncodingType variable to save it.
When I save the file, save it according to the FileEncodingType value.
I tried to wrote a function for detecting file formats and added a fields FileEncodintType to the Clsdocument class
it is mycode:
    Declare Function CheckFileEncoding(Byval pwszName As Wstring ptr) As String
'#########################

   Function CheckFileEncoding(Byval pwszName As Wstring ptr) As String       
       Dim As Long f,k
       Dim As String headBuffer,st
       f=freefile
       If Open(*pwszName For  Input   As #f)=0 Then
          Do Until EOF(f)
               For k=1 To 3
           Input #f,st
          headbuffer=headbuffer & st
          Next
          loop
   Close #f
           Select Case headBuffer[0]
         Case 255,254               'Make sure this.FileEncodingType is Unicode
         function= "unicode"
         Case 139                      'Make sure this.FileEncodingType is utf8+BOM
         function= "utf8+Bom"
         Case else
         function= "ascii"
      End Select
      Else
      afxmsg(WStr("Error opening: ") & *pwszName,48)
    End If     
   End function

'#############################
when  open File ：
……
     this.FileEncodingType=CheckFileEncoding(pwszFile)
       Select Case this.FileEncodingType
       Case "unicode"
                    hResult=Open(*pwszFile For Input  Encoding "utf16"  As #f)
       Case "utf8+Bom"
               hResult=Open(*pwszFile For Input  Encoding "utf8"  As #f)
            Case Else
          hResult=Open(*pwszFile For Input  Encoding "ascii"  As #f)
       End Select       
       If hResult=0 then           
               Do Until Eof(f)
                  Line Input #f, st
                  sText = sText & st & vbCrLf
      Loop
      close #f
                    if len(sText) > 2 THEN  ' or will GPF                          
               If IsTextUnicode(StrPtr(sText), 2, Cast(LPINT, @nResult) ) Then
                                this.FileEncodingType="utf8"                 'sorry,I don't know how to use the Istextunicode () function, it seems to be here not to work                       
                               sText = AfxACode( Cast(WSTRING Ptr, StrPtr(sText)) )
                       End If 
                   End if      
               this.SetText( sText )
               this.DateFileTime = AfxGetFileLastWriteTime( *pwszFile )
            else
               print "Error opening: "; *pwszFile
            End If

when save File:


   ' Save text buffer to disk by directly accessing buffer rather
   ' saving it to an intermediary string variable first.
   Dim As ZString Ptr psz = Cast( ZString Ptr, SciExec(this.hWindow, SCI_GETCHARACTERPOINTER, 0, 0) )
   f = FreeFile
   Select Case this.FileEncodingType
   Case "unicode"
   'if gConfig.UnicodeEncoding THEN
      Open wFilename for output Encoding "unicode" As #f
    Case "utf8+bom"
       Open wFilename for output Encoding "utf8" As #f
   Case else   
      Open wFilename for output As #f
   End select
   if Err > 0 then
      ' error saving file
      print "Error saving file: "; wFilename
   else
      print #f, *psz
   end if   
   close #f
’######################
This seems to be able to correctly open the text containing the Chinese character file, whether the Unicode,utf8+bom,utf8,ansi file, edit the file, and then save, no problem. I use notepad++ to reopen the file, and can display it correctly.

thanks
ganlinlao

Thanks ganlinlao, I appreciate the work you have done with this and the help you are giving. I will implement the changes and I will also download the Notepad++ source to take a look to see how they are handling different file formats. I hope to post new EXE's for you today. Thanks!

There is a lot more to this than I realized.

- Opening the file, as expected I need to read the first few bytes to determine if a BOM exists. If it does, then:
-- If it is UTF16 unicode then I expect that I need to convert it to UTF8 in order to display in the editor.
-- If it is UTF8 BOM then no problem. Simply load the file and display it.
-- Problem is that UTF8 can also exist without a BOM so I need to research how to determine if the text is UTF8 or simply ASCII. From what I read this has do do with 7bit or 8bit usage of the bytes in the file to determine if variable number multiple bytes are needed to represent a character (UTF8).

Once I know the file encoding then saving the file afterwards should be easier. FB's built in file commands allow for the use of Encodings. Only thing to be aware of is that UTF16 and UTF8 both save using a BOM. If a file is opened as UTF8 without a BOM then I would need to ensure that it is also saved without a BOM.

Interesting stuff. I will do some searches on FB's forum to see if anyone has already written code to determine if file is UTF8 if BOM doesn't exist. The Notepad++ source code contains many functions for conversion to and from unicode and utf8 but it also adds the complexity of dealing with big endian and little endian file formats.

Jose already has the AfxUcode and AfxAcode functions available so converting to and from unicode/utf8/ascii using codepages should be okay. I think the only thing I need now is a function to determine if a file being opened without a BOM is UTF8 and not just regular ASCII. I believe Notepad++ has code for this that I will look at.

Ok, looks like I have all of this figured out now. Working on incorporating it into the editor. Hope to post EXE's soon.

...only one problem that I can not figure out why there is an error. For UTF-16 unicode files I need to convert the UTF-8 string in the Scintilla control to a UTF-16 string in order to output it to the disk file. The Scintilla control has been set to use SC_CP_UTF8.

Here is the code that I thought would work but does not. It returns 0 and GetLastError returns 1004 which indicates a bad parameter.

Jose, maybe you know why it does not work?


' ========================================================================================
' Maps UTF-8 string to Unicode (UTF-16)
' ========================================================================================
FUNCTION Utf8ToUnicode(BYREF sUtf8 AS STRING) AS STRING
   dim sUnicode AS STRING
   dim dwLen    as long

   dwLen =  MultiByteToWideChar(CP_UTF8, _                 'Set to UTF8
                       0,                       _          'Conversion type
                       cast(LPCSTR, STRPTR(sUtf8)), _      'UTF8 string to convert
                       LEN(sUtf8), _                       'Length of UTF8 string
                       0,  _
                       0)   
   sUnicode = string(dwLen * 2, 0)

' The conversion below returns 0 so need to use GetLastError
print   MultiByteToWideChar(CP_UTF8, _                 'Set to UTF8
                       MB_COMPOSITE, _          'Conversion type
                       cast(LPCSTR, STRPTR(sUtf8)), _     'UTF8 string to convert
                       LEN(sUtf8), _              'Lenght of UTF8 string
                       cast(LPWSTR, STRPTR(sUnicode)), _  'Unicode string
                       len(sUnicode))             'Lenght of Unicode buffer

print "GetLastError = "; GetLastError()   ' this is code 1004

   FUNCTION = sUnicode

END FUNCTION


Why don't you use AfxUcode? Even if your function worked, the returned value will be converted by FB to ansi automatically, since the return type is AS STRING.

If you don't want to use AfxUCode for some reason, you need to use and return a WSTRING pointer. First allocate a buffer and cast it to a WSTRING pointer, pass it to MutiByteToWideChar, return it as the result of the function and later deallocate the memory.

Also don't use MB_COMPOSITE.

To be honest, I thought I had tried AfxUcode and it still failed.... I will try again. I believe I used something like:

PUT #f,, **AfxUcode(*psz, CP_UTF8)

Well, it will not be converted to ansi because the returned variable is a string, not a wstring.

Try this function:


' ========================================================================================
PRIVATE FUNCTION AfxUcode2 (BYREF ansiStr AS CONST STRING) AS STRING
   DIM dwLen AS DWORD = MultiByteToWideChar(CP_UTF8, 0, STRPTR(ansiStr), LEN(ansiStr), NULL, 0)
   IF dwLen THEN
      DIM s AS STRING = SPACE(dwLen * 2)
      dwLen = MultiByteToWideChar(CP_UTF8, 0, STRPTR(ansiStr), LEN(ansiStr), CAST(WSTRING PTR, STRPTR(s)), dwLen * 2)
      IF dwLen THEN RETURN s
   END IF
END FUNCTION
' ========================================================================================


This works:


               ' convert utf8 to utf16
               dim cws as CWSTR = AfxUcode(*psz, CP_UTF8)
               dim as byte ptr lpBuffer = Allocate(LEN(cws))
               lpBuffer = cws
               put #f, , lpBuffer[0], LEN(cws) * 2
               Deallocate(lpBuffer)


Thanks Jose, your function worked perfectly :)

Here is the code I am using when saving the UTF-16 BOM encoded file.

         case FILE_ENCODING_UTF16_BOM
            ' Output the BOM first
            put #f, , chr(&HFF, &HFE)
            if sciCodePage = SC_CP_UTF8 THEN   
               ' convert utf8 to utf16
               put #f, , Utf8ToUnicode(*psz)
            else
               ' need to convert ansi to unicode
               put #f, , WStr(*psz)
            end if   


New EXE's uploaded to GitHub. You will need to download the English.lang file as well because new top menu option "File Encoding" added under the Edit menu.

WinFBE will attempt to determine the type of file being loaded. You will notice that files with words such as Jose will load as UTF-8 because of the é character.

You can change file encoding either by clicking on the encoding label in the status bar or by selecting the encoding from the "Edit", "File Encoding" option.

Please let me know if you run into any problems.

Quote
dim as byte ptr lpBuffer = Allocate(LEN(cws))
lpBuffer = cws
put #f, , lpBuffer[0], LEN(cws) * 2
Deallocate(lpBuffer)

I don't see the need to allocate and deallocate lpBuffer, since it is simply a matter of casting, i.e. lpBuffer = cws does not copy the data, but it simply does cast(ANY PTR, m_pBuffer).


dim as byte ptr lpBuffer = cws
put #f, , lpBuffer[0], LEN(cws) * 2


Maybe this will also work:


put #f, , cast(BYTE PTR, cws.m_pBuffer)[0], LEN(cws) * 2


Thanks Jose, I ended up using the function that you posted for the sake of simplicity. I have posted the new EXE's. I will now to see how ganlinlao makes out with the new code and functionality.

Above all, don't use


dim as byte ptr lpBuffer = Allocate(LEN(cws))
lpBuffer = cws
put #f, , lpBuffer[0], LEN(cws) * 2
Deallocate(lpBuffer)


lpBuffer = cws will assign a pointer to the CWSTR buffer to lpBuffer. Therefore, Deallocate(lpBuffer) will try to deallocate the CWSTR buffer and the buffer allocated with Allocate won't be deallocated.

Wow, I'm glad to see the code changes, the status bar of the ANSI and UTF8, Unicode can switch freely, and save the corresponding code file without any problems.
But the GetFileToString function has a problem. If an existing Unicode text file cannot be opened with a string variable, you must use Wstring or wtring ptr.
     if an unicode file：
         dim wsText as wstring*65536  'or  dim wsText as cwstr,  can not use aniStr as string
         if Open( wszFilename for input encoding "utf16" As #f ) = 0  then
               Get #f, , wsText
        end if
       
Thanks
ganlinlao

Hi ganlinlao, thanks for the feedback. Here are the changes I am making to the code. If you can copy them into your code and test then that would be great. I will not be able to post new EXE's for a few more hours yet.
New version of UnicodeToUtf8 function:


' ========================================================================================
' Maps Unicode character string to a UTF-8 string.
' ========================================================================================
FUNCTION UnicodeToUtf8(byval pswzUnicode as wstring ptr) AS STRING
dim sUtf8 AS STRING

'Maps Unicode character string to a UTF-8 string.
sUtf8 = string(LEN(*pswzUnicode), 0)
WideCharToMultiByte(CP_UTF8, _                 'Set to UTF-8
                     0, _                       'Conversion type
                     cast(LPCWSTR, pswzUnicode), _  'Unicode string to convert
                     LEN(*pswzUnicode), _       'Length of Unicode string
                     cast(LPSTR, STRPTR(sUtf8)), _     'UTF-8 string
                     LEN(sUtf8), _              'Length of UTF-8 buffer
                     BYVAL 0, _                 'Invalid character replacement
                     BYVAL 0)                   'Replacement was used flag
FUNCTION = sUtf8

END FUNCTION


Replace this portion of GetFileToString (around line 300 of modRoutines.inc)


      case FILE_ENCODING_UTF16_BOM
         ' Convert to UTF8 so it can display in the editor
         ' Need to pass a WSTRING pointer to the conversion function.
         txtBuffer = UnicodeToUtf8( cast(WSTRING ptr, strptr(ansiStr)) )


Basically, I am mapping the raw binary received from loading the file to a WSTRING PTR. I am then feeding that pointer to the new UnicodeToUtf8.

Thanks,
Paul

I have also added code that sets the buffer to dirty whenever a user changes from one file encoding to another. This ensures that the file will be saved with the new encoding.

New source files and EXE's uploaded to GitHub.

When you will finnish playing with utf8, remember to change my name from JosÃÆ'Ã,© to Jose :)

Alternatively, you can use Josep (it is my name in Valencian, my mother language).

Updated español.lang file.

Quote from: Jose Roca on April 24, 2017, 05:58:28 PM
When you will finnish playing with utf8, remember to change my name from JosÃÆ'Ã,© to Jose :)

Alternatively, you can use Josep (it is my name in Valencian, my mother language).

Actually, looks like I had a logic error in the file loading code. If the text file was UTF-8, I was (for whatever stupid reason) running it through AnsiToUtf8 conversion function. I have fixed that now so your name should now appear correctly.  :)

hi,TechSupport

thanks
ganlinlao

Quote from: TechSupport on April 24, 2017, 09:55:56 PM

Quote from: Jose Roca on April 24, 2017, 05:58:28 PM
When you will finnish playing with utf8, remember to change my name from JosÃÆ'Ã,© to Jose :)

Alternatively, you can use Josep (it is my name in Valencian, my mother language).

Actually, looks like I had a logic error in the file loading code. If the text file was UTF-8, I was (for whatever stupid reason) running it through AnsiToUtf8 conversion function. I have fixed that now so your name should now appear correctly.  :)

Nope. And I can't use the editor because as soon as it finds an accented letter it switches to UTF-8.

This code is not correct (many European languages use 8 bit character sets) and is causing that the editor is wrongly identifying all my ansi files as UTF-8. It is also slowing the load of the file.

If it has no BOM, simply load it as ansi.


      ' Could be ANSI or UTF8(no BOM)
      ' Test the string to be ANSI by checking that BIT7 of each character is unset
      dim as Boolean IsAnsi = true   ' default that this is an ansi file
      for i as long = 0 to len(ansiStr) - 1
         if bit(ansiStr[i], 7) THEN
            IsAnsi = false: exit for
         END IF
      NEXT


Thanks Jose, I will change the code and go with the "If it has no BOM, simply load it as ansi".
I will upload the release this evening.

Hi Jose,

Attached is the WinFBE32 exe binary. I won't be able to compile the 64 bit until I get home later today.
I removed everything related to the UTF-8 encoding (if no BOM is present).

(Edit: All files 32/64 bit uploaded to GitHub and v1.3.2 released)