PlanetSquires Forums

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3

Author Topic: WinFBE Suite 1.9.7 (November 2, 2019)  (Read 931 times)

Paul Squires

  • Administrator
  • Guru Member
  • *****
  • Posts: 8913
  • Windows 10
    • PlanetSquires Software
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #15 on: November 05, 2019, 12:08:48 PM »

I mentioned in a previous response that the Listview is limited by system memory. I have implemented this control as a virtual listview in that the text data is NOT stored in the listview itself (the listview is created with the LVS_OWNERDATA style). It is stored in data collections within the listview classes that I have written for the visual designer code generation. When text is need to be displayed on the screen, the listview asks for those particular items and/or subitems via a Windows api notification LVN_GETDISPINFO. I return the pointers to the data and then the listview displays it. Also, the listview uses CustomDraw (responds to the NM_CUSTOMDRAW notification) in order to handle the coloring of the foreground and background of each item/subitem.

The vlist code that Borje posted on the PB Forums and from which I adapted for the corresponding Firefly custom control, uses a similar approach. The data is held in an array(s) and during WM_PAINT, the necessary rows to show on the screen are requested from the array and displayed. I expect that you'd never be able to fit the millions of data lines into those arrays either.

Hope that helps to explain it a bit better.
Logged
Paul Squires
PlanetSquires Software
WinFBE Editor and Visual Designer

SeaVipe

  • Senior Member
  • ***
  • Posts: 229
  • Windows 10
WinFBE Suite 1.9.7 - NEW Down Arrow Behaviour
« Reply #16 on: November 05, 2019, 12:27:40 PM »

Hi Paul, click on the Button Bar "New" down arrow will crash WinFBE64.exe but not WinFBE32.exe. I've gone back to 1.9.0 and tested all releases through to 1.9.7 but only WinFBE 1.9.7 exhibits this behaviour.
« Last Edit: November 05, 2019, 01:30:24 PM by SeaVipe »
Logged
Clive Richey
There is nothing government can give you that it hasn't already taken from you in the first place. Winston Churchill

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #17 on: November 05, 2019, 01:26:53 PM »

Quote
It is stored in data collections within the listview classes that I have written for the visual designer code generation

Perhaps you could explain with an example? with comments/explanation.I used Borje's vlist, in December, and it adequately handles 2.5 million lines of my data. I created a list, iirc, and then the list box sort of scrolled up and down that list. (Don't you love my mastery of technical terms!) Now, if you have included the vlist algorithm, within listview, then where/how do I refer to the list of data that I have read in. I'll get there in the end, but I think you are maybe thinking I have more background in windows programming than I have.

so, the following is incorrect?
Code: [Select]
open filename for input as #fnum
         while not eof (fnum)
         line input #fnum,a
         frmmain.text2.Text = str(j)
         frmmain.ListView1.items.add( a)
         frmmain.ListView1.item(j).subitems.add ("")
         frmmain.ListView1.item(j).subitems.add ("")
         frmmain.ListView1.item(j).subitems.add ("")
         frmmain.ListView1.item(j).subitems.add ("")
         frmmain.ListView1.item(j).subitems.add ("")
         frmmain.ListView1.item(j).subitems.add ("")
          frmmain.ListView1.item(j).subitems.add ("")
          frmmain.ListView1.item(j).subitems.add ("")
          frmmain.ListView1.item(j).subitems.add ("")
          frmmain.ListView1.item(j).subitems.add ("")
          frmMain.ListView1.SelectedIndex = 0
       
          j=j+1
         wend
             close #fnum

for wherever j is a large value, or incorrect in general? It is populating the listview along the lines of your example. It is slow and works for up to about 600000 lines of data, (as I said before) and most likely for larger files as well. When I used vlist with firefly, (I can check how I did it) I believe I read the file into a single array (i.e copied the file to memory) and processed it there, instead of reading and processing line by line. If I do a similar thing again, are you saying that I can dispense with my vlist routine if I use list view? 

As I mentioned, a commented example would really help.
« Last Edit: November 05, 2019, 01:30:17 PM by raymw »
Logged

Paul Squires

  • Administrator
  • Guru Member
  • *****
  • Posts: 8913
  • Windows 10
    • PlanetSquires Software
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #18 on: November 05, 2019, 02:06:21 PM »

Hi Ray,

Here you go. I created this example based on the code that you posted. Ask as many questions as you need until you understand.

Code: [Select]
''
''
Function frmMain_Initialize( ByRef sender As wfxForm, ByRef e As EventArgs ) As LRESULT
   
   dim as long j, i
   dim as string a
   
   ' Ray, it looks like you want to have 11 columns of data total. I say this
   ' because in your example you are adding many subitems.
   ' You must define your columns before adding the item data. You can
   ' then choose to show or hide those column headers based on the
   ' ListView's HeaderStyle property (which you can set via code or via
   ' the visual desgner's propertylist).
   for i = 0 to 10
      frmMain.ListView1.Columns.Add( "SubItem" & i, 100, TextAlignment.Left )
   next
   
   ' Hide the column headers
   'frmMain.ListView1.HeaderStyle = ColumnHeaderStyle.None
   
   ' Show the column headers
   frmMain.ListView1.HeaderStyle = ColumnHeaderStyle.Clickable
   
   ' Open a test file and display it in the ListView line-by-line
   ' We will use a copy of Jose's AfxWin.inc file as a test file.
   dim as string filename = "AfxWin.inc"
   dim as long fnum = freefile
   
   open filename for input as #fnum
   while not eof (fnum)
      line input #fnum, a
     
      ' Add a new item to the ListView. The items.add call returns
      ' the newly added index value into the variable i. The newly
      ' created item automatically creates a subitem 0. Basically,
      ' the item and subitem(0) are the same thing and they can be
      ' referenced interchanably.
      j = frmmain.ListView1.items.add(a)   ' this creates subitem0
     
      ' Display in the TextBox the value of the newly created index.
      frmmain.text2.Text = str(j)
     
      ' Now add any subitems for the newly created item
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem1
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem2
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem3
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem4
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem5
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem6
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem7
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem8
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem9
      frmmain.ListView1.item(j).subitems.add ("")   ' subitem10
     
   wend

   ' Select the first line in the ListView
   frmMain.ListView1.SelectedIndex = 0

   close #fnum
               
   Function = 0
End Function

Logged
Paul Squires
PlanetSquires Software
WinFBE Editor and Visual Designer

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #19 on: November 05, 2019, 05:44:55 PM »

Thanks Paul, I'll try it out dreckly, and let you know.
Logged

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #20 on: November 05, 2019, 06:23:00 PM »

Hi Paul, thanks again.
Here is what I had to initialise the listview
Code: [Select]
Function frmmain_Initialize( ByRef sender As wfxForm, ByRef e As EventArgs ) As LRESULT
   'set up list view
  frmMain.ListView1.Columns.Add( "G-code", 400, TextAlignment.Left )
frmMain.ListView1.Columns.Add( "G", 60, TextAlignment.Left )
   frmMain.ListView1.Columns.Add( "X", 80, TextAlignment.Left )
   frmMain.ListView1.Columns.Add( "Y", 80, TextAlignment.Left )
   frmMain.ListView1.Columns.Add( "Z", 80, TextAlignment.Left )
frmMain.ListView1.Columns.Add( "A", 80, TextAlignment.Left )
   frmMain.ListView1.Columns.Add( "R", 80, TextAlignment.Left )
   frmMain.ListView1.Columns.Add( "I", 80, TextAlignment.Left )
     frmMain.ListView1.Columns.Add( "J", 80, TextAlignment.Left )
     frmMain.ListView1.Columns.Add( "F", 80, TextAlignment.Left )
frmMain.ListView1.Columns.Add( "Comment", 400, TextAlignment.Left )       '11 cols
   
   Function = 0
End Function

I can see nothing wrong there. The rest of the code, populating the listview,was/is similar to yours, I can see no significant difference in your method, and mine (seeing as mine was based on yours, too.)

In general, I think it is quicker to read in a file to an array in one burst, and then parse the entries in the array, and 'fill' the listview. I was reading a line and parsing each line as read. For smaller files, it is not noticeable, but for longer ones, I believe it is.

So, if you have included the vlist algorithm in listview, then I'm guessing it is not the same 'listview' which Jose said would be crazy to use on a mass of data.

But, I'd better run your last example code, on my data, and see for myself.

For quickly going through a million lines of data, then for the list box I made a column of adjacent small buttons (like a scroll bar) but buttons further away from the center pair, selected at plus or minus a multiple of ten items (and the listbox showed 20 or so items, so individua lines could be selected). e.g. centre pair of buttons would select +/- 10 from selected item, next outer pair select +/- 100, next pair select +/- 1000 - a sort of logarithmic scroll bar. A normal scroll bar is not very usable. Of course, line numbers and a search facility helps.
Logged

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #21 on: November 05, 2019, 07:10:57 PM »

here you go -

I will be able to speed it up, or at least give the impression of speeding it up. What happens, with a large file, for the example code as you have just posted, But with my large file, (I added a print to console to show something happening) the form does not appear until the listview is populated, and reading a line from file, processing that line takes time. List view works as I expected, and it will handle large files - so 'winner,winner, chicken dinner!'
Logged

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #22 on: November 05, 2019, 08:13:02 PM »

Finding that I had more or less understood listview, as far as i needed to know, I rerun yesterday's code, and waited (I should have timed it, but couldn't find a calendar). It eventually displayed the parsed g-code. Now to get into speeding it up. I would like to include my 'logarithmic scroll bar' so I think I may use two listviews - one for the file data , one column wide, which should get populated pretty quick as the file is read in, a line at a time, and one about ten columns for the parsed data, gleaned from the first listview. I may or may not tie the selected item rows together, it is handy to be able to return to a previously selected line- I can probably achieve that with colors. This of course, is only a part of what it will be doing. If i can get it to run as fast as, or faster than my firefly version, I'll migrate my firefly program, if not, I'll modify the firefly version.
Logged

Paul Squires

  • Administrator
  • Guru Member
  • *****
  • Posts: 8913
  • Windows 10
    • PlanetSquires Software
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #23 on: November 05, 2019, 08:31:30 PM »

Also realize that if you are outputting every line to the console as well then there will be a significant slow down. Printing to the console and scrolling the console is a very slow operation.
Logged
Paul Squires
PlanetSquires Software
WinFBE Editor and Visual Designer

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #24 on: November 06, 2019, 07:29:51 AM »

Thanks Paul, doing anything takes time. But somehow you need something to happen, else it can appear it is frozen/crashed. What I've done before is alter the file name text box to show loading + filename, and wait until it has loaded, and then change it back to filename. Testing the same two million or so lines of data, it is loaded and displayed in my firefly version in under five seconds, and then parsing to the separate lists takes about 20 seconds. I say parsing, but it does much more than that, finding max/min values, etc. I see no reason that using the same technique will not give similar timings using your listviews.

Editing individual values - will that be possible directly, or will it be necessary to use a separate textbox and do a copy/replace, as I've used in my firefly version using listboxes? I'm not sure if the longer entries ending in ellipsis can be expanded in situ, but not a major issue for me.

Thanks for your patience and expertise, it is much appreciated. 
edit to add  - just timed it. reading in the file into an array, less than 2seconds. Total time to read into and then get that array into a single column listview - straight copy, 8.93 minutes.  That, in Firefly takes 5 seconds, using vlist and a custom control. Plan B beckons.
« Last Edit: November 06, 2019, 06:45:39 PM by raymw »
Logged

raymw

  • Senior Member
  • ***
  • Posts: 355
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #25 on: November 06, 2019, 07:53:25 AM »

I find the width of listview as set in designer does not exactly tie with the width in pixels set in code. I'm guessing the width in designer includes the frame, whereas that in code is the usable width for data. I'm wondering, if not too difficult, if it would help for the cursor in designer to display it's location wrt the form?
edit to add - which has got me thinking wrt the grid. Does it 'steal' pixels from each item, or is it a sort of overlay on the whole tool. I imagine, that if a value in a sub item fills the area, that turning on the grid could cause ellipsis to display, if the grid steals pixels, but not necessarily so if it's an overlay.
« Last Edit: November 06, 2019, 08:33:47 AM by raymw »
Logged

Paul Squires

  • Administrator
  • Guru Member
  • *****
  • Posts: 8913
  • Windows 10
    • PlanetSquires Software
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #26 on: November 07, 2019, 12:01:05 PM »

Hi Ray,

I found the source of the slow down when adding bulk items to the ListView. If items are added during Initialize then it is very fast but if once the ListView is created (ie. the hWindow is valid) then the process slows down considerably. This is because I was making a call to ListView_SetItemCountEx after every Add if the hWindow existed. I was surprised of the huge performance effect this had.

I have added two new ListView methods:  BeginUpdate, and EndUpdate. If you bookend your bulk adding loop with these calls then the adding is extremely fast (less than 2 seconds for 500,000 items on my testing computer).

So use code like this:

Code: [Select]
   frmMain.ListView1.Items.Clear

   frmMain.ListView1.BeginUpdate

   dim i as Long
   for ii as long = 0 to 500000
      i = frmMain.ListView1.Items.Add( "Line " & ii )
      frmMain.ListView1.Item(i).SubItems.Add( "L" & ii & "Sub1" )
      frmMain.ListView1.Item(i).SubItems.Add( "L" & ii & "Sub2" )
   next

   frmMain.ListView1.EndUpdate

I will post an update to the Visual Designer tonight.
Logged
Paul Squires
PlanetSquires Software
WinFBE Editor and Visual Designer

SeaVipe

  • Senior Member
  • ***
  • Posts: 229
  • Windows 10
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #27 on: November 07, 2019, 12:55:10 PM »

Hi Paul, will BeginUpdate and EndUpdate be required or optional? I'm guessing this block of code will still work with the new bookends excluded, just slower:

Code: [Select]
   dim i as Long
   for ii as long = 0 to 500000
      i = frmMain.ListView1.Items.Add( "Line " & ii )
      frmMain.ListView1.Item(i).SubItems.Add( "L" & ii & "Sub1" )
      frmMain.ListView1.Item(i).SubItems.Add( "L" & ii & "Sub2" )
   next


My current ListView requirements are for list of less than 200 so I have no need for speed in this case.
Logged
Clive Richey
There is nothing government can give you that it hasn't already taken from you in the first place. Winston Churchill

Paul Squires

  • Administrator
  • Guru Member
  • *****
  • Posts: 8913
  • Windows 10
    • PlanetSquires Software
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #28 on: November 07, 2019, 02:20:47 PM »

Hi Clive, it is optional.
Logged
Paul Squires
PlanetSquires Software
WinFBE Editor and Visual Designer

SeaVipe

  • Senior Member
  • ***
  • Posts: 229
  • Windows 10
Re: WinFBE Suite 1.9.7 (November 2, 2019)
« Reply #29 on: November 07, 2019, 02:46:58 PM »

Perfect. Thanks, Paul.
When I port my big FF project over to WinFBE, the file sizes will be much larger but still probably less than 50K individual entries (xml file). Mostly this will be for diagnostics though (I think) I can see the advantage of having all the entries in a single scrollable ListView.
I'll do some testing on larger file importing once the new release is out. It involves reading in and parsing an xml file (which is where the bottleneck will occur). The parser is my own so I'll be on the lookout for some optimizing ideas then.
« Last Edit: November 07, 2019, 02:57:05 PM by SeaVipe »
Logged
Clive Richey
There is nothing government can give you that it hasn't already taken from you in the first place. Winston Churchill
Pages: 1 [2] 3