Firefly fails handling really large strings?

[John Waalkes]

I'm working on a program that expunges "bad" installer strings from a Window.reg file. This file is over 300 meg in size. I also have shorter versions I use for debug. These are a meg or less in size, and they work just fine.

Using FF, the large file fails every time with ztrace showing the correct size, and then another ztrace showing the first 40 bytes of OutStr$

Stripping out the FF specific stuff and running it in PB10 works fine for any size file.

Do I need to break the string down into smaller chunks?


Thanks,

John



Below is the output from ztrace. My comments are preceeded with a "..."

OutStr$ len =  313383428

...Makes first cut just fine, 100 Meg

OutStr$ len =  203255540
outstr$ = [

...Next time through the size is reported okay, but OutStr$ has just a "[" in it.

OutStr$ len =  203255540

...Check the size of OutStr$ a second time to see if it really is the same size. It is.

...Then exits the routine thinking that OutStr$ is null.

OutStr$ = ""

[James Padgett]

No code, = no way to see what is going on ...

[Paul Squires]

Is it zTrace that is failing with the very large string?

[James Padgett]

Set ztrace to write to a file .. ?

[Theo Gottwald]

String handling is done using the OS String functions. I believe that they need continouse memory to allocate large strings. If you have already allocated 300 MB large string and you allocate anotehr one in another application, this may fail doe to memory fragmentation. It has nothing to do with FF. Just restart your Computer and try againm but notice that after you use your computer for a while, it may fail again. Because the memory gets fragmented.

You can test this yourself, with a very simple PB Program that just tries to allocate several strings of 500 MB.
And see when you get an error.

[John Waalkes]

Is it zTrace that is failing with the very large string?

The example that I'm including with this post has zTrace turned off. It still failed.

This example was first done in PB10, where it ran fine and exited without any problems. It didn't find any "illegal" installation strings, but it shouldn't have, and that is beside the point. But, for the record, doing a file compare on the input file, "Win7.txt" (what I renamed my windows .reg file to), and the output file "Rockwell.xxx" produced no differences.

To make this program work, you will need to export your windows registry to a .reg file and relocate it to C:\ and rename it to "Win7.txt". Running the PB10 program will take about 10 - 15 minutes depending on what you have installed on your PC.

What the program does is to strip out any "illegal" (in Rockwell's eyes) installation strings and store them into "NonRockwell.xxx". The plan being to recreate the registry without the "illegal" installation keys so that the Rockwell installer would work. It turns out that there weren't any "illegal" installation keys (defined as any installation key that wasn't exactly 32 Hex characters long) anyways. So the program was for nought.

The FF example uses the PB code with an absolute minimum of changes to get it to work in FF. And, it ain't pretty (but it has also already outlived its usefulness).

And while the wisdom of diddling with 300+ meg strings can be discussed all day, it still shouldn't fail...


Thanks for taking a look at it!

[John Waalkes]

It has nothing to do with FF. Just restart your Computer and try againm but notice that after you use your computer for a while, it may fail again. Because the memory gets fragmented.

You can test this yourself, with a very simple PB Program that just tries to allocate several strings of 500 MB.
And see when you get an error.

Au contraire!

Since it works fine in straight up PB, and fails once the same program is put in a FF wrapper, I would say that it has everything to do with FF.

I followed up on my original post with a couple of sample programs, which you can try out if you like. I didn't include my .reg file since it is the property of General Motors, and they would frown on that kind of behavior.

But your point is well taken, I tried out your recommendation by running the PB version, the FF version, and then the PB version without rebooting (I'll do that next, but what can you say about a program that needs to be rebooted to run correctly?)

So to be clear, before the reboot, I got a good run with the PB version, failed with the FF version, and then another good run with the PB version. Not a memory problem that I can see.

Okay, so I rebooted the PC and tried the FF version and ended up with the same results.


Thanks for your input, and it was worth a try...

[Paul Squires]

Hi John,

Interesting problem. It looks like your PB version is running under PBCC, right? FF is using PBWIN. My initial reaction is that it could be a FF GUI problem. I expect it takes many minutes for your processing to occur and maybe there is something in the handling of FF's GUI and message pump that triggers an invalid pointer reference during that processing. I don't know exactly what condition could trigger something like that...maybe the window gets focus and under the strain of processing the file it causes a pointer not to be correct.... really don't know at this point. I wonder if you put your file processing in a separate thread would that help? This could very well be a FF problem - not related to the file processing or string handling, but more geared towards a corruption of internal FF pointers for the handling of the GUI.

[John Waalkes]

Thanks Paul,

First off, sorry about being away for so long, last week was a bugger at work (55+ hours) and yesterday I took the front diff out of my Jeep just to find that it wasn't the problem. I feel like an old man today... 

I'm using PBWin, although I do all of my GUI work in FF. And FWIW, I used PBWin 10 with FF 3.5

So how do I put my file routine in a separate thread? Could you post an example please?


Thanks,

John