P3D v5 Unhandled Exceptions with TFE machine learning

[arno]

What are you testing Chris? The script or in the texture filter editor?

 

[MyPC8MyBrain]

What are you testing

testing both (the 3% utilization was during running the script itself, though operantly with the wrong image),

did bit of research, not sure if you are utilizing OpenCL or OGR-GPU?
with OpenCL you can take advantage of the parallel processing capabilities of a GPU to speed up certain GDAL operations, such as raster processing and image manipulation (i think that is implemented).
with OGR-GPU already built on top of GDAL and support GPU acceleration, a GPU-accelerated library that extends GDAL to support vector data processing on the GPU.

 

[arno]

Hi Chris,

GDAL and OGR are mainly used for import/export in scenProc, so read from GIS data and writing to GIS data. That is in general not a task that can be done on a GPU. The internal processing is not done using these libraries. For the image processing OpenCV is used primarily.

As for your CPU usage, most steps in the scenProc script will run in parallel for the different gridcells there are. So it depends on how many cells you have in your area how many threads/cores are used. With the SplitGrid step you can control this in your script. There are a few exceptions, the steps that read data from file and write data to file in general don't run in parallel, because they would have to share access to the same file in that case. Another exception are the steps that work with raster data, these also don't run in parallel because that would require too much memory.

In the texture filter some steps do use multiple threads, because they are computation intensive. This is mianly for the object based steps, so calculating mean or std per object and the multi resolution segmentation. For the last one it is the maximum tile size that determines how big the segments are that are processed in parallel. If you leave it at the default of 4096 you get optimal quality, but in general it will not run in parallel. When I run tests here I see the CPU close to 100% all the time when the object segmentation is running.

Except maybe for one or two of the new steps, in general scenProc runs quite fast. Remember that we could process an entire state (OK not a huge one) in a few hours? And that was everything from feature detection, photoreal production and autogen creation. So I don't think there is a big performance problem and that everything should be changed.

 

[MyPC8MyBrain]

in that case i must be drawing conclusion from a flawed process here on my end, ill see if i can figure it out why it is not multi threading here,
do you see an obvious flaw with this script that could be crippling/limiting performance to a single core?

ImportGDAL|D:\work\sample-tiff\sample.tif|*|AUTODETECT
SplitGrid|LOD15|*
DetectFeatures|FTYPE="RASTER"|D:\work\sample-tiff\sample-tif2.tf2|String;VegType|tree|NONE|DONTPROCESSHOLES
MergeGrid
ExportOGR|VegType="tree"|ESRI Shapefile|D:\work\sample-tiff\Vegetation\sample-tif2.shp|tree

 

[arno]

If you sample is small it might be only one cell. You should see that in the split grid event log output.

But then still your texture filter should depending on the multi resolution segmentation maximum tile size value use multiple threads.

 

[MyPC8MyBrain]

ok figured it out, it was something with the tiff image not having coordinate and projection set, once i gave it fake coordinates it started working as expected,
in the first image below you can see how the process is running over one core, once tiff was fixed second image is the run start where instantly i could see different behavior,
the third image show the time its taken to complete, fourth image is showing script segmentation settings i run it with,
i don't think its bad performance or slow at all! (could be due to me processing the wrong image you both are testing with, the image i tested with attached )

 

[MyPC8MyBrain]

i run the test again only this time with 4096 grid, to my surprise that only taken extra 2 seconds from the 256 grid,
(if you look closely at dgpu utilization it appears that something did get executed on there, the spike in iGpu is because i tested in hybrid mode where there's a passthrough from the iGpu to the dGpu),

footnote: we could prob use a built-in function/button in texture editor to re-train the image instead of manually jumping from one point to another (for after changing parameter's)

 

[Clutch Cargo]

As Arno said, could be testing "apples-to-pears", ha! Chris if you want, I can link my test image in a PM to you if you want to try it? It's a large 1.3GB tiff image hence why it is not posted here.

Performed another test using this script with my test sample. I call it my "K-Means Test":

And I have to say the results were great. I'd say 90-95% accuracy. Again, 7.5 minutes to process. Now, what I see is that I had to add an extra sample on an exact sports field. The script just wasn't deleting veg there without it so maybe this script just needs more attention if it can't find a good match for sample points. I used 22 sample images. Here's a few screenshots of the results on that test image. Below showing how good it filtered out veg on the golf course.

This is where I had to add another sample image as both the baseball and football field had trees on the running track and the red portion of the baseball field.

Another great result below on a golf course. Lot's of trees. In fact, I saw a warning message in scenProc that I have never seen before:
"Warning - 37 cells contain more than 6000 vegetation objects, this can give issues in the sim" I'm aware of the potential issue, just did not know scenProc was so robust to report on it.

One area that did have a little tree overrun on a field and on the tennis courts. But if this is the only area I could see with an minor issue... I am sold on this method. My next test is to process five similar areas at once for process timing, then an entire Section (25 areas) to see how well the script holds up on filtering a larger 30 square mile area. Good times!

 

[arno]

footnote: we could prob use a built-in function/button in texture editor to re-train the image instead of manually jumping from one point to another (for after changing parameter's)

Not sure what you mean, if you change any of the parameters of the SVM step it will automatically retrain. If you change settings on the steps that are input to the SVM step dito.

 

[arno]

And I have to say the results were great. I'd say 90-95% accuracy. Again, 7.5 minutes to process. Now, what I see is that I had to add an extra sample on an exact sports field. The script just wasn't deleting veg there without it so maybe this script just needs more attention if it can't find a good match for sample points. I used 22 sample images. Here's a few screenshots of the results on that test image. Below showing how good it filtered out veg on the golf course.

That's my workflow as well. If I find out that a certain area does not work well I add a sample based on that area to ensure that training data for that specific problem is included. So sounds like a good workflow.

In fact, I saw a warning message in scenProc that I have never seen before:
"Warning - 37 cells contain more than 6000 vegetation objects, this can give issues in the sim" I'm aware of the potential issue, just did not know scenProc was so robust to report on it.

That's a feature I have recently added, it had been on the wishlist for quite a while already.

But your results look good!

 

[MyPC8MyBrain]

can link my test image in a PM to you if you want to try it?

yes, please do
the file i am working with for testing atm is only a few mb in size (its the tiff you posted on your first post on this thread),

Not sure what you mean, if you change any of the parameters of the SVM step it will automatically retrain

sounds like i did something i shouldn't,
after manipulating or swapping tiff around (same image just a name change etc.) scenProc didn't like running the script, it complained right at the beginning it needed more training,
didn't quit get what it was asking at first, i found that i needed to touch each point in the flow for something to happen,
even after i "retrained" i found that i had to save the changes before it would run without complaining,

 

[arno]

Yes, if you run the texture filter from the script and it is untrained you need to train it.

That's done by clicking on the svm step and then saving the filter again to store the training results.

 

[MyPC8MyBrain]

if you run the texture filter from the script and it is untrained you need to train it

it was initially trained and saved, i just swapped the tiff itself as i was trying to fix the projection issues,
kept the image name just coordinates changes in header and auxiliary files, but scenProc didn't like running it before i retrain and save again,
its ok not a big deal ill be more carful if i have to swap physicals files again in the future,

 

[Clutch Cargo]

Chris, PM me your email as I need that so you can d/l from my cloud.

 

[arno]

it was initially trained and saved, i just swapped the tiff itself as i was trying to fix the projection issues,
kept the image name just coordinates changes in header and auxiliary files, but scenProc didn't like running it before i retrain and save again,
its ok not a big deal ill be more carful if i have to swap physicals files again in the future,

You can't just swap images, the sample images are kept in a scenproc temp folder. So overwriting the file that you have loaded will not change the scenproc copy.

And once you remove a file the sample points are removed and thus the training is no longer valid.

 

[Clutch Cargo]

Chris, check your PM (you'll also get an email). Data provided will be the same I am using like in the images above.

Arno, we had a good stretch of no UE's but just got one. UE8 – 01/27/2023 I believe all I was doing was switching from one TF2 to another. I thought maybe the TF2 was corrupt now?, but I closed and reopened scenProc\TFE and opened the wanted TF2 file Ok and it worked ok.

 

[arno]

I'll see if I can reproduce that error. I guess something did not clear right on the new load.

 

[arno]

Hi,

I have implemented the panning and zooming in the texture filter steps picture box now, it will be in the next release.

I think I have also fixed the last reported UE in that build. Once the regressions tests have all ran, I will push a new build.

 

[MyPC8MyBrain]

run USAF_B2 sample, 23min to complete on my laptop,
for whatever reason the entire script run my cpu usage did not exceed 66% at best,
at the low end it was hovering at 46%, do you have core count hardcoded somewhere?
(in texture filter editor during texture learning cpu was pegged at 100% but not in scenProc script run)

same script 34min to complete on my desktop,

while training that huge 1GB tiff in texture filter editor i did get few UE's just as learning completed,
got the same on both systems i tested with "Array dimensions exceeded supported range." (that UE happened after i rebooted, changed to 256 segments,
total system ram usage was relatively low around maybe 7-12gb through the whole process, no ram shortage or reason to complain about memory),

i also noticed it seem to trigger the same UE repetitively on its own if texture filter editor window is open and in focus while scenProc is processing the script,

 

[MyPC8MyBrain]

@arno LOVE the new zoom in steps work area w00t
i think we need learning button as separate function, it cant do the auto re-learn process automatically it has to be manually triggered,
maybe a checkbox to "auto learn" similar to merge results for smaller files, but for dealing with larger files it has to be manually triggered otherwise the current logic makes it impossible to work with,
i am trying for fun run 4096 grid, i had it on 256 when the project loads after closing it (after initial learn and being saved) navigating from image input step one to multi res step makes me wait 20+ minutes while it tries to relearn the file or something happens that's way too long to be working with comfortably, I just wanted to change the segments in that step from 256 to 4096 i have to wait 20 min just to do edit the parameter,

managed to run 4096 grid rendered relearned and saved, that's after the step auto run 3 times needlessly
(plus 3 cold reboots from dark screen after all ram was consumed by the script in first few seconds, that's on me though i disabled pagefile with 64GB ram)
20 min wait to go from source image to multi res step,
20 min wait after changing parameters to 4096 to re-render the grid,
20 min wait to relearn the whole steps again all the way to output image,
if the learn was triggered manually from output image step it would potentially complete 60% faster
(that's before running the actual script)

below is about 30min into running the 4096 grid script, first few minutes scenProc pulled well over 64GB ram and cpu was around 40% usage,
10 min into the run ram was freed and cpu usage reduced to about 10% i am guessing this will be the case for the reminder of the run,
(btw texture filter editor window shown below is as wide as it will allow to expand on my screen)