P3D v5 Unhandled Exceptions with TFE machine learning

[arno]

Hi,

Got some performance figures here. I ran a test on a LOD11 area. The script splits it into LOD13 cells, so there are 16 raster features being processed by the TFC. Processing this area took 485 seconds and 6612 polygons where detected in the area.

By comparison on old TFC file that does not use machine learning and object based classification took only 50 seconds to process the same area, detecting 4389 polygons.

I had a look with the profiler, it seems the multi resolution segmentation steps takes most of the time. So I guess I have to see if I can optimize that step somehow further.

 

[Clutch Cargo]

My new findings - Using the video sample again, I ran a couple more tests. At tile size at 256 and Scale set at 200, process time 23minutes but zero detection?! Again the preview screen looked great. I then ran a second test, changing the scale to 300. This time it took 26 minutes and also zero detection?

I have emailed you the link where you can download the exact image I have been testing with and also samples for that image. The image is a little over 1GB in size. Its an LOD10 image at .60cm resolution covering about 40 square Kilometers. 15 samples included for that image. Still just mainly one sample image but more there if you wish. To process I was using the scenProc script I posted above in this thread. Using QGIS to observe the results.

 

[arno]

Thanks, I'll have a look at your data tonight.

 

[Clutch Cargo]

Correction on the image I sent you. It is not 30 square Kilometers but around 9.6 km.

 

[arno]

Hi,

I had a look to see if I could improve the performance of the multi resolution segmentation step, but any attempts I made until now to optimize things only made it slower .

I send you a request to get access to the Google drive for the files you send me to check why the detection works so different in the editor compared to the script. Did you receive that?

 

[Clutch Cargo]

I did not? I sent you an invitation for that folder, ha! I have sent you the invitation link again. Also chk your PM if that is faster.

 

[arno]

Hi,

I had a look at your files. The samples have 4 bands (RGB and NIR), while the big image you feed the script has only 3 bands (RGB). That is why you get different detection results. The machine learning has been trained on using the NIR data as well and since your imagery has none it will not find many areas that match. So you really need to make sure that the samples and the imagery you are processing are the same.

 

[Clutch Cargo]

How can that be? The samples were taken from that main image? Naturally I will take another look, but that has me confused, ha!

Update - Well, that was it... my mistake. I guess after testing so many variations of 3-band/4-band bmps and 3-band/4-band tiffs I inserted the wrong original 4-band image. You can bet I will not forget to check this in the future. You have seared this into my brain. But we're cookin' now! Saw my first results. Looking very good. Issue now for me is processing time.

Using the straightforward NVDI string I can process 25 images (25 LOD 10s like the sample main image I sent you), in 7 1/2 minutes. With the veg machine learning method it would take 12.50 hours. I have to decide if it better to use the faster method and then manually cleanup the results. Lot's of trial tests and tweaking still to be had but this is a huge step forward. But in all, this is such a game changer. Can you imagine if we had this say 5-years ago before MS2020?

Ok, back to testing...

 

[MyPC8MyBrain]

Issue now for me is processing time

are you processing workload with your CPU or GPU?

 

[arno]

Hi,

Using the straightforward NVDI string I can process 25 images (25 LOD 10s like the sample main image I sent you), in 7 1/2 minutes. With the veg machine learning method it would take 12.50 hours. I have to decide if it better to use the faster method and then manually cleanup the results. Lot's of trial tests and tweaking still to be had but this is a huge step forward. But in all, this is such a game changer. Can you imagine if we had this say 5-years ago before MS2020?

From the tests I did I do indeed see quite a big performance difference (about a factor 6 on my machine). Most of this processing time is in the multi resolution segmentation step. I am not sure how much I can improve that performance, but it might indeed turn out to be a trade of between processing time and quality.

 

[arno]

are you processing workload with your CPU or GPU?

scenProc does most of the processing on the CPU.

 

[MyPC8MyBrain]

scenProc does most of the processing on the CPU.

i am aware,
is it possible to add a "process with gpu" switch?
shouldn't he still be able to assign the exe process to run with gpu to speed things along?
machine learning load will benefit substantially running with gpu then with cpu
(calculating with DDR6 is way faster and the more gpu's the faster it will learn,
try adding all exe in scenProc dir to run with NVIDIA)

 

[arno]

Well, most time is not spend in the machine learning bit, but in the object segmentation phase before. So I'm not sure how much performance would increase.

And you can't just transfer the exe to the gpu as far as I know. That's completely different programming. The texture filter logic could benefit from that in theory.

 

[Clutch Cargo]

If Segmentation is the time killer, is there another step that could replace it yet still use the SVM step? Still gonna try the example in the manual as well.

"The texture filter logic could benefit from that in theory." Ok, that's your next project.. HA!

 

[arno]

You could try the K-means step instead of the multi resolution segmentation, but in general quality is much less.

 

[MyPC8MyBrain]

So I'm not sure how much performance would increase

I'm sure the current samples tested are significantly smaller then a full scale project requirements,
(not less important for larger scale projects GDDR beside being faster can be scaled while DDR is somewhat capped/bound by one cpu)
the combination of memory bandwidth, memory bus, and memory speed difference between GDDR6 & DDR4 should be roughly x 4 the performance speed (its the reason miners flock to GDDR),

 

[arno]

I'm sure the current samples tested are significantly smaller then a full scale project requirements,
(not less important for larger scale projects GDDR beside being faster can be scaled while DDR is somewhat capped/bound by one cpu)
the combination of memory bandwidth, memory bus, and memory speed difference between GDDR6 & DDR4 should be roughly x 4 the performance speed (its the reason miners flock to GDDR),

Might be, but it is an illusion to speed things up four times. You get the added work that scenProc has to transfer all the data to the GPU first, which does not happen now. And it would only benefit to certain parts of the texture filter chain.

I had a look in the past, the OpenCV libraries used allow parts of the calculations to be done on the GPU. But often that is for NVidia GPU only, so that makes it not applicable to all users.

 

[arno]

Hi,

I did some more performance testing and also made some optimizations that should speed things up. In the smaller tests that I use for developing they speed things up quite a lot, but in a bigger scale test the difference is smaller. That also gives me the impression that performance is determined by multiple things.

Let me try to explain that a bit more. In my complex project I am processing an entire state. The feature detection is done per LOD11 tile. So that means the script gets an input image for the entire LOD11 tile, which is 8192x8192 pixels in size. Before calling the FeatureDetect step there is first a SplitGrid step that divides the data into 16 cells. So the feature detection is ran on 16 2048x2048 images. Then when we come to the multi resolution segmentation step it has a maximum tile size attribute as well. I had this set to 256 initially, which means that the 2048x2048 image is divided into 64 256x256 segments again and these segments are being processed in parallel for the object segmentation. I did another test where I changed this tile size to 512, so that there are only 16 512x512 segments to run the object segmentation on. On my machine that gave slightly better performance, so although the object segmentation might need more time because it has more data to process, the fact that there are less calculations in parallel (less threads) still speed things up. I guess this also depends on the number of cores the CPU has, etc.

So the final performance depends on quite a lot of things. @Clutch Cargo what kind of image sizes, split grid sizes and multi resolution segmentation tile sizes are you using in your project now?

 

[Clutch Cargo]

The image you downloaded from me a is a perfect example of what I use for processing. It is an LOD10 image, .60cm in resolution and the size is 21680 x 16260 px. I believe that is around 10 square Kilometers. I typically work on 25 of those at one time. Depending on my scripts I might process one at a time or group them to speed up processing time. For that Image my settings are:

I run a 1st scenProc script that I use for initial vegetation detection. It is the simple script I posted above in this thread. I use SplitGriod|LOD15| so I can use DONTPROCESSHOLES which sped up processing time, at least it did dramatically when I was using the NDVI method shown in manual. But now using "machine", it takes on average 23 minutes to process a single image. I did test processing five images at one time. It took around 45mins to complete them all at one time. So I am looking at 4 hours to complete a group of 25 and the CPU is running at 80-90% most of the time for this test. I have 64GB of RAM and the process only seemed to use about 50% of that most of the time.

The multi resolution segmentation tile size (Maximum Tile Size) was set at 256 and I set the scale at 300.

Not sure if an issue during processing, but I was using more 1200x1200px examples than you use as I need to capture certain sports fields, golf courses, etc. I am using in the range of 20 samples if that is of any importance. It takes 20-30 seconds to move from sample to sample while in the TFE to preview their outputs or add sample points.

On a different note, I got a new UE (haven't had one in awhile):
UE7 – 01/20/2023 Was changing script from video example to the example in the Manual. That is to say I left all samples intact and was deleting steps and adding new steps to re-create the manual's example. When I connected the lines back together I got an UE when I clicked on the SVM step or any step to the right of it. The goal was I was hoping I could leave all my sample images and sample points intact when I created a new script?

I then decided to start fresh, new script (again using the Manual's example, reload all the image samples and recreate the sample points. FYI - I noticed when I clicked on new everything is deleted however the preview screen still shows what was left from the previous TF2. It does not go blank (refresh?).

By reconstructing the script from scratch, I did not get the UE this time. Used sames settings as above but only one example... processing time still took 23 minutes

 

[arno]

Hi,

Since you are processing LOD10, where I was testing with LOD11 images, I think the performance we see is similar. Yours is even a bit better. On my laptop the LOD11 area takes around 7 minutes. Your imagery has higher resolution (so more pixels), about a factor 1.3 and you process 4 times more data due to the LOD10. So I would estimate that takes 36 minutes based on my performance. Your 23 minutes is even better than that, I guess you might have a faster CPU than my laptop has.

With a split grid at LOD13/AGN you would be processing the images as 2710x2032 pixel tiles in the feature detection. With a maximum tile size in the multi resolution segmentation of 256 it means you have 11x8 tiles to process, so there are 88 parallel segmentations done. You might want to test if increasing the maximum tile size to 512 helps. I have seen that too many parallel calculations sometimes slows things down as the threads have to wait for CPU time. I don't know how many cores your CPU has, but try to aim a number of segmentation tiles that is near the number of cores you have.

I think I have seen that UE before, it looks like the data SVM training data might not have cleared correctly. Let me see if I can reproduce that.