P3D v5 Run multiple instances of scenPROC on same PC?

[Clutch Cargo]

I'm conducting some processing tests with scenPROC v3.1 and of course that brings up a few questions on my observations.

1. Can you run multiple instances of scenPROC at the same time on one PC?

It appears you can as I am running a test right now but here is the issue. I am using the Texture Filter tool to detect vegetation. My test is to run 1 LOD13 image (call it test A) and then run a second LOD13 image (call this B), which is right next to image A. I had samples of image A create for the TFE to create a proper near infrared setting. I start up both instances of scenPROC that use identical scripts but referring to the different images, A & B.

scenPROC A finished its complete feature detection and completed processing in 3 minutes! Fantastic! But scenPROC B has been running now for 55 minutes and is still in the feature detection phase!?!? As you can see the processing difference it extreme. It does not appear to be frozen or crashes as I see it chugging along on my CPU and there is plenty of RAM to spare (using 6GB out of 32GB memory).

FYI - It has now been over an hour and still no results so I am going to stop it and try it again to see if there is any time difference.

Notes -
A) the images are the same size in coverage, and true not the exact image the geography is quite similar, farmlands, small towns.
B) The 1000x1000px sample images were taken from image A, while image B is relying on the settings based on the image A settings. Could that have some impact?

Why the extreme differences?

Question 2
I created my own little code to create scenPROC scripts to run individually or in a batch mode. While I am reading up on your batch method in the User's Manual I have never tried it. Do you think your method reduces processing time over mine? Only difference I see is my method leaves the scenPROC GUI in view while your batching works in the background.

 

[arno]

Hi,

The content of the image file processed can influence how long it takes. But this difference sounds too big. If you run only B is it quicker?

There should be no issue in running scenproc in parallel, it has been designed for that. I often run multiple instances at the same time. For feature detection sometimes 4 instances and for steps thst use less memory sometimes even more at once.

You can start scenproc with your own script indeed as well. The main difference with the batch runner is indeed the GUI, but I don't think that should matter too much for performance.

 

[Clutch Cargo]

Update - re-ran the test of image B but this time I used TFE with samples from image B and then created a custom TFC profile for that scenPROC script. Unfortunately, is seems to make no difference in processing time. I can only assum image B has so much more vegetation it just takes way long for that image. And granted is does has more but again the processing time differences seemed so extreme. So this brings me to question #3:

Question 3 - If I use the step (sorry, do not know the language yet for this step), "If trees are detected within a certain poly they are to be skipped". If I use that type of step, will scenPROC create this rectangle polys first and then remove them or will it just skip making all those veg polys found inside that poly? Thinking of time-saving methods.

 

[Clutch Cargo]

I have attached the two images for your impression. I believe the crops on the top rightside in Image B is causing the prolonged processing, hence question 3. As even a further update, for reference, Image A took 3-4 minutes to process (roughly 126,000 veg polys), Image B took 1hour 15 minutes to process with 266,000+ veg polys.

Image A
View attachment Image A.jpg

Image B

View attachment Image B.jpg

 

[MyPC8MyBrain]

the difference that jumps for me... image B has water features while image A doesn't,

 

[arno]

Hi,

It is quite likely that the crops influence this. They might cause a lot of pixels that have to be processed further, also depending on how complex your texture filter is.

Do you use SplitGrid in your script when running the DetectFeatures step? It might help to split your image into segments and process them separately instead of processing the whole image at once.

I am not sure what you mean with "If trees are detected within a certain poly they are to be skipped". Are you talking about the masking featue that is optional in the DetectFeatures step? That can be used to exclude certain areas from processing with a polygon.

 

[Clutch Cargo]

With the test above, I thought Image B would be covered with trees in the upper right corner. Guess what, there not trees there at all! That whole right side was filtered out pretty good.

Chris - good eye as I did not even think of the river.

I noticed the river which runs from N to S crosses several images. So I conducted a new time test with four tiles right above the previous A/B tiles. The tile are arranged in a simple 2 x 2 pattern:

C / D
E / F

Results:

Image_______Rectangular Veg Objects Created_________Processing time________Notes

C____________155,348____________________________________1 min 30 sec

D____________122,534____________________________________49 min_________________Has river running through it
D____________119,501____________________________________48 min_________________ 2nd run using a black mask over river

E_____________100,530_____________________________________1 min

F_____________255,000+ (from memory)__________________3 hours+______________Has river running through it and is still processing!

So I am thinking Chris is onto something. So I performed a new test on image D but I masked out the water with an Orange fill color. I also created the same image with a black fill color.
Sadly, neither masked seem to improve processing time. Could it just be the complexity of the river twisting and turning, snaking down the image? I am baffled!?!?

Also Arno, yes I always use the SplitGrid step mostly as SplitGrid|AGN|* but I have also tried SplitGrid|LOD13|*

"Are you talking about the masking feature that is optional in the DetectFeatures step? That can be used to exclude certain areas from processing with a polygon."

Not sure if it DetectFeaturtes as I am talking about removing veg based on a polygon, not an image. Looking thru the manual as I type to try this as my next test. It's basical remove any veg from inside a polygon that represents a "farm" or "crop"... you get the idea.

 

[arno]

Hi,

Maybe you can show a screenshot of how your texture filter looks and which steps it uses? That might help to give you performance advice.

In general most steps in the texture filter process all pixels, so their performance does not really depend on what is in the image. So if there is a river or not should not matter too much in general. Only when there is a lot of small pixels things like erode, dilate or vectorizing has a variable time. Especially when you return polygons with holes that can have an impact.

Also Arno, yes I always use the SplitGrid step mostly as SplitGrid|AGN|* but I have also tried SplitGrid|LOD13|*

That's good. For feature detection I think in general using AGN size tiles is a bit too small, I think bigger tiles will work better in that case as a balance between memory usage and performance.

Not sure if it DetectFeaturtes as I am talking about removing veg based on a polygon, not an image. Looking thru the manual as I type to try this as my next test. It's basical remove any veg from inside a polygon that represents a "farm" or "crop"... you get the idea.

Removing polygons after they have been detected already would not really help in improving the performance of course. The mask you can specify in the DetectFeatures step is used to select polygons that specify areas where no detection should be run. If you want to remove polygons afterwards that are inside others you can use the FilterFeatures step for that.

 

[arno]

Just had another idea, let me see if i can add some reporting of the time each step in the texture filter takes. That might give some insight. Just like at the end of the script it is reported how long each step took.

 

[Clutch Cargo]

oh.... that would be nice with that added timing feature. So I have spent the day with a very basic script to work on filtering. But to answer your question, attached is a screenshot of my Texture Filter. As you can see, it is an exact duplicate of your example in the manual. This is what I am using in my test.

I did adjust the Threshold a tad to see different results. It works good with the exception of taking extreme detection time on certain images. However, because it does work very fast on other images, I am conducting a test on one of those to try and filter out trees in cropland areas.

As you said, I have two ways to basically filter out unwanted attributes, Detect Features and Filter Features. So for this test I am only concerned about Filter Features.

To prepare for the test, in QGIS I created a shapelayer called 'CHICO_X20_landcover.shp'. In the creation of the layer I added the Field 'type' as a string and the Value called "crop". The polys follow the image precisely and cover over many crop areas. From there I created a very basic script. With this image it is very fast, 1 minute to process.

ImportGDAL|B:\CHICO\Section_161\Images\CHICO_X20.tif|*|AUTODETECT
ImportOGR|B:\CHICO\Section_161\CHICO_X20\Vegetation\CHICO_X20_landcover.shp|*|*|NOREPROJ

DetectFeatures|FTYPE="RASTER"|B:\Scripting\scenPROCv3\scriptsVEG\Veg Basic1.tfc|String;veg|tree|NONE
SplitGrid|AGN|*

# I have tried all three types of filters below, one at a time, with no success. Always get "Filtered out 0 features"
FilterFeatures|type="tree"|type="crop"|OVERLAP_BBOX|0
FilterFeatures|FTYPE="POLYGON" AND type="tree"|type="crop"|DISTANCE_FEATURE|10
FilterFeatures|type="tree"|type="crop"|DISTANCE_CENTER|10


ExportAGN|P3D v4|B:\CHICO\Section_161\CHICO_X20\Autogen\texture


FYI - You manual shows the bounding box filter as:
FilterFeatures|type="building"|type="vegetation"|OVERLAP_BBOX

However, I get an error in scenPROC saying it needs four arguments, so I have been adding a '|0' at the end.

Here is where I sit at the moment? Obviously my code is off but where?

 

[arno]

Hi,

Have you tried to out SplitGrid before the DetectFeatures step? I think that should speed things up quite a bit already.

The timing feature is almost done, I just need to test it on a bigger image. But that will take a few days, as we are away with the family now.

 

[Clutch Cargo]

Ah, no problem. On this script it is so fast does not seem to matter where I put SplitGrid. Any quick thoughts on the above as to why my FilterFeature steps do not work? Sounds like you are gone for the weekend so I will let you go,

 

[arno]

You create your trees with an attribute named veg, not named type. So type="tree" will not match any features.

 

[Clutch Cargo]

I did replace tree with veg and still did not get any filtering? Since I am not getting the "ah-ha" moment where this is all clicking together for me to understand, I will be as explicit as I can. First here is a screenshot showing the polys I created (in pink), the attribute table shows a the field I created called VegType and the value is "crop". The filename is called CHICO_X20_landcover.shp.

Next here is my script:

ImportGDAL|B:\CHICO\Section_161\Images\CHICO_X20.tif|*|AUTODETECT
ImportOGR|B:\CHICO\Section_161\CHICO_X20\Vegetation\CHICO_X20_landcover.shp|*|*|NOREPROJ

SplitGrid|AGN|*

DetectFeatures|FTYPE="RASTER"|B:\Scripting\scenPROCv3\scriptsVEG\Veg Basic1.tfc|String;veg|tree|NONE
FilterFeatures|FTYPE="POLYGON" AND type="veg"|type="crop"|OVERLAP_BBOX|20

CreateAGNRectVeg|FTYPE="POLYGON"|{970d2095-a188-4773-bdc8-838918e1ba67}
ExportAGN|P3D v4|B:\CHICO\Section_161\CHICO_X20\Autogen\texture


Using your manual example, I thought |String;veg|tree| is where the attribute scenProc created via the tfc file a string field called 'veg' and the value for that field is called "tree". Wrong? So what does 'tree' represent in that statement? Does the word 'type=' just a generic argument that scenPROC uses or what is required is the actual Field name I create, in this test I should be using VegType="crop" ? I have tried both ways with no success.

In addition, the |OVERLAP_BBOX|20 I am using 20 meters or since my polys are right on top of the trees I wish to filter out, I could use 1 or 0 as well? Just trying to dissect every word until I get it. Does one have to click 'save' every time I make an edit and run the script or not really required to test the changes?

 

[Clutch Cargo]

Ha, no sooner than posting the above I got the filtering to work.

I used the step:
FilterFeatures|FTYPE="POLYGON" AND veg="tree"|VegType="crop"|OVERLAP_BBOX|20

So trying to understand what I did, the word 'type=' is not what should be used. It is a generic "placeholder" for the field. So veg is the field and tree is the value?

Think I am about to get my "ah-ha" moment

 

[Clutch Cargo]

Just another update. So I moved SplitGrid before DetectFeatures and I read about and added the option DONTPROCESHOLES. It took around an hour to find out there is a typo in the manual on this option. That 1 extra "s" but found the issue in another thread. Now look at the results:

Image D former process time was 49min. Now 27 seconds!
Image F former process time was 3hours. Now 29 seconds!

Now were cookin'

 

[arno]

Hi,

Glad to read you got it working now. Yes, like I tried to indicate the problem was that you used the wrong name/value combination for the attribute. You can name the attribute whatever you want in the DetectFeatures step. type is commonly used in OSM data, but you can name it veg or whatever you want. As long as your filter matches that it will work fine.

I will fix the missing S in the DONTPROCESSHOLES attribute in the next release (that will break your scripts, but then at least it is consistent everywhere).

Although in general I think it is best to process holes as well, as else your detection might be less accurate in some areas.

 

[arno]

The timing feature is almost done, I just need to test it on a bigger image. But that will take a few days, as we are away with the family now.

I had another look at the timings. I start to doubt if it is really useful. In my representative script the step took 35 seconds for a 2x2 km area with a filter more complex than yours. But 33 seconds was spend loading the image data, and only 2 seconds on the actual filter. So the timing information does not help much to optimize.

 

[Clutch Cargo]

I tend to agree Arno, it may not give me any additional info as to why the detection processing took so long. Of course I say this after my timings dramatically reduced. I am about to test my same script again but remove the PROCESHOLES statement and check the time. If it creates long processing times again, would it be more informative to show how many holes were processed? Maybe not?

Just ran Image D again, the one that originally took 49mins. 15 minutes if I take out DONTPROCESHOLES, 24 seconds if I leave in DONTPROCESHOLES. Plus I had moved up the SplitGrid. So it definately has an impact. I really like processing holes as it adds some nice looks in forests where there might be a pasture or a recent cut for logging. I'll just have to decide when and where to use it or take it out for time sake if it is worth it.

 

[arno]

Hi,

The timings I mentioned in my previous post are with processing of holes. But I guess it also depends on the output of the texture filter how much holes there are. Only in areas with a lot of forest you should see many holes. For the test I run the vectorization of the raster took less than 0.1 second on the total 35 seconds. And it is this step where the processing of holes has an effect.

Are you sure your imagery has a NIR channel included, since you use the NDVI step? Else the output might also be incorrect, influencing the processing time.