ScenProc script suggestions

[crushnik]

I was hoping no one would see my mistake. I didn't catch it until I had posted and noticed there was no output file.

 

[crushnik]

Good news/bad news

Good news: I can do eight 3.75-minute ortho tiles at once. I found a new (and hidden) way to copy the file's path in Windows Explorer, which is 1,000 times easier than typing C:\blahblahblah out a bunch of times. Your code worked like a charm - and after just one run of GeoTiFF to INF, then setting CompressionQuality to 85, and I was way ahead of schedule and quite happy.

More good news: inexplicably, JRobinson's process somehow had the Mississippi River vector already showing through.

Bad news: it looked like Day 3 of rain for Noah and his Ark. Water showing through everywhere. No, water doesn't stick to hillsides in Missouri, and I didn't do a multisource INF or any watermask work for this area, so I'm stumped. And out of time.

In the last picture, you can see the end of the "waterworld" scenery I freshly turned out with GDAL, a tile of scenery I manually cranked out in QGIS in my tedious one-by-one process, and on the top of both pictures (the areas with plenty of trees) is stock P3D scenery.

 

[JRobinson]

I found a new (and hidden) way to copy the file's path in Windows Explorer

set path=C:\Users\Mary Beth\Downloads and "%path%\source1.jp2" "%path%\source2.jp2" ?

Bad news: it looked like Day 3 of rain for Noah and his Ark

It's the NIR channel.

 

[crushnik]

set path=C:\Users\Mary Beth\Downloads and "%path%\source1.jp2" "%path%\source2.jp2" ?

My discovery is probably new to no one but me, but I found that by holding the "Shift" key while right-clicking the mouse gives you a "Copy as Path" option. Your %path%\... shortcut looks even better, but probably wont work for me in this instance because the USGS filenames are long and complicated.

It's the NIR channel.

It was late for me and I had to get up very early the next morning, but an alpha channel seemed to be the likely culprit for me at the time. Now that I know it can extract watermask data from the NIR/alpha channel, is the GDALwarp process you recommended in Post #17 still my best option for burning in a watermask? It is a bit more work on the front-end but doing eight files at once in a process that is mostly stuff the computer does in the background is great.

Also, is there an efficient way to automatically adjust brightness, contrast, vibrancy/saturation outside of doing it for each TIFF file in a photo editing program? It would be nice if there was an external setting (the in-sim settings make the ortho conflict horribly with the sim's built-in textures and autogen) that I could adjust if necessary without having to redo thousands of square miles of TIFF files when I find a better setting.

 

[crushnik]

Regarding the IR channel that generated a wonky watermask, now I don't know how to get GIMP to edit the TIF file now generated by the GDALwarp process above. Rather than a color image that I can adjust brightness and contrast levels on -- or edit the channels -- I get a greyscale image with just one layer.

In P3D the scenery has color, and looks good except from there being water everywhere. Much better in fact than burning in watermask from vector polygons; the IR channel has water where it belongs and seems to actually blend the watermask, but the downside is watermask is also all over on hillsides and other places it doesn't belong as indicated by my previous attached images.

Does anyone have an idea what to do with this mess I've created? I was thinking that if I removed the IR layer that it would fix the water issue, and perhaps eliminate the overexposed effect that I have to overcome by manually decreasing the brightness and increasing the contrast. I say that because when I do a multisource INF file, the water shows just a little bit of the ortho water despite the fact that my watermask is straight black and white - no grey which if I understand right will let some of the ortho water peek through the P3D/FSX water texture.

 

[GaryGB]

Hi Chris:

The (several) sub-topics under discussion in this thread:

* Creating autogen buildings based on population density using Microsoft building footprints for P3D

* Creating Land-Water masks for 3.75-minute ortho tiles from 'large' USGS waterbody *.SHP shapefiles

...while also using a lesser number of "regional" SDK Resample Land-Water and/or Blend 'Mask' 8-Bit gray-scale *.TIF files for larger areas

...and 'sampling' only certain Geographic areas of those data sources for SDK Resample "local" output BGLs of smaller areas

...using these SDK Resample Parameter values in a multi-source *.INF:

[Destination] Georeferencing Parameters to set 'limit of sampling region in degrees' values in the "local" *.INF file

* Removing a NIR Band from custom photo-real imagery source data before processing via SDK Resample

* 'Burning' a B+W Land-Water Mask into custom photo-real imagery source data ...versus use of discrete "Channels".


First, I believe it might be best to separate these tasks into several threads so each inquiry is given the detailed discussion required for a more thorough understanding during the learning process.


Remove the NIR Band from custom photo-real imagery source data before processing via SDK Resample:

Ortho has ruined my scenery

Once I had P3D looking good with high-resolution terrain mesh, accurate water and road vectors, millions of buildings, and even vegetation shapefiles, I made the mistake of adding golf courses and quarries from ortho scenery. Now everything else looks ridiculous. I wonder if there is a more...

www.fsdeveloper.com

Regarding the creation and use of a B+W Land-Water Mask and/or a Blend Mask as 8-Bit Gray-scale TIFs:
these may be easier to work with as discrete files rather than as layers within TIF / GeoTIFF graphic files used as custom photo-real imagery source data for the "visible" (aka "diffuse") daytime terrain textures.

Although one could 'burn' a pure Black (RGB 0,0,0) color painted over water areas into source imagery submitted to legacy versions of SDK Resample in pre-FSX versions of FS in order to get the run time rendering engine to swap in default hydro attributes and default local water class texture colors in those pure Black areas, it is best to use the FSX et seq. SDK Resample methods for FSX / P3D.

That means using discrete "Channels" in a multi-source *.INF to define Land-Water Mask and/or a Blend Mask source data used for custom photo-real imagery BGLs.


Technically one 'can' use 24-Bit color and even 32-Bit color images derived from source data used for custom photo-real imagery as discrete "Channels" in a multi-source *.INF to define a Land-Water Mask and/or a Blend Mask for SDK Resample custom photo-real imagery BGLs.

There are methods to 'paint' / 'burn' / 'merge' colors into such 24-Bit color and even 32-Bit color images used as ex: 1-piece Masks to cause the run time rendering engine to swap in default hydro attributes and default local water class texture colors.

However, color versus gray-scale leads to larger SDK Resample source data 'work' data sets and larger output BGL sizes.

IMHO, these techniques in addition to being rather obscure, are also limited as to the number of terrain texture options which may be utilized by this method.

AFAIK, it is best to instead 'window' (aka "punch a hole" as it was once described by ACES' Terrain lead developer Adam Szofran) in the "visible" (aka "diffuse") daytime terrain textures using graphical "Channel" transparency Mask methods with 8-Bit, 256 gray-scale step TIF / GeoTIFF files for Masks ...rather than to use colors in such 24-Bit color and even 32-Bit color images used as ex: 1-piece Masks, or the even more obscure "Color Key" methods ...for FSX / P3D custom photo-real imagery BGLs.

Thus, MS-ACES implemented the use of (smaller) 8-Bit, 256 gray-scale step TIF / GeoTIFF files for Masks.


To review info on FSX / P3D SDK methods for custom photo-real imagery BGLs, you may wish to review at least the first few pages of this (long) thread:

https://www.flightsim.com/vbfs/printthread.php?t=250762&pp=40

How to create photoreal scenery for FSX


NOTE: Jim Robinson, as a knowledgeable, major contributor to that very informative thread, recommends a different method for working with terrain vectors, which AFAIK, represents a "best practices" consensus of the FS Development community:

EXCLUDE, THEN REPLACE DEFAULT / 3RD PARTY SCENERY ...DO NOT DISABLE / EDIT DEFAULT SCENERY BGLS


You may also wish to review these excellent tutorials by Luis Feliz-Tirado:

Make photo-real ground textures

File Description:
It is very easy to create your very own high-resolution, custom (photo-real) ground textures. This document explains the concepts and techniques and illustrates the use of SBuilderX with which you can quickly and easily download aerial images and make this type of scenery. So, why hesitate? Make Flight Simulator scenery as real as it gets! Very sorry - no support of any kind is offered. Please do not write. For any questions, please post in the Avsim Scenery Design Forum.

Filename: make_photo-real_ground_textures_in_fs_x.zip
License: Freeware
Added: 21st November 2009, 23:20:06
Downloads: 17714
Author: Luis Feliz-Tirado
Size: 2143kb

https://library.avsim.net/esearch.php?CatID=fsxsd&DLID=140539


Terrain Design

File Description:
Terrain Design for Flight Simulator X

This simple illustrated document explains the basic concepts in Flight Simulator X terrain, and how to change them using SBuilder for FS X. Make new water bodies, roads, vehicle traffic, airport flattens, remove autogen, add more detail - anybody can do it easily and quickly.
My most sincere apologies - I am unable, for lack of free time, to offer support of any kind. For questions or problems, please post in the Avsim scenery design forum.

Filename: terrain_design_for_flight_simulator_x.zip
License: Freeware
Added: 14th December 2009, 13:29:15
Downloads: 16648
Author: Luis Feliz-Tirado
Size: 3991kb

https://library.avsim.net/esearch.php?CatID=fsxsd&DLID=141643


Using the Autogen Annotator

File Description:
The Autogen Annotator is a powerful tool for modifying the autogen of default ground textures or adding autogen to custom (photo-real) ground. This document illustrates the use of the tool, and includes a pictorial guide to all autogen object types. Very sorry - no support of any kind is offered. Please do not write. For any questions, please post in the Avsim Scenery Design Forum.

Filename: using_the_autogen_annotator_264833.zip
License: Freeware
Added: 21st November 2009, 16:18:51
Downloads: 4220
Author: Luis Feliz-Tirado
Size: 8930kb

https://library.avsim.net/esearch.php?CatID=fsxsd&DLID=140537


Hope this helps a bit more.

GaryGB

 

[crushnik]

The (several) sub-topics under discussion in this thread:

Good grief, there ARE a lot of sub-discussions.

Hopefully other aspiring scenery designers can be helped from all my questions and your answers.

 

[GaryGB]

Hi Chris:

IMHO, all sub-topics are pertinent to this complex multi-step work-flow, and we can all learn from such discussions .

GaryGB

 

[crushnik]

Your link led me to another simple OsGeo4W script from JRobinson (see below) that seems to have worked to strip out the IR band from my TIF file. The filesize is about 25% smaller and the output file it still looks normal to me in QGIS... resampling now.

qdal_translate -b 1 -b 2 -b 3 -of GTiff -co INTERLEAVE=PIXEL "path\to\input.tif" "path\to\output.tif"

I'll break further discussions into a new thread/threads after solving this puzzle.

 

[JRobinson]

My discovery is probably new to no one but me...

Nope, I didn't know that one! That's an awesome trick! Thanks!

OK, here's the big info dump:

First a little more on the %path% trick, it possibly might help you automate things including burning your watermask.shp onto watermask.tifs if that's what you decide to do:
In a command window you can "set" an environment variable and that's how I've been doing all my gdal stuff for years, I'll paste as the first command I run when I open the OSGeo4W Shell for example set path=D:\Temp2\mesh\sources and from that point forward whenever a command contains %path% the command interpreter sees D:\Temp2\mesh\sources, so ie "%path%\USGS_13_n41w107.tif" is interpreted as "D:\Temp2\mesh\sources\USGS_13_n41w107.tif". (until I close the OSGeo4W Shell, if I open it again later I have to re-set the variable)

For that matter you can also type cd /d "D:\Temp2\mesh\sources" (cd is "change directory", /d is needed if you're changing from one drive to another) and then you can just use bare filenames like USGS_13_n41w107.tif, USGS_13_n41w108.tif, etc without needing the path, maybe that would be easier. Six one way, half a dozen the other I guess.

I use another command to gather a list of my .tifs. You don't need to do this from the OSGeo4W Shell but you can if you want to, any command window will work, and you can open one in your working folder by clicking off in the white space (with no files selected) while holding the shift key down, then choose "open a command window here". Type dir /b *.tif > tifs.txt, or if you're doing it from the OSGeo4W Shell and you've already set your variable you could type/paste dir /b "%path%\*.tif" > "%path%\tifs.txt". Either will generate a txt file named "tifs.txt" with a list of your .tifs. Type: dir /? for help on the dir command, there may be other things you can use it for.

(EDIT: Read this paragraph if you don't read anything else: Macro in Notepad++)
So then open tifs.txt in notepad++ and do "Macro > Start recording". With the cursor at the beginning of the first filename type "%path%\ then press End key, type ", press down arrow key; press Home key; press backspace; press space bar. Now do "Macro > stop recording". Next do "Macro > Run a macro multiple times" and put a dot in "run to end of file". Now your .tifs are all on one line with the environment variable in place and a space between each file. Ready to paste into your gdal_warp command. Do "Macro > Save current recorded macro", give it a name you'll remember like "tifs_on_one_line" or something and then all you have to do is "Macro > Run a macro multiple times", select the saved macro, and tick "run to end of file". Macros can help you automate all sorts of things so take a good look at them and think about how you might use one whenever there are repetitive coding operations that need to be done.

Now that I know it can extract watermask data from the NIR/alpha channel, is the GDALwarp process you recommended in Post #17 still my best option for burning in a watermask?

On using the IR channel as a watermask - In photoshop there's a magic wand tool, there's also a channels palette that lets you edit the alpha channels like a layer. The magic wand tool allows you to select a color and you can set a tolerance plus or minus for that color where "near" colors can be selected too, there's a "continuous" option where this range of colors must be connected. I've done this before with the IR channel, when you get your tolerance right it takes pretty much one click to select an entire river, then you fill that selected area with black, invert the selection, and fill the remainder with white. Presto - watermask. It isn't always perfect, in fact it can be pretty rough in some spots, you'd just have to try it and see what you think. I do know there's an equivalent tool in Gimp, I've played with it, can't remember what it's called. Also I have no idea how you'd edit the alpha channel in Gimp but there must be a way. Also note that you can add a second alpha channel for your blendmask, although that isn't needed with what you're doing here at the moment. In a .tif channel 3 (alpha 1) can optionally be your watermask and channel 4 (alpha 2) can optionally be your blend mask.

is there an efficient way to automatically adjust brightness, contrast, vibrancy/saturation outside of doing it for each TIFF file in a photo editing program?

In photoshop there are "actions", they're very much like the Notepad++ macro I described above. I have one that adds a pre-set selective color layer followed by a pre-set hue & saturation layer, followed by a brightness & contrast layer. Then I use another action to flatten the image, do unsharp mask (twice), replace black (0,0,0) with "not quite black" (2,2,2) and I use NullValue=0,0,0 in the .inf.

That said I've been playing with nconvert.exe lately which is a command line tool and I'm pretty impressed. I've been using nconvert for years, never thought it had the power to do proper color adjustments, or at least I always thought PhotoShop was a better choice, but I messed with the command and the options available last night and IMO came up with a pretty good approximation of what I've been doing in PhotoShop, all from the command line without any actual graphical editing.

XnView Software · NConvert

www.xnview.com

You do gdal_warp generating a .tif, then you extract the geo data out of it (because nconvert will strip it), run the nconvert command, and finally put the geo data back in. Something like this:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "%remote%\keystone_NAD83.tif" "%remote%\SE_keystone_WGS84.tif"
::
::extract the geo data to be put back in after the nconvert step
listgeo -d "%remote%\NW_keystone_WGS84.tif" > "%remote%\NW_keystone_WGS84.gtf"
::
::the nconvert command, to see all the nconvert options type: nconvert -help > nchelp.txt
"%nconvertpath%\nconvert.exe" -overwrite -balance 1 5 -10 -brightness -5 -contrast 40 -sharpen 20 -replace 0 0 0 2 2 2 "%remote%\NW_keystone_WGS84.tif"
::
::finally put the geo data back in so GeoTiffToInf can read it
geotifcp -g "%remote%\NW_keystone_WGS84.gtf" "%remote%\NW_keystone_WGS84.tif" "%remote%\NW_keystone_WGS84a.tif"

Last night I put all that (and a ton more, the .shp-to-watermask command, a similar one for the blendmask, and finally the resample step) into a batch file, just double click it and it poops out a .bgl when it's done running, takes it about five minutes to run starting with a .bmp from SBuilderX (google tiles) and ending with a water and blend masked photoreal .bgl, love stuff like that, lol. TBH I haven't looked at it in the sim yet and I probably need to work on the values to make it match but in TMFViewer (irrelevant) I do like the look of it better.

I was thinking that if I removed the IR layer that it would fix the water issue

If you wanted to automate that with a command you could use gdal_translate, -b 1 -b 2 -b 3 equals band1, band2, band3 - band4 would be the alpha so just leave it out (I see you've found this already)

gdal_translate -b 1 -b 2 -b 3 -of GTiff "%remote%\m_4406861_ne_19_1_20150711_20151103_WGS84.tif" "%remote%\m_4406861_ne_19_1_20150711_20151103_no_alpha.tif"

Jim

 

[JRobinson]

Oh, and one more note, on the gdal_rasterize command for turning your watermask.shp into watermask.tif you can use -i to invert it, then you can skip the step of opening the output file in Gimp and inverting it


gdal_rasterize -i -burn 255 -burn 255 -burn 255 -tr 0.000009266010702 -0.000009266010702 -te -126.02494697 45.07114956 -125.97001807 45.11255936 -ot BYTE "%path%\SBuilder\shapes\fantasy_island_blend.shp" "%path%\PR\fantasy_island_blend.tif"

 

[crushnik]

Oh, and one more note, on the gdal_rasterize command for turning your watermask.shp into watermask.tif you can use -i to invert it, then you can skip the step of opening the output file in Gimp and inverting it

Are you Santa Claus? Your "big info dump" is a Christmas miracle!

I have a process that works pretty darn good now, and it will be even better by making some tweaks based on your recommendations. Merge eight JP2 files into one TIF, copy and preserve the original for future edits, make an INF, rip out the IR layer (which will be used later for autogen trees), then resample. It looks great and so much of the process is automated. Removing the IR layer got rid of a lot of the over-exposed effect, just like I suspected it would once I saw all the water everywhere from my massive tiles earlier in the post.

What troubles me is JUST before I shut down for the night, I noticed a dreaded black strip running between two tiles. I'll have to tackle that later in the week, but great day -- it's amazing to fly around my home state now with sharp terrain mesh, photoreal scenery, and the beginnings of realistic water effects popping through, tons of accurate buildings, and coming soon, trees!

Thank you JRobinson!

 

[crushnik]

As I realize that photoreal scenery on a large scale isn't going to be feasible for just one guy, I am trying to work with autogen scenery as much as possible. That said, I would like for urban areas to have accurate building footprints.

Using Microsoft's building footprints, I would like to clip out the polygons that fall within the USGS' urban area shapefile layer. I prefer to work with pre-filtered shapefiles from QGIS, but attempts to clip statewide data just stay stuck in the processing stage. Exporting a 1-degree tile of buildings and working with that limited dataset doesn't get anywhere either.

In ScenProc I tried the following, which is based on an example in the manual:

ImportOGR|path\to\Missouri buildings.shp|*|*|NOREPROJ
ImportOGR|path\to\Urban Areas.shp|*|*|NOREPROJ
#
SplitGrid|AGN|*|FROMFILE="Missouri buildings.shp"
#
AddCellAttribute|*|String;area|rural
AddCellAttributeIfInside|*|FROMFILE="Urban Areas.shp"|String;area|urban
#
SetAGNBuildingHeight|area="rural"|1.0;0.0;0.0;0.0
SetAGNBuildingHeight|area="urban"|0.8;0.5;0.1;0.0
#
CreateAGNGenBuild|FROMFILE="Missouri buildings.shp" AND FAREARAT>0.7 AND FWIDTH<20|{5ae04eb6-934c-4f63-bb48-5e7dee601212}|MAXRATIO=2
CreateAGNGenBuild|FROMFILE="Missouri buildings.shp" AND FAREARAT>0.7 AND FWIDTH>20|{6089A0BD-CED1-4c47-9A9E-64CDD0E16983}
#
ExportAGN|P3D v2|path\to\Texture

I never get past the "SplitGrid" step with the above formula (I haven't yet found a good way to strip out "rural" shapefiles, but since I'm not making it past SplitGrid, that doesn't seem to be a factor yet). Scenproc can generate a 1-degree tile of buildings (numbering in the tens of thousands) in a few seconds if there are no filters, so I'm puzzled why I can't get anywhere with a filter, either in QGIS or ScenProc.

Does anyone have any suggestions?

 

[bernb]

I am not even an amateur at this, but I can point you to a place and script that may help you.
The program is Ortho4XP for FSX/P3D - do a Google search for a discussion on it.
and the program and source can be found here on github - Ortho4XP

There is a scenProc script that processes files downloaded to produce autogen for the area. While it is not complete it will give you some ideas.
I have modified this script to the best of my abilities and that modified script is attached below.
Check out the program and default script and also the modded one attached.
When you improve on it please post the improved script.
Hope this helps you.

Cheers

 

[MyPC8MyBrain]

hello bernb,
the script you speak of was created by whomever is working on the tool you mentioned,
it is not an official scenProc component,

 

[bernb]

hello bernb,
the script you speak of was created by whomever is working on the tool you mentioned,
it is not an official scenProc component,

What do you mean scenProc component? Who said that it was a component? And what exactly is an "Official component"? scenProc runs scripts.
It is clearly mentioned who the author of the original script is and I made modifications to it to add/adjust the autogen creation. This is what member
crushnik asked about and not some imaginary component.

Please comment something useful. Download the script and read it before talking about official components.

 

[MyPC8MyBrain]

i thought you were suggesting modifications to the script included with the tool mentioned rather than a reference,

 

[crushnik]

Oh, and one more note, on the gdal_rasterize command for turning your watermask.shp into watermask.tif you can use -i to invert it, then you can skip the step of opening the output file in Gimp and inverting it

gdal_rasterize -i -burn 255 -burn 255 -burn 255 -tr 0.000009266010702 -0.000009266010702 -te -126.02494697 45.07114956 -125.97001807 45.11255936 -ot BYTE "%path%\SBuilder\shapes\fantasy_island_blend.shp" "%path%\PR\fantasy_island_blend.tif"

Jim, A couple months ago you mentioned this process that I just now found time to experiment with. This results in an RGB file that is just as big as my orthoimagery TIFF. Am I doing something wrong, or do I need to have a 1.5 gigabyte watermask to go with my 1.5 gigabyte scenery?

Thanks as always,
Chris

 

[JRobinson]

Yeah, it'll be the same size as the original imagery but I suppose you could convert it to 8-bit gray scale to save a few Mb. My approach has always been resample the PR and then delete anything you can reproduce with a command if you need to. If you have the .shp and the target resolution (-tr) and extents (-te) saved in a .txt file you can make a new water mask in just a few seconds if you need it again.

 

[crushnik]

That's great. I hadn't thought of being able to delete the file since I haven't got my multisample INF file to resample yet. Thank you!