Creating free 1/3 arc-second mesh for US and need some guidance

[crushnik]

What is the download URL for the 1/3 Arc Second 37N121W tile data source you used above in QGIS ?

I use USGS National Map (https://viewer.nationalmap.gov/basic/) so we are looking at the same file. I just wonder why it won't convert without giving me the unsettling "ballpark" message (and perhaps throwing off the accuracy and consistency I have maintained so far for several thousands of square miles.

 

[GaryGB]

Once one has a list of required files for CONUS, one could also use this info for direct downloads.

Here's a way to more directly access USGS data files without using The National Map Viewer web portal:

Can The National Map data be downloaded via direct links? | U.S. Geological Survey

Direct access to The National Map data is provided via browsable links through:  Amazon's Cloud USGS Server - https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/ Navigate to the appropriate theme folders to access staged products for download.  Data can also be downloaded using The...

www.usgs.gov

Example URL:

ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation/13/TIFF/


GaryGB

 

[crushnik]

Whoa, mind is blown. Since unplugging and re-plugging usually fixes things for me, I just re-downloaded the file and tried again. I didn't run into the same problem. I blame it on gremlins, but I'll see if it checks out in the sim.

 

[crushnik]

37N119W is giving me giant peaks surrounding a large pit at a certain elevation. Examining the BGL in TMFViewer gives me this...

... mousing over the purple areas shows good elevation data, but I get this:

Which turns into this when you get close:

This is around Mount Whitney in California. This tile is another instance of the issue of re-projecting the data I mentioned in a previous post.

Chris

 

[crushnik]

... mousing over the purple areas shows good elevation data, but I get this:

Upon further review (and fresh eyes after sleep and coffee) I just noticed that the purple areas show the correct elevation numbers but they are NEGATIVE instead of positive values. And then there's this wonky action going on outside the borders (as seen in TMFViewer), where there shouldn't be any data:

 

[GaryGB]

Hi Chris:

IIUC, you are referring to:

ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation/13/TIFF/n37w119/USGS_13_n37w119.tif


Please post in a reply here, the *.INF file you used with the 1x1 degree tile in question.

GaryGB

 

[OzWookiee]

Love seeing all of this learning and creating truly wonderful 3D terrain for your native area. FreeMeshX have their 10m mesh available for the entire USA: http://ninetwopro.com/freemeshx/bit-torrent-download-links/

By no means meant to discourage, as the way you are progressing is fantastic and you get a much better appreciation for something when you try to do it yourself! I've done a similar thing for Australia.

 

[crushnik]

Love seeing all of this learning and creating truly wonderful 3D terrain for your native area. FreeMeshX have their 10m mesh available for the entire USA: http://ninetwopro.com/freemeshx/bit-torrent-download-links/

By no means meant to discourage, as the way you are progressing is fantastic and you get a much better appreciation for something when you try to do it yourself! I've done a similar thing for Australia.

Outstanding, I like the fact that other designers can see the progression and how to solve the problems that always come up. This has been a highly rewarding experience; it's like art you can play with!

I used the mesh you link above, and it made flying outside the United States a LOT better. What source are you using for your Australian mesh? I don't like being limited to the United States, but higher-resolution data in one place can ruin the lower-resolution areas.

 

[crushnik]

Please post in a reply here, the *.INF file you used with the 1x1 degree tile in question.

As per our conversations above, here is what I have. I don't like coming here (to you, essentially) with every little problem and know enough now to solve many issues myself, so please know that I work very hard to find the answers myself first. This has been quite a learning experience, and while this is most likely a very simple fix, I do know a LOT more than I used to and fortunately many other members of the flight sim community can learn from this exchange for years to come!

[Source]
Type=GeoTIFF
Layer=Elevation
SourceDir="."
SourceFile="N37W119.tif"
PixelIsPoint=0
NullValue=-32767
[Destination]
DestDir="."
DesFileType=BGL
DestBaseFileName="N37W119"
CompressionQuality=100
LOD=Auto
FractionBits= 3

 

[OzWookiee]

Outstanding, I like the fact that other designers can see the progression and how to solve the problems that always come up. This has been a highly rewarding experience; it's like art you can play with!

I used the mesh you link above, and it made flying outside the United States a LOT better. What source are you using for your Australian mesh? I don't like being limited to the United States, but higher-resolution data in one place can ruin the lower-resolution areas.

Haha I can't remember it was a few years ago now. I think there are some sources available from the gov website. mix it in with the LiDAR data and you got mega res

 

[bdf369]

Missouri is so horribly flat in flight simulator normally and this kind of data would really make flying anywhere a blast, don't you think?

Yes indeed. I made a freeware KPLK and had to improve the mesh there as well (you can grab KPLK at AVSIM if you are interested).

 

[GaryGB]

As per our conversations above, here is what I have. I don't like coming here (to you, essentially) with every little problem and know enough now to solve many issues myself, so please know that I work very hard to find the answers myself first. This has been quite a learning experience, and while this is most likely a very simple fix, I do know a LOT more than I used to and fortunately many other members of the flight sim community can learn from this exchange for years to come!

[Source]
Type=GeoTIFF
Layer=Elevation
SourceDir="."
SourceFile="N37W119.tif"
PixelIsPoint=0
NullValue=-32767
[Destination]
DestDir="."
DesFileType=BGL
DestBaseFileName="N37W119"
CompressionQuality=100
LOD=Auto
FractionBits= 3

Hi Chris:

No worries; I'm confident that you are applying yourself to the self-study process with due diligence.


I'm glad we also have such opportunities to share worked examples of how to enhance FS terrain, and how to resolve challenges with some of the obscure SDK 'gotchas' one may encounter during development.


Indeed, those spikes and pits were definitely not there in the 1970's when I climbed Mt. Whitney twice IRL.


Ordinarily, if working with an older and/or 'inferior' USGS data set, this would require dealing with the 'dreaded' obscure FS SDK documentation on Using Scaled Elevation Values located below the section on Resample:

Terrain Overview

FYI: I previously posted an example use of FractionBits in the *.INF file which I thought you might intend to use with the terrain at 0T3 that would otherwise involve a merged 1-Meter LiDAR and USGS 10-Meter data set:

https://www.fsdeveloper.com/forum/t...-us-and-need-some-guidance.447467/post-843860


AFAIK, this part of your project is only using the cited USGS 1/3 arc-sec. (10-Meter) data set, and it is increasingly apparent from inspection of that data set, it has been thoroughly cleaned and enhanced as a 'hybrid' of several elevation sources.

So, as I have thus far not seen any residual "contour terracing" in my own tests today with the above cited USGS_13_n37w119.tif 1x1 Degree tile, I can now recommend that you try this edited version of your *.INF:

[Source]
Type=GeoTIFF
Layer=Elevation
SourceDir="."
SourceFile="N37W119.tif"
PixelIsPoint=0
NullValue=-32767
[Destination]
DestDir="."
DesFileType=BGL
DestBaseFileName="N37W119"
CompressionQuality=100
LOD=Auto

Note that this removes the FractionBits parameter value from the *.INF instructions for SDK Resample.

Be aware that when working with 'other' data sets derived from mixed sources which do show residual "contour terracing", one would have to implement individualized parameter values for FractionBits and BaseValue ...according to the maximum Altitude of each "higher" mountain peak as an individual data source listed in a section of a "multi-source" *.INF.

This is all discussed in FS SDK docs on Using Scaled Elevation Values below the section on Resample.

If this terrain rendering anomaly persists after using the above edited version of your *.INF, feel free to inquire further.

GaryGB

 

[GaryGB]

Love seeing all of this learning and creating truly wonderful 3D terrain for your native area. FreeMeshX have their 10m mesh available for the entire USA: http://ninetwopro.com/freemeshx/bit-torrent-download-links/

By no means meant to discourage, as the way you are progressing is fantastic and you get a much better appreciation for something when you try to do it yourself! I've done a similar thing for Australia.

PS: IIUC, OzWookiee refers to this FreeMeshX "Beta" CONUS Data set:

http://ninetwopro.com/freemeshx-usa-open-beta-announcement/

GaryGB

 

[crushnik]

Indeed, those spikes and pits were definitely not there in the 1970's when I climbed Mt. Whitney twice IRL.

Twice?! I get tired flying up there on the computer. Seriously though, I bet the view was spectacular. And what gets me is that Mount Whitney is the highest point in the Continental United States and Death Valley - the lowest point - is right next door.

And thanks for the instruction on stripping out the fraction bits; that removed the purple negative value areas.

It works!

 

[crushnik]

Is there anything I can do on the coding end of my custom scenery to keep it from building as I fly closer to a mountain? The sim makes slight elevation changes to make the terrain more complex once I get close to a terrain feature and the texture often adjusts as well. I feel like this could be my graphics card/cpu doing what they can to keep frame rates reasonable (30 fps) at the visual settings I fly with on a low-to-mid-range system, but if there is a setting I can tweak before I have to redo/edit a thousand or so scenery tiles, I'd prefer to make that change now.

P.S.: If anyone would like to download (or contribute their time) to the nationwide scenery product, it is freeware. Link is in the signature below. I added most of southern Missouri yesterday.

 

[GaryGB]

Is there anything I can do on the coding end of my custom scenery to keep it from building as I fly closer to a mountain? The sim makes slight elevation changes to make the terrain more complex once I get close to a terrain feature and the texture often adjusts as well. I feel like this could be my graphics card/cpu doing what they can to keep frame rates reasonable (30 fps) at the visual settings I fly with on a low-to-mid-range system, but if there is a setting I can tweak before I have to redo/edit a thousand or so scenery tiles, I'd prefer to make that change now.

Hi Chris:

Here is what the guys at FreeMeshX had to say about working with what IIUC, is the same data set you are working with:

http://ninetwopro.com/freemeshx-usa-open-beta-announcement/

"To reduce mesh popping (where higher LOD mesh pops in — another limitation of the ESP engine), use a higher LOD_Radius in your fsx.cfg or p3d.cfg. We recommend a minimum of 6.5. I personally use 8.5 which is a nice compromise between performance, memory, and visuals. The higher the LOD_Radius, the further out the mesh will be drawn at its maximum LOD, less mesh popping, and the better it looks !"


Generally speaking, LOD switching is an inherent attribute of a 3D world rendering program that uses a gridded quad tree.


As far as what one may do with input elevation source data format and SDK Resample *.INF parameter values, there are a few things one may try.

Because the data set you are using in this case appears thus far to have already been rather well-cleaned and AFAIK is free from "contour terracing", technically, we not only do not need to use FractionBits, but also do not need 32-Bit Floating Point format source files.

Theoretically, GIS application (down-sampled) output of 16-Bit Integer format elevation source files may allow faster initial loading and LOD-switching at run time for terrain mesh BGLs. with an approximately 20% reduction of data footprint size ...without having used any percentage SDK Resample compression via a CompressionQuality parameter value at anything less than 100.

One may use a percentage SDK Resample compression via a CompressionQuality parameter value less than 100, ex:

CompressionQuality=97 (already a "significant" reduction of compiled imagery BGL size)

...or:

CompressionQuality=85 (up to 40% reduction of compiled imagery BGL size)


Whether / how that impacts terrain rendering performance in FS at run time may vary by "end user" configuration.


You may wish to review these informative threads on this sub-topic:

FSXA - Resample Tool CompressionQuality

I am working at the moment with a mesh for Vancouver Island. I decided to try half of the area (the southern portion with portions of the BC Mainland and the US San Juan Island area) but ended up with a bgl of 168 MB which I consider too large a file. I started to experiment on a smaller...

www.fsdeveloper.com

One Bgl Vs multiple bgls: Which is faster to load?

Does anyone have any idea/done any testing about which is more efficient: Covering a large area with one Bgl (Sim only has to load one file....but it is a large one) or multiple smaller bgl (sim has to load many files, but they are smaller files and it doesnt have to load data that isnt in...

www.fsdeveloper.com

There are also various things one may try to optimize run time rendering performance on a given end users hardware and software milieu in Windows and in FS.

Most end users may not tweak FS system configuration to the extent that Nick Needham does, but his guide helps:

THE FSX COMPUTER SYSTEM: THE BIBLE - BY: NickN

Let me make something perfectly clear, and believe me when I say this, its with a wide open...

www.simforums.com

Hope this helps as an introduction to some of the options you may wish to explore.

GaryGB

 

[crushnik]

Gary,
I only discovered NineTwoPro's Beta mesh scenery for the United States (I use their global product) after I spent nearly a month spining my wheels on laying down my first tile, I had to find something "different" to make my project relevant. I do remember seeing that they compress their files in some areas, and my scenery does not. By trimming out the fraction bits from the .INF file as you suggested, I cut off about a third of the file size. I do have a few dozen tiles to recompile again if your suggestion works for me, but that is a WHOLE LOT better than having to go back and re-do a thousand tiles.

Your link to Nick Needham's FSX Computer System "Bible" hooked me from the start: I tend to like anyone salty enough to tell readers to buzz off if they don't agree with what he has to say or finds his humor offensive! Sounds like Ricky Gervais telling off Hollywood at the Oscars and that makes me happy.

Cheers!

 

[crushnik]

With P3Dv6 coming out I'm revisiting my Flyover Country mesh scenery. USGS has some updated 1/3 arc-second data and I recompiled a BGL using LOD=Auto instead of the LOD=4,12 I used previously. File size about 75% bigger, and I'm wondering if the increased size helps. No matter how big I made the LOD radius, my mesh would still make changes about a mile away from the aircraft, which is terribly noticable with stock scenery textures. Wondering if that was because of the LOD restrictions in the INF file. Does anyone have insight on that?

 

[GaryGB]

With P3Dv6 coming out I'm revisiting my Flyover Country mesh scenery. USGS has some updated 1/3 arc-second data and I recompiled a BGL using LOD=Auto instead of the LOD=4,12 I used previously. File size about 75% bigger, and I'm wondering if the increased size helps.

Hi Chris:

Sorry for the delay in posting a reply; I have been quite busy with a number of matters IRL this year.

[EDITED]

I (also) would not limit the max terrain mesh LOD to 12, as FS / P3D SDK Resample will automatically read the optimum output resolution achievable with the supplied source data, and set it to the maximum achievable based on the supplied terrain mesh source data set.

Thus, I (also) would try using LOD=4,Auto.

[END_EDIT]

If output file size is an issue, consider using CompressionQuality=97
(see above): https://www.fsdeveloper.com/forum/t...-us-and-need-some-guidance.447467/post-844382

If you post some info on your current project goals for a small-sized example area you intend to update, I may be able to offer some ideas on how to configure the source data and the INF file to yield the desired result.

No matter how big I made the LOD radius, my mesh would still make changes about a mile away from the aircraft, which is terribly noticeable with stock scenery textures. Wondering if that was because of the LOD restrictions in the INF file. Does anyone have insight on that?

Are you describing a de-crease of detail in terrain mesh within a mile of the user aircraft on / near ground at an airport ?

If so, this may be due to increased local airport flatten extents.

Or are you instead describing an in-crease of terrain mesh detail within a mile of the user aircraft on / near ground at an airport ?


BTW: Has P3Dv6 changed the default supplied terrain mesh and/or terrain texture resolution ?

IIRC, L-M was rumored to begin using Unreal Engine for P3Dv7; will that happen ?

If so, how will that impact displayable terrain resolution ?

https://www.avsim.com/forums/topic/...e-based-p3dv7/?do=findComment&comment=4981114


Regardless, FSX / P3Dv6 SDK and rendering engines can now create / display 1 Meter resolution terrain mesh at run time with no FPS hit.

[EDITED]

FYI: The USGS now has seamless coverage of the USA (CONUS) with 10 Meter DEM data, and is no longer limited to a "cleaned" 1/3 Arc Second (aka "30 Meters") between elevation data points as still seen in (certain) earlier subsets of the largely SRTM-derived data.

[END_EDIT]

GaryGB

 

[crushnik]

Great to hear from you Gary, I hope all is well.

I am looking at the USGS National Map downloader, and I see a few states have 1/9 arc second data. Is this the 10 meter DEM data you're talking about?