• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

MSFS Everything about CGL generation [Custom DEM works]

Messages
15
Country
finland
While waiting for Asobo to get their SDK together I've been slowly digging into cgl files. I'm creating this thread to keep possible discussion in the correct part of the forum.

Like [theisomizer] explains in another thread on this forum, cgl is a container format. It has an uncompressed header, which contains high-level information of what the cgl contains. Namely compression parameters, object/tile count and such. This uncompressed header is followed by a lzma compressed "data header" that holds information about individual objects/tiles in the cgl. Their delta compressed quadkey position relative to cgl base quadkey, delta compressed compressed data length and uncompressed data length relative to compressed data length. After that follows lzma compressed data streams, one for each object in the cgl.

In a dem-cgl each object is a 257*257 array of elevation values (8 or 16 bits) with a 7 bytes long header. That header has height multiplier (Float32), height offset(Int16) and bitdepth/bitsperpixel (Int8).

Python code of a hacky implementation of dem and cgl generation plus some more documentation can be found at GitHub: muumimorko/MSFS2020_CGLTools
Also a proof of concept mod is available at flightsim.to: DEM for Finland - 20 meter resolution

[This post will update]
 
Messages
34
Country
taiwan
You are simply amazing!! However it's still too deep for me. I got several DEM tif files ready but may need some easier guidance. Great job!!!
 
Messages
127
Country
italy
You are simply amazing!! However it's still too deep for me. I got several DEM tif files ready but may need some easier guidance. Great job!!!
Hello Breadeater (Aka Morko), I can only confirm all this. Thanks for your commitment.

## Prereg
```
Python (tested on 3.8) >>>>> I use 3.9
numpy==1.19.3 >>>>> ? |
matplotlib >>>>> ? | Where to get these from, and where to install?
opencv-python >>>>> ? |
Blue Marble Global Mapper (other mapping packages can be adapted) . >>>>> I use QGIS 3.16
```
For me this is important to understand in order to leave.

I have source .tif files with a resolution of 12.50 meters to feed your application.
 
Messages
6,728
Country
us-illinois
Hello:

While I appreciate the good intentions of posts trying to help others in their native language, English is the declared language to be used in FSDeveloper forums.

Here is a Google translated version of the latter web page linked above:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiL3eqI27ntAhURbs0KHYSvB5YQ7gEwAHoECAEQCQ&url=https://translate.google.com/translate?hl=en&sl=it&u=http://www.andreaminini.com/python/matplotlib/come-installare-matplotlib-su-python&prev=search&pto=aue&usg=AOvVaw2KICFsEO2Q7emCxwRzdYTv


FYI: Google has free translation features which may help us all understand typed posts ...and linked web sites: :idea:



If a web site URL is entered into Google for a search:


...notice we also get a link to a translated version of the web site URL; AFAIK, that link automatically translates to the language version configured in Windows on one's computer. :pushpin:


Please, lets get everyone 'on the same page', by making certain we can all understand one another.

Perhaps then, we may build the knowledge base for the FSDeveloper Community to an even greater height of accomplishment. ;)



GaryGB
 
Last edited:
Messages
15
Country
finland
Hello Breadeater (Aka Morko), I can only confirm all this. Thanks for your commitment.

## Prereg
```
Python (tested on 3.8) >>>>> I use 3.9
numpy==1.19.3 >>>>> ? |
matplotlib >>>>> ? | Where to get these from, and where to install?
opencv-python >>>>> ? |
Blue Marble Global Mapper (other mapping packages can be adapted) . >>>>> I use QGIS 3.16
```
For me this is important to understand in order to leave.

I have source .tif files with a resolution of 12.50 meters to feed your application.

Hi Bavarello :)

Sorry I missed your (and all the others) post.

numpy, matplotlib and opencv-python are python modules, the can be installed with "pip3".
At this moment, QGIS is not supported. There is no reason it could not be, but a day is only 24 hours...
 
Messages
127
Country
italy
While waiting for Breadeater to light up the way for the DEM / CGL of MSFS2020 and after an accelerated run-up, also with the precious help of SprightlyOldMan, I managed to complete the first approach (## Prereg) of the # MSFS CGL Tools.​
Python installed (How to install). *** From GaryGB Notice we also get a link to a translated version of the web site URL; AFAIK, that link automatically translates to the language version configured in Windows on one's computer. **​
Now let's go to install numpy == 1.19.3
Microsoft Windows [Versione 10.0.19042.662]​
(c) 2020 Microsoft Corporation. Tutti i diritti sono riservati.​
C:\Users\Utente>cd /d H:\Python39​
H:\Python39>pip uninstall numpy​
Found existing installation: numpy 1.19.4​
Uninstalling numpy-1.19.4:​
Would remove:​
h:\python39\lib\site-packages\numpy-1.19.4.dist-info\*​
h:\python39\lib\site-packages\numpy\*​
h:\python39\scripts\f2py.exe​
Proceed (y/n)? y​
Successfully uninstalled numpy-1.19.4​
H:\Python39>pip install numpy==1.19.3​
Collecting numpy==1.19.3​
Downloading numpy-1.19.3-cp39-cp39-win_amd64.whl (13.3 MB)​
|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 13.3 MB 6.8 MB/s​
Installing collected packages: numpy​
Successfully installed numpy-1.19.3​
H:\Python39>​
Let's go to install matplotlib
H:\Python39>pip install matplotlib​
Requirement already satisfied: matplotlib in h:\python39\lib\site-packages (3.3.3)​
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in h:\python39\lib\site-packages (from matplotlib) (2.4.7)​
Requirement already satisfied: python-dateutil>=2.1 in h:\python39\lib\site-packages (from matplotlib) (2.8.1)​
Requirement already satisfied: numpy>=1.15 in h:\python39\lib\site-packages (from matplotlib) (1.19.3)​
Requirement already satisfied: pillow>=6.2.0 in h:\python39\lib\site-packages (from matplotlib) (8.0.1)​
Requirement already satisfied: cycler>=0.10 in h:\python39\lib\site-packages (from matplotlib) (0.10.0)​
Requirement already satisfied: kiwisolver>=1.0.1 in h:\python39\lib\site-packages (from matplotlib) (1.3.1)​
Requirement already satisfied: six in h:\python39\lib\site-packages (from cycler>=0.10->matplotlib) (1.15.0)​
Requirement already satisfied: six in h:\python39\lib\site-packages (from cycler>=0.10->matplotlib) (1.15.0)​
H:\Python39>​
Let's go to install opencv-python
H:\Python39>pip install opencv-python​
Collecting opencv-python​
Downloading opencv_python-4.4.0.46-cp39-cp39-win_amd64.whl (33.5 MB)​
|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 33.5 MB 85 kB/s​
Requirement already satisfied: numpy>=1.19.3 in h:\python39\lib\site-packages (from opencv-python) (1.19.3)​
Installing collected packages: opencv-python​
Successfully installed opencv-python-4.4.0.46​
H:\Python39>​
 
Last edited:
Messages
127
Country
italy
At this moment, QGIS is not supported. There is no reason it could not be, but a day is only 24 hours...
Hello Breadeater :),

this will be highly appreciated. Meanwhile the first scripts starts up ... but there is still a lot to figure out such as how to do it "quadkeys"

# Splits DEM to tiles inside specified quadkeys


# Make a list of quadkeys, base quadkey is topleftmost tile to be reconstructed
# padding fills additional tiles around target qkeys,
# if not padded, edges will go to zero elevation



N:\MSFS2020_CGLTools>py 1_GM_create_tilescripts.py
1022310
1022311
1022312
1022313
4

N:\MSFS2020_CGLTools>
 
Messages
45
Morning breadeater
Some tiles show gap between each others
Not sure why - do you have any suggestion how to avoiid ?
Tx
 
Messages
15
Country
finland
Are both tiles generated with the script?
If so, could you zip and upload the scripts, Tile and Delta folders used to generate them somewhere so I can take a look?
 
Messages
15
Country
finland
Not strictly about CGLs, but latest update made it possible to generate rectangle heightmaps, only brush tools in-game, but intermediate file format is simple elevation grid, before game compiles it to BGL. That intermediate format is easy to generate. So replacing small areas with very high detail is super easy. Works also at airports, so real 3D runways <3. Integrates seamlessly with CGL-DEMs if generated from same source.
Untitled.jpg

Above is 3x3km with ca. 12 meter resolution (256x256 values) results in 129 kByte file.
 
Messages
34
Country
taiwan
Not strictly about CGLs, but latest update made it possible to generate rectangle heightmaps, only brush tools in-game, but intermediate file format is simple elevation grid, before game compiles it to BGL. That intermediate format is easy to generate. So replacing small areas with very high detail is super easy. Works also at airports, so real 3D runways <3. Integrates seamlessly with CGL-DEMs if generated from same source.
View attachment 66565
Above is 3x3km with ca. 12 meter resolution (256x256 values) results in 129 kByte file.
Hi Breaeater,
I see heightmap data like this, indeed a grid files with height only, and seems each map can not exceed 256x256 points. Looks like a wider area needs separate files. Any good instruction for us to teansfer DEM into it? I got QGIS and DEM data already. Thanks a lot!
 

Attachments

  • 794DEEB2-AD30-47ED-A741-D4C8988ADA1A.jpeg
    794DEEB2-AD30-47ED-A741-D4C8988ADA1A.jpeg
    3.8 MB · Views: 330
Last edited:
Messages
15
Country
finland
Hi Breaeater,
I see heightmap data like this, indeed a grid files with height only, and seems each map can not exceed 256x256 points. Looks like a wider area needs separate files. Any good instruction for us to teansfer DEM into it? I got QGIS and DEM data already. Thanks a lot!
QGIS is still a strange thing for me :p

Basically (This is in Global Mapper)
  1. Load elevation data (Workspace projection EPSG:4326)
  2. Correct elevation data to egm2008
  3. Generate "centerline" that goes north to south
  4. "Expand" that line to rectangle
  5. Export area inside that rectangle to 256x256 to (float32) BIL
  6. Flip the data "left to right"
  7. Generate "space separated" list of elevation data
  8. Replace data in intermediate xml with that generated data
  9. Replace coordinates with "centerline" start and end points
  10. Replace width with rectangle width in meters.
  11. Reload intermediate file in MSFS and check if it works
1608809514387.png

This code can do steps 6-10
Python:
from xml.etree.ElementTree import Element, SubElement, Comment
import matplotlib.pyplot as plt
import numpy as np
import os
from xml.etree import ElementTree
from xml.dom import minidom
# https://pymotw.com/2/xml/etree/ElementTree/create.html
def prettify(elem):
    """Return a pretty-printed XML string for the Element.
    """
    rough_string = ElementTree.tostring(elem, 'utf-8')
    reparsed = minidom.parseString(rough_string)
    return reparsed.toprettyxml(indent="  ")

top = Element('FSData', version='9.0')
widthMeters = 512.0
falloff = 500.0
priority = 0
latitude = 61.86315965120010
longitude = 26.67812731587493
altitude = 700.0
latitude2 = 61.85856547919839
longitude2 = 26.67812731355588
altitude2 = 700.0
widthPixels = 256
data = ""
infile = open(
    r"C:\MSFS SDK\Samples\SimpleAerial\AtestHeightmap\heightmap\srcbil.bil", 'rb')
tilearr = infile.read()
infile.close()
nparr = np.frombuffer(tilearr, np.float32).reshape((256, 256))
flipped = np.fliplr(nparr)


for x in flipped:
    for y in x:
        data += str(y)+" "
data = data[:-1]
rect = SubElement(top, 'Rectangle', width=str(widthMeters), falloff=str(falloff), surface="{47D48287-3ADE-4FC5-8BEC-B6B36901E612}", priority=str(
    priority), latitude=str(latitude), longitude=str(longitude), altitude=str(altitude), latitude2=str(latitude2), longitude2=str(longitude2), altitude2=str(altitude2))
heightmap = SubElement(rect, 'Heightmap', width=str(widthPixels), data=data)
outfile = open(
    r"C:\MSFS SDK\Samples\SimpleAerial\AtestHeightmap\heightmap\out.xml", 'w')
outfile.write(prettify(top))
outfile.close()

Sorry for no better explanation atm. Merry Christmas :)
 
Messages
34
Country
taiwan
QGIS is still a strange thing for me :p

Basically (This is in Global Mapper)
  1. Load elevation data (Workspace projection EPSG:4326)
  2. Correct elevation data to egm2008
  3. Generate "centerline" that goes north to south
  4. "Expand" that line to rectangle
  5. Export area inside that rectangle to 256x256 to (float32) BIL
  6. Flip the data "left to right"
  7. Generate "space separated" list of elevation data
  8. Replace data in intermediate xml with that generated data
  9. Replace coordinates with "centerline" start and end points
  10. Replace width with rectangle width in meters.
  11. Reload intermediate file in MSFS and check if it works
View attachment 66572
This code can do steps 6-10
Python:
from xml.etree.ElementTree import Element, SubElement, Comment
import matplotlib.pyplot as plt
import numpy as np
import os
from xml.etree import ElementTree
from xml.dom import minidom
# https://pymotw.com/2/xml/etree/ElementTree/create.html
def prettify(elem):
    """Return a pretty-printed XML string for the Element.
    """
    rough_string = ElementTree.tostring(elem, 'utf-8')
    reparsed = minidom.parseString(rough_string)
    return reparsed.toprettyxml(indent="  ")

top = Element('FSData', version='9.0')
widthMeters = 512.0
falloff = 500.0
priority = 0
latitude = 61.86315965120010
longitude = 26.67812731587493
altitude = 700.0
latitude2 = 61.85856547919839
longitude2 = 26.67812731355588
altitude2 = 700.0
widthPixels = 256
data = ""
infile = open(
    r"C:\MSFS SDK\Samples\SimpleAerial\AtestHeightmap\heightmap\srcbil.bil", 'rb')
tilearr = infile.read()
infile.close()
nparr = np.frombuffer(tilearr, np.float32).reshape((256, 256))
flipped = np.fliplr(nparr)


for x in flipped:
    for y in x:
        data += str(y)+" "
data = data[:-1]
rect = SubElement(top, 'Rectangle', width=str(widthMeters), falloff=str(falloff), surface="{47D48287-3ADE-4FC5-8BEC-B6B36901E612}", priority=str(
    priority), latitude=str(latitude), longitude=str(longitude), altitude=str(altitude), latitude2=str(latitude2), longitude2=str(longitude2), altitude2=str(altitude2))
heightmap = SubElement(rect, 'Heightmap', width=str(widthPixels), data=data)
outfile = open(
    r"C:\MSFS SDK\Samples\SimpleAerial\AtestHeightmap\heightmap\out.xml", 'w')
outfile.write(prettify(top))
outfile.close()

Sorry for no better explanation atm. Merry Christmas :)
Merry Christmas! Thanks for your help! Lighten us, always.
 
Messages
2
Country
russia
QGIS is still a strange thing for me :p

Basically (This is in Global Mapper)
  1. Load elevation data (Workspace projection EPSG:4326)
  2. Correct elevation data to egm2008
  3. Generate "centerline" that goes north to south
  4. "Expand" that line to rectangle
  5. Export area inside that rectangle to 256x256 to (float32) BIL
  6. Flip the data "left to right"
  7. Generate "space separated" list of elevation data
  8. Replace data in intermediate xml with that generated data
  9. Replace coordinates with "centerline" start and end points
  10. Replace width with rectangle width in meters.
  11. Reload intermediate file in MSFS and check if it works
View attachment 66572
This code can do steps 6-10
Python:
from xml.etree.ElementTree import Element, SubElement, Comment
import matplotlib.pyplot as plt
import numpy as np
import os
from xml.etree import ElementTree
from xml.dom import minidom
# https://pymotw.com/2/xml/etree/ElementTree/create.html
def prettify(elem):
    """Return a pretty-printed XML string for the Element.
    """
    rough_string = ElementTree.tostring(elem, 'utf-8')
    reparsed = minidom.parseString(rough_string)
    return reparsed.toprettyxml(indent="  ")

top = Element('FSData', version='9.0')
widthMeters = 512.0
falloff = 500.0
priority = 0
latitude = 61.86315965120010
longitude = 26.67812731587493
altitude = 700.0
latitude2 = 61.85856547919839
longitude2 = 26.67812731355588
altitude2 = 700.0
widthPixels = 256
data = ""
infile = open(
    r"C:\MSFS SDK\Samples\SimpleAerial\AtestHeightmap\heightmap\srcbil.bil", 'rb')
tilearr = infile.read()
infile.close()
nparr = np.frombuffer(tilearr, np.float32).reshape((256, 256))
flipped = np.fliplr(nparr)


for x in flipped:
    for y in x:
        data += str(y)+" "
data = data[:-1]
rect = SubElement(top, 'Rectangle', width=str(widthMeters), falloff=str(falloff), surface="{47D48287-3ADE-4FC5-8BEC-B6B36901E612}", priority=str(
    priority), latitude=str(latitude), longitude=str(longitude), altitude=str(altitude), latitude2=str(latitude2), longitude2=str(longitude2), altitude2=str(altitude2))
heightmap = SubElement(rect, 'Heightmap', width=str(widthPixels), data=data)
outfile = open(
    r"C:\MSFS SDK\Samples\SimpleAerial\AtestHeightmap\heightmap\out.xml", 'w')
outfile.write(prettify(top))
outfile.close()

Sorry for no better explanation atm. Merry Christmas :)
Greetings! And you can learn more about points 2-5? Thanks.
 
Last edited:
Messages
33
Country
germany
Any news from this?

It would be a good possibility with the Hightmap. Unfortunately I don't know anything about Global Mapper. I only use Qgis to create aerial images in xyz tiles, then transferred via tiles2bing. It would be super nice if you could somehow get the elevation information into the hightmap via Qgis. I have 1m DEM Data ready in QGIS for a very special runway with slopes in all directions. The handling with rectangles and polygons driving me crazy, a solution for this would be high appteciated :)
 
Messages
197
Country
unitedkingdom
Hello spritelyoldman (a.k.a. Breadeater?),

I was looking at your ItalyDEM, which is excellent work and a pleasure to see! :) Many thanks indeed for making this available to the community! πŸ‘πŸ‘

I was reading your documentation on GitHub. You wrote "Game height 0 is -16 meters relative to MSL" (which tallies with your instruction 2, above, namely "Correct elevation data to egm2008"), which I believe is an oversimplification of some really complex geodesy. A delta of 16m may work in some locations, but not others. It also depends on the geoid/ellipsoid and vertical datum of the source DEM. There is a lot of nuance and detail in this (hence entire textbooks and careers devoted to the topic), which I believe Asobo are also getting wrong in some places (I sense they could do with having an experienced geodesist on their team).

Also, why is the process [currently] limited to a GSD of 20m?

Many thanks indeed

Andy (a.k.a SpatialProf/Airtropper, depending on the site)
 
Top