BGLDEC - A resample BGL (and CGL) DECompressor

[Clutch Cargo]

Hey Sean.... Marcus from waaaay back. Remember simwestCONTROL? ha! Glad to see you are still in flightsim. Just released my first product. Two more are about to drop.

 

[Vogel]

Hi Theisomizer,

first a great thank for this new release, it's a great work opening amazing possibility !!!

working on an elevation sample, I have detected some strange think into .inf file generated with RAW files (.bin)

My sample file is a 16bits elevation Geotiff. Here is a raw view in QGIS before any work & conversion:

Details and geolocalisation data are stored into a .tfw file. here it is:

0.0000091084
0.0000000000
0.0000000000
-0.0000091084
-61.2346257144
14.8993192805

Step 1: Geotiff -> BGL with resample.exe from FSX SDK

I use this .inf file for resampling for FSX BGL format:

[Source]
Type = GeoTIFF
Layer = Elevation
Channel_BlendMask = 2.0
SourceDir = E:\MNT compil decompil Test\
SourceFile = Image.tif
GCS = WGS84
nCols = 8954
nRows = 9641
nBands = 1
PixelIsPoint = 1
SamplingMethod = Gaussian
SampleType = SINT16
XDIM = 0.0000091084
YDIM = 0.0000091084
ULXMAP = -61.2346257144
ULYMap = 14.8993192805

[Destination]
DestDir = E:\MNT compil decompil Test\out\
DestBaseFileName = MNT_test
UseSourceDimensions = 1
DestFileType = BGL
LOD = Auto

Step 2: BGL -> RAW (.bin) with BGLdec 0.9.4

Here is the .inf file generated by BGLdec 0.9.4 from this .bgl:

[Source]
Type = MultiSource
NumberOfSources = 2

[Source1]
Type = Raw
Layer = Elevation
Channel_BlendMask = 2.0
SourceDir = E:\MNT compil decompil Test\out\
SourceFile = DEM-17-32430-27343.bin
GCS = WGS84
nCols = 6145
nRows = 8449
nBands = 8449
PixelIsPoint = 1
SamplingMethod = Gaussian
BandLayout = BIL
ByteOrder = Intel
SampleType = UINT8
XDIM = 0.021887
YDIM = 0.021887
ULXMAP = -61.237793
ULYMap = 14.900208

[Source2]
Type = Raw
Layer = None
SourceDir = E:\MNT compil decompil Test\out\
SourceFile = DEM-17-32430-27343-MASK.bin
GCS = WGS84
nCols = 6145
nRows = 8449
nBands = 8449
PixelIsPoint = 1
SamplingMethod = Gaussian
BandLayout = BIL
ByteOrder = Intel
SampleType = UINT8
XDIM = 0.021887
YDIM = 0.021887
ULXMAP = -61.237793
ULYMap = 14.900208

[Destination]
DestDir = E:\MNT compil decompil Test\out\
DestBaseFileName = DEM-17-32430-27343
UseSourceDimensions = 1

Step 3: Create a .hdr file to get this raw file readable by QGIS

In order to watch the Raw file (.bin) into Qgis I create manualy a .hdr file from this .inf file, here it is:

BYTEORDER I
LAYOUT BIL
NROWS 8449
NCOLS 6145
NBANDS 8449
NBITS 8
ULXMAP -61.237793
ULYMAP 14.900208
XDIM 0.021887
YDIM 0.021887

Into Qgis the binary file look like a stripped black square wit hlot of artefact...

I have to change some value into the .hdr file to get it work better:
NBANDS 8449 -> NBANDS 1
NBITS 8 -> 16

not bad but there are still two problems:
- the image that should be more square (EPSG:4326) seems to be streched verticaly.
- Lat/Long boundaries dimensions are completely disproportionate -> I have figured at top left corner with a Red square the bounds of my original geotiff dimensions.

After a lot of manualy "try and error" step, I could fit pretty much to my Geotiff boundaries if I change:
XDIM 0.021887 to -> XDIM 0.0000143046
And
YDIM 0.021887 to -> YDIM 0.0000107285

to sum up here is my final .hdr file:

BYTEORDER I
LAYOUT BIL
NROWS 8449
NCOLS 6145
NBANDS 1
NBITS 16
ULXMAP -61.237793
ULYMAP 14.900208
XDIM 0.0000143046
YDIM 0.0000107285

I Hope it could help you ;-)

just one or two more remark :
- could you add the possibility to change the default "nodata" value (black color for RGB imagerie and 0 value for elevation) which for now create a black outline around the Image to fit FSX Level boundaries. Or another possibility it's to add an alpha band to keep this part transparent.

thank you for all your hard work

Vogel

edit:
XDIM and YDIM should be easily calculated from QMID Level width and height no ?
XDIM = QMID width / nCols
YDIM = QMID height / nRows
right ?

edit2:
yes that was it !
XDIM = (QMID max longitude - QMID min longitude) / nCols
YDIM = (QMID max latitude - QMID min latitude) / nRows

and for my sample: calculation allows to be a little more accurate than "try & error"
XDIM 0.000014302786818551668022782
YDIM 0.000010727566224553201562315

 

[theisomizer]

Hi @Vogel, sorry for the delayed reply.

Would it be possible to share via DM or similar your BGL file with me? I am revamping bgldec and can take a look. Thanks

 

[rafaeldamasceno]

Hey there,

While working with worldlc.bgl from FSX I've found out that binary output has bytes set to 00 where they shouldn't be.
For example, the bitmap for QMID(7,0,0) is all ice, but the binary, instead of being full of 7A (122 in decimal, for Ice), has some bytes set to 00. I thought it was some sort of row separation, but the grids are allegedly 257x257 which is the resolution of the bitmap AND the size of the binary output, so something is definitely off. Both my tool and BGLViewer from @Patrick Germain decompress that QMID grid as full 7A.

Also, I was working with some FSX default .dem files which no other tool can really decompress and what I've noticed is that the output of the decompression doesn't seem to match all the existing TRQ1 records from the .bgl file. For example, dem0502.bgl seems to have TerrainElevation from QMID level 4 up to 11, yet the tool only outputs levels 4 to 8 and there's some missing grids even within those.

Perhaps I'm doing something wrong, but I've checked the documentation and while there is some mention of LOD settings, it appears they no longer exist in the latest versions.

 

[fs1]

Hey there,

While working with worldlc.bgl from FSX I've found out that binary output has bytes set to 00 where they shouldn't be.
For example, the bitmap for QMID(7,0,0) is all ice, but the binary, instead of being full of 7A (122 in decimal, for Ice), has some bytes set to 00. I thought it was some sort of row separation, but the grids are allegedly 257x257 which is the resolution of the bitmap AND the size of the binary output, so something is definitely off. Both my tool and BGLViewer from @Patrick Germain decompress that QMID grid as full 7A.

Also, I was working with some FSX default .dem files which no other tool can really decompress and what I've noticed is that the output of the decompression doesn't seem to match all the existing TRQ1 records from the .bgl file. For example, dem0502.bgl seems to have TerrainElevation from QMID level 4 up to 11, yet the tool only outputs levels 4 to 8 and there's some missing grids even within those.

Perhaps I'm doing something wrong, but I've checked the documentation and while there is some mention of LOD settings, it appears they no longer exist in the latest versions.

The DEM data and the landclass data is PTC encoded via different algorithms depending on the non null data. Have you been using BGLDEC? Is dem0502.bgl you are having issues with BGLDEC? Can you post your QMID and output? Thanks

 

[theisomizer]

While working with worldlc.bgl from FSX I've found out that binary output has bytes set to 00 where they shouldn't be.
For example, the bitmap for QMID(7,0,0) is all ice, but the binary, instead of being full of 7A (122 in decimal, for Ice), has some bytes set to 00. I thought it was some sort of row separation, but the grids are allegedly 257x257 which is the resolution of the bitmap AND the size of the binary output, so something is definitely off. Both my tool and BGLViewer from @Patrick Germain decompress that QMID grid as full 7A.

Also, I was working with some FSX default .dem files which no other tool can really decompress and what I've noticed is that the output of the decompression doesn't seem to match all the existing TRQ1 records from the .bgl file. For example, dem0502.bgl seems to have TerrainElevation from QMID level 4 up to 11, yet the tool only outputs levels 4 to 8 and there's some missing grids even within those.

Hi Rafael,

Thanks for bringing this up, and both issues have been identified. The latter has already been fixed in a more recent build yet to be published, it was an issue with stitch cells that started with null data due to a full missing data mask in the top left cell. I hadn't noticed the former as I am usually testing in stitch mode. That is happening for pointispixel types (257X257). For some background, when the image is stitched together it will only blit 256X256 for all interior chunks, with only the final row and column blitting a single extra pixel per. This is mathematically correct, as the final size of the image is 256^n + 1, not 257^n. This extra column /row can be viewed as forming the final bottom right vertices and edges binding the pixel array, and the data is used for blending the edge of an LOD patch with the next neighboring higher or lower LOD (which is coplanar to the 1st vertex of that patch in the y dimension). What happened is that in adding that logic for the stitch code, I broke the individual tiles mode, and the final 257th column is being written as 0. I can fix this, I just need to think of how to do it in a way that won't break the stitch functionality.

The DEM data and the landclass data is PTC encoded via different algorithms depending on the non null data. Have you been using BGLDEC? Is dem0502.bgl you are having issues with BGLDEC? Can you post your QMID and output? Thanks

Thankfully this is not a decompression problem, just how the output is being rendered to a file. FYI, the 8 bit raster types (everything except for DEM and Aerial) are not encoded in PTC and only every use the delta, bitpack, and LZ permutations. This is because PTC does not specify a single channel 8 bit compression mode, and even if one was to customize the algorithm to add it I imagine you would probably get lousy results. That sort of data lends itself better to a standard dictionary coder than the entropy + transform of an image coder.

Standby for an update.

Thanks,
Sean

 

[theisomizer]

New version is up with bugfixes for the above issues mentioned by @rafaeldamasceno and @Vogel. Thanks for the help!

Note that BGLDEC is now a 64-bit program, which will require the latest VC++ 2019 runtime which can be found here: https://aka.ms/vs/16/release/vc_redist.x64.exe

Future work is focused on multithreading, further output options, and (without breaking any NDAs) transcoding support for the upcoming MSFS.

 

[=Hollywood=]

Awesome work. I somehow lost some of my source photosscenery dev files I had used for FSX that are now useful for MSFS so this is a huge help!!!

 

[SprightlyOldMan]

Hi,
is now FractionBits supported?
I tried to decompress some dem files, but with bad results, like this:

Thanks!

 

[theisomizer]

Hi @SprightlyOldMan , it should be working for bin mode but I have only tested that previously using the SDK sample. Can you PM me your BGL file and I will try to debug? Thanks

 

[SprightlyOldMan]

Of course!
Thanks

 

[SprightlyOldMan]

Hi Sean,

I realized that I had made a mistake in hdr file:
i used NBITS 16 instead of 32

anyhow, last version (with incorrect elevations, too high 11.865 meters!) was this:

This is original bgl in TmfViewer:

with your new version elevations are correct, and result is:

There is (maybe) a problem in generated inf file

now is:

[Source1]
Type = Raw
Layer = Elevation
Channel_BlendMask = 2.0
SourceDir = E:\bgldec\dolomitiA1\
SourceFile = DEM-15-12996-3925.bin
GCS = WGS84
nCols = 9473
nRows = 5377
nBands = 5377
PixelIsPoint = 1
SamplingMethod = Gaussian
BandLayout = BIL
ByteOrder = Intel
SampleType = FLOAT32
XDIM = 0.000028610229492188
YDIM = 0.000021457672119141
ULXMAP = 10.371094
ULYMap = 46.878662



but as you can see, there is a vertical shrinkage and bad coordinates

i've made some test, and with this hdr:

BYTEORDER I
LAYOUT BIL
nCols 9473
nRows 5377
NBANDS 1
NBITS 32
ULXMAP 10.371094
ULYMap 46.878662
XDIM 0.0000526
YDIM 0.0000526

result is similar to TmfViewer and with correct coordinates (0.0000526 value is approximate):

summing up:
I think it is only necessary to check XDIM and YDIM values

thank you very much!!


Chris

 

[theisomizer]

Hi Chris,

Thanks for the detailed reply.

You are correct about the ULXMAP and ULYMAP being wrong in this instance, they are off by one QMID level for these files so the values need to be multiplied by 2. I will correct this bug in this build.

Regarding the stretch / skewing, I think the values are correct, see the post from Vogel on this page a few messages up for the rationale. Basically, the output data has still been resampled by the original compiler to the QMID grid, which has a 3:2 aspect ratio with 6 root cells.

It looks like you are trying to use this output directly in a WGS84 / EPSG:4326 view, which mismatches the source data. For example, even though the image isn't "stretched" in your image from TMFViewer, that is because TMFViewer is already rendering in a QMID projection in the given aspect ratio - if it rendered the data in WGS84 without an additional resample, the the grid lines shown above would be "shrunk" horizontally to make the entire world fit in a square (the inverse of your stretched screenshot from Global Mapper).

You can see in TMFViewer with your file, that the BR lat / lon corresponds (roughly, my mouse isn't pixel perfect here )to ULXMAP + XDIM * nCols; ULYMAP - YDIM * nRows which seems correct.

Unfortunately I am not very familiar with Global Mapper myself, but most GIS tools are capable of built in projection conversion which is why I think it is ok to "overfeed" it data where the number of pixel in the XDIM is 1.5X the number in the YDIM due to the screwy QMID projection system, while still maintaining the geographic reference datum. If it does need to be truly square in WGS84 you could use something like gdalwarp to resample the bin output yourself to WGS84 in the XDIM. This is what resample uses internally to project input data to QMID anyways, if people find it useful I could add a "desample" capability / option to bgldec itself pretty easily.

 

[SprightlyOldMan]

Hi Sean,
all clear: Global Mapper image "seem" vertically compressed, but really it is just different aspect ratio compared to TmfViewer.
With XMID and YMID values multiplied by 2, it's all OK!
Maybe I'm asking too much, but: in addition to .inf files, could you also add creation of .hdr files?
It would allow immediate opening of .bin files from both GM and QGIS.
Thanks so much for your support.
Chris

 

[theisomizer]

Sure, that shouldn't be difficult at all to write associated hdr files for each bin/inf. I'll get a new build up with that and the above fixes in the next few days.

 

[theisomizer]

New version 0.9.6 is up with the associated bug fixes and will now write .hdr files (in the GDAL recommended format) alongside .inf files. Apologies for the delays.

I'll take this opportunity to plug a few related things I have going on:

1. Included in this version is an alpha CGLDEC tool for MSFS. It is fairly complete in that CGL is an extremely simple format, but lacks some of the image processing capabilities I have in BGLDEC. Additionally there are some further data types like vector and building footprints that, once extracted from the CGL, still have to be figured out. I have been sitting on this code for the past year and originally decided against releasing it due to the active development on the sim. Asobo has indicated to me that they want sim mods to be done the way they end up exposing them through the sdk, as they don't want a bunch of incompatible things out there in the future as the sdk tools evolve. However, as far as CGL goes, since @Breadeater has also figured this out and published it on github, there's no reason for me not to move forward with releasing this. Furthermore, I suspect that not all of the terrain capabilities of the sim will ever be exposed through the sdk, which is just the nature of it being based on really advanced proprietary systems like Blackshark's PGG and the Bing Maps engine. Therefore the onus is really on us as the community to fill in the gaps where appropriate and where we are able to, while still respecting all of the awesome work these smart folks are doing and playing nicely with the capabilities exposed through the SDK.

2. For those that aren't aware, as I haven't exactly advertised it heavily, the core decompression routines of bgldec are available in a simple C++ interface under an MIT license here: https://github.com/seanisom/flightsimlib . I won't be able to make bgldec fully open source as it uses some image processing routines that I don't have the ability to redistribute in source form, but my aim is to eventually make this library complete enough where one could build something similar with just a little bit of effort. The library is very small now, but I'm already working on adding all of the BGL types to the library, including P3D and MSFS. I have a lot of other tool code that I've written over the years that I'd like to clean up and port to this repo, as there's very little open source available for the FS community in general, especially in C/C++ (no offense to the .Net folks). I think a common set of base libraries could really increase the availability and depth of sim design tools like the many that have been built by folks here, especially as we are entering this brave new "multi-sim" world.

3. Building off of #2, I have been working slowly on a set of 3D terrain rendering and design tools as I find working in 2D kind of limiting, and also am not a big fan of working "in-game" like the dev tools in MSFS, which also aren't usable for other sim versions. Right now I can do things like load resample BGLs, a little CGL data, and stream tin / buildings, dem, and aerial from bing maps or google, MSFS gltf and FSX+ MDLs, and render it with basic global illumination for a believable looking world in a simple D3D11 engine. It doesn't look exactly like it would in any sim without access to their shaders and engine, but it is plausible. Unlike these sims it is a mostly data driven engine / pipeline, so you could make it look really like anything you want. If I was to add the ability to edit and draw data on top of this and read and save common sim formats I feel this could be a useful generic tool for terrain and facility design.

My major limiting factor on all of this is time, as I am very busy at my career, but if I am able to find some collaborators and the time, is this something people would be interest in using? Useful?

Cheers

 

[Bavarello]

Hello theisomizer,
the bgldec v0.9.6 with the associated .hdr file, now works well in both QGis and Global Mapper. Thank you!
As for the cgldec v0.1.0, thanks for your time dedicated to the new CGL format, from a first a first analysis with a default file (120 \ dem330.cgl) I expected:

"Folder fs-base-cgl \ 020 \ dem112.cgl for example would have a quadkey prefix of 020112.
Therfore output 3112.dat, for example, would have a fully qualified quadkey of 0201123112. "

Instead, in output the filename is 8 characters in total and the Tile ID is different.

I apologize if I ask a trivial question ... but can this .dat file be read only with hexadecimal reader?

Cheers

 

[theisomizer]

Hi @Bavarello ,

I apologize for the confusion, that was a bit of an artificial example, not based on the actual file. You are correct in that the quadkeys in that specific file do appear to be 8 digits for a total Bing level 14 (d02011201000000, for example, as the second on the list). https://kh.ssl.ak.tiles.virtualearth.net/tiles/a02011201000000?.jpeg?n=z&g=8293

I don't know why this DEM data set does not cover the full area and doesn't have a full tile pyramid generated. I'll inspect the header and let you know if its decompressing correctly. If so, this becomes a mystery of the terrain system to figure out (and the answer my lay in @Breadeater 's python scripts.

Yes, all output can just be read in a hex editor. A dem tile is 257X257 for example, so a 65kb output (0x10208 total bytes) has a 7 byte header which looks to store a base value and byte stride, then the raw point data which, for this file size, appear to be 8 bit signed residuals. So each next value is additive to the previous value (0x03 = 3 meters higher than the last byte, 0xFE = -2 meters less, etc).

 

[theisomizer]

Version 0.9.7 is up. This is a tiny bug fix to resolve a crash for certain types of images that missing data in the middle (no anchor tile). Thanks @Maestr0 for the report.

Otherwise there are no other changes in this build. I'm continuing to evaluate MSFS to see what can be done there.

 

[theisomizer]

New BGLDEC version 0.9.8 is up. This is a minor release before 1.0 that corrects a couple of minor annoyances around bitmap processing, index types, and a little performance issue.

For CGLDEC, version 0.2.0 is up (same download). This is a major release that fixes several bugs with CGL files and adds the capability to extract vector data (`vecXXX.cgl`) tiles to geojson files. This is a frustrating missing part of the MSFS SDK, and I hope to have code to recompile them back to CGL ready soon.

Example decoded tile: https://gist.github.com/seanisom/1c7f10e55d50bc0f05efe2f14ad11407

More info on the wiki (WIP): https://github.com/seanisom/flightsimlib/wiki/CGL-vecXXX-File-Format

I will probably release BGLDEC 1.0 soon with multithreading for a modest performance boost. Given the direction of MSFS that will likely be the final version of BGLDEC in its current form, except for any critical bug fixes. As I've previously alluded to, from there I will be rolling the functionality of BGLDEC, CGLDEC, and another compiler/decompiler tool I have into a single tool that will be open source. A really compelling use case for said tool is conversion of scenery between these sim platforms. More on that soon once I am able to clean up some loose ends here.

Happy MSFSing!