BGLDEC - A resample BGL (and CGL) DECompressor

[rhumbaflappy]

Hi Sean.

Any updates for us?

 

[theisomizer]

Thanks Dick. I should have an update ready to go in about a week that can write vector data.

 

[Bavarello]

Thanks Dick. I should have an update ready to go in about a week that can write vector data.

It's been a while since CGLdec (0.2.130.0) doesn't give any news.

In the meantime, there are some things I would like to share here. Wanting to create the stats.txt I encountered some difficulties, but in the end I managed to get a file of 2,335,417 lines.

I can't set in the parameter -o "M: \CGL" only accepts -o "M: \"

Another problem with the command
cgldec extract -f "H: \ MSFS_2020 \ Official \ OneStore \ fs-base-cgl \ CGL \ 032 \ dem010.cgl" -t dem -p 032010 -q 013113 -o "M: \"

there is no output.

 

[rhumbaflappy]

I think the first problem is that you need to create the CGL folder in M: before you add anything to it.
The second problem is that -p 032010 -q 013113 doesn't exist. I think it would need -p 032010 -q 0131 so...

cgldec extract -f "H: \ MSFS_2020 \ Official \ OneStore \ fs-base-cgl \ CGL \ 032 \ dem010.cgl" -t dem -p 032010 -q 0131 -o "M: \"

 

[theisomizer]

Hi Giovanni, Dick is correct on the above. The tool does not create the folder, make sure it exists. For your target file above, the max quadkey level is 10, you can see this by dumping all quads in that file:

cgldec decompress -f "C:\FS\MSFS\Official\Steam\fs-base-cgl" -t dem -p 032010 -o "C:\CGL"

As a convenience you don't need to specify the full filename when using the prefix, just the base CGL directory, so this is equivalent to Dick's command (with my paths):

cgldec extract -f "C:\FS\MSFS\Official\Steam\fs-base-cgl" -t dem -q 032010013113 -o "C:\CGL"

They have made a significant amount of changes to the sim internals in the past year, which can be discouraging, but a recent quick check shows that vector tiles are thankfully still compatible with only very minor binary differences.

It's been a while since CGLdec (0.2.130.0) doesn't give any news.

Time makes fools of us all. Unfortunately I don't have much news to share, as life has often gotten in the way, other than that writing vector tiles is in progress, and something that I very much still want to complete. It actually is a significantly more complex task than initially anticipated, however it is still possible.

For unrelated reasons, for anybody that uses bgldec, I have replaced the PTC compression library with a fully open source decoder which means flightsimlib is now platform independent and bgldec is almost stripped of closed code.

Hopefully I will have more to share on cgldec and MSFS soon.

 

[theisomizer]

A quick progress update: Good news and bad news!

The good news is I can read at least building footprint geometry from arbitrary bldo/bldn tiles. These are called ".bld" files in Blackshark parlance, and the file format is an unbelievably convoluted proprietary mess.

The bad news: first, there is a lot of additional metadata there that is going to be even more convoluted to decode. I can perhaps get the code to a point where I can write some arbitrary bytes in some of these areas and observe the effect in-sim, but this is really going to be a slog. But first, there's an even more pressing problem:

The 16-bit fixed point grid is stretched/skewed over the tile bounds (and as you'd expect give the above, in longitude the tiles overlap, and have a gap in latitude). The amount of stretch/skew depends on Lat/Lon of the tile, but is not 0 at the equator as you would expect if this was related to projection distortion. I've tried countless projection techniques and I can get close, but have not found an algorithm yet that perfectly fits without manually tweaking a single tile, which obviously does not scale. This doesn't affect vec* tiles, which project fairly cleanly to web mercator - however the sim reads these directly instead of the PGG engine, two different codebases. To compound this, it's not really just a 16-bit grid, the algorithm selectively expands vertices to 32-bit at tiles bounds (for normalized values below 0.0, or greater than 1.0) to handle buildings with coordinates that overlap a tile boundary.

I'm going to keep investigating and hopefully figure something out. But I've been doing terrain programming for years, even decades now, and believe me I am stumped.

Anyways, thanks for your patience and cgldec updates to follow soon!

 

[hesicong]

A quick progress update: Good news and bad news!

The good news is I can read at least building footprint geometry from arbitrary bldo/bldn tiles. These are called ".bld" files in Blackshark parlance, and the file format is an unbelievably convoluted proprietary mess.

View attachment 83473

The bad news: first, there is a lot of additional metadata there that is going to be even more convoluted to decode. I can perhaps get the code to a point where I can write some arbitrary bytes in some of these areas and observe the effect in-sim, but this is really going to be a slog. But first, there's an even more pressing problem:

View attachment 83474

The 16-bit fixed point grid is stretched/skewed over the tile bounds (and as you'd expect give the above, in longitude the tiles overlap, and have a gap in latitude). The amount of stretch/skew depends on Lat/Lon of the tile, but is not 0 at the equator as you would expect if this was related to projection distortion. I've tried countless projection techniques and I can get close, but have not found an algorithm yet that perfectly fits without manually tweaking a single tile, which obviously does not scale. This doesn't affect vec* tiles, which project fairly cleanly to web mercator - however the sim reads these directly instead of the PGG engine, two different codebases. To compound this, it's not really just a 16-bit grid, the algorithm selectively expands vertices to 32-bit at tiles bounds (for normalized values below 0.0, or greater than 1.0) to handle buildings with coordinates that overlap a tile boundary.

I'm going to keep investigating and hopefully figure something out. But I've been doing terrain programming for years, even decades now, and believe me I am stumped.

Anyways, thanks for your patience and cgldec updates to follow soon!

Thanks for the update, you are still working on it. Can you open sourced the code and let us know how to figured this out?

 

[=Hollywood=]

A quick progress update: Good news and bad news!

The good news is I can read at least building footprint geometry from arbitrary bldo/bldn tiles. These are called ".bld" files in Blackshark parlance, and the file format is an unbelievably convoluted proprietary mess.

View attachment 83473

The bad news: first, there is a lot of additional metadata there that is going to be even more convoluted to decode. I can perhaps get the code to a point where I can write some arbitrary bytes in some of these areas and observe the effect in-sim, but this is really going to be a slog. But first, there's an even more pressing problem:

View attachment 83474

The 16-bit fixed point grid is stretched/skewed over the tile bounds (and as you'd expect give the above, in longitude the tiles overlap, and have a gap in latitude). The amount of stretch/skew depends on Lat/Lon of the tile, but is not 0 at the equator as you would expect if this was related to projection distortion. I've tried countless projection techniques and I can get close, but have not found an algorithm yet that perfectly fits without manually tweaking a single tile, which obviously does not scale. This doesn't affect vec* tiles, which project fairly cleanly to web mercator - however the sim reads these directly instead of the PGG engine, two different codebases. To compound this, it's not really just a 16-bit grid, the algorithm selectively expands vertices to 32-bit at tiles bounds (for normalized values below 0.0, or greater than 1.0) to handle buildings with coordinates that overlap a tile boundary.

I'm going to keep investigating and hopefully figure something out. But I've been doing terrain programming for years, even decades now, and believe me I am stumped.

Anyways, thanks for your patience and cgldec updates to follow soon!

Sean your work is incredible. I'm looking forward to being able to compare the building footprint data in sim, especially around monument valley to the stuff MSFT released on github, as there seemed to be a few false positives in the github data but not in the MSFS dataset.
Also curious if a neighborhood near me has the building footprints actually encoded as no autogen buildings show up in some locations. I noticed this at Rosamond Skypark, no buildings in sim but when I turned on dev mode to rework the airport the autogen buildings appeared.

 

[=Hollywood=]

Hey Sean, I was just digging through the files and for the landclass you're correct that they're WEBP format. It's RIFF is WEBPVP8.

You can open the data directly in Google Chrome as it uses the same file format as Google Maps. But you can also rename the .dat as .webp and Windows will generate thumbnails but the files can also be opened in Adobe Photoshop and any other software that can read .webp files.

So far I've seen the landclass classifications of:

(0,20,0) green
(0,50,0) green
(0,70,0) green
(0,100,0) green

on the few files I've opened. Would love to find a way to convert the quadkeys back up to the files so they can be loaded into GIS software.

Here's a screenshot of the landclass files:

 

[PK-LKP]

Hello all...

First I must say thank you for this helpful utility.

I have few simple questions. I have tried to decompile default P3D landclass (worldlc.bgl) and some very old landclass addon to either .bin or .bmp and its corresponding .inf file. Can someone help me how can I convert those files to .raw file so I can append it to sbuilderx or edit it in Photoshop? I have no problems recompiling the output .bin and .inf files to bgl with resample, I just don't understand how to modify the landclass.

Thank you for your help.

Sorry for my bad english

 

[theisomizer]

Hi @PK-LKP! If you rename the .bin file to .bil and use the .hdr file it spits out, you should be able load it in QGIS: https://www.qgistutorials.com/en/docs/open_bil_bip_bsq_files.html

Likewise, the file itself should be compatible with SbuilderX .raw if you rename it, although I haven't tried this workflow myself.

Does that meet your needs? I could explore adding Photoshop Large Document Support if not, but it is not a pleasant format to work with.

 

[PK-LKP]

Hi @PK-LKP! If you rename the .bin file to .bil and use the .hdr file it spits out, you should be able load it in QGIS: https://www.qgistutorials.com/en/docs/open_bil_bip_bsq_files.html

Likewise, the file itself should be compatible with SbuilderX .raw if you rename it, although I haven't tried this workflow myself.

Does that meet your needs? I could explore adding Photoshop Large Document Support if not, but it is not a pleasant format to work with.

Thanks for the reply. I have tried renaming the output.bin to output.raw directly and opening it in photoshop, then inputted image width and height with numbers of nRows and nCols from output.inf file and it loaded perfectly! Now I just have to change the RGB value and recompile it. Thank you very much for your help.

PK-LKP

 

[PK-LKP]

Hi again!

I have one more question. Is it normal that extracted bmp from photo scenery has some color degradation (e.g. some dark area become completely black)?

Thanks.

 

[theisomizer]

Hi @PK-LKP - what does the photo BGL look like in TMFViewer? The BGLDEC output should exactly match the image in TMFViewer - if that also is pure black it is as a result of the lossy image compression algorithm and you could try to rebuild as lossless or a smaller quantization step (CompressionQuality setting in the .inf). If the two do not match, it is a bug with BGLDEC - you can PM me the BGL and I can take a look.

 

[PK-LKP]

Thanks again for the reply!

The bgl I'm trying to decompile is default TNCM island (/scenery/cities/StMaarten/StMaartenAerial.bgl). I have attached two images for comparison below.

Original bgl in tmfviewer:

Extracted (highest LOD) bmp in photoshop:

As we can see some of green and blue color became pure black. If I recompile it it looks similar to the bmp not the original bgl regardless compression quality value.

 

[theisomizer]

Hi @PK-LKP, thanks for the report! I can reproduce with the TNCM bgl and it looks as above. It looks like this is a bug in the decompression algorithm where it is not reading a single band of coefficients. I will debug and fix this and post a new build.

 

[theisomizer]

Bug fix version 0.9.9 is up in the top post. Please note this now requires the VS2022 redistributable which you likely already have installed if you are using MSFS or P3D6: https://aka.ms/vs/17/release/vc_redist.x64.exe

This is pretty embarrassing - I fixed this bug a while ago in flightsimlib and never posted a new version of BGLDEC here with that. This is an interesting one if you're interested in image compression- PTC encodes in a YCoCg color space for better compression performance. While the luma bitstream is always present, there are instances where either and/or both of the chroma bitplanes are empty. If that is the end of the data stream for the chunk, the code worked fine, but if the first was empty and there was subsequent HF data that was missed. This is a valid case and was not present in my test data set previously (it is now).

https://github.com/seanisom/flightsimlib/commit/6763027d63b2062b9ceea0526a198fc655e76c7f

I have some other fun things that I'm working on for bgldec 1.0, cgldec, and more but I don't want to hold this fix up for that and will share more soon.

 

[PK-LKP]

Wow that's pretty quick response! Thank you very much sir! I will try it soon. I hope the fix will be useful for others too.

 

[PK-LKP]

I can confirm that the latest version works perfectly! Thank you very much for your hard work.

 

[Draken]

Hello!
I'm processing a photo terrain BGL that I created for P3D v4.
I de-compiled it with the stitch parameter and I got the following .inf file:

GCS = WGS84
nCols = 21760
nRows = 20224
PixelIsPoint = 0
SamplingMethod = Gaussian
BandLayout = BIL
ByteOrder = Intel
SampleType = UINT16
nBands = 3
XDIM = 0.000003576278686523
YDIM = 0.000002682209014893
ULXMAP = 8.649902
ULYMAP = 45.560303

since I would like to edit (i.e. black some areas in such a way they aren't compiled) and recompile it through SbuilderX 3.15, I need to position the map precisely.
I guess that ULXMAP = 8.649902 ULYMAP = 45.560303 give the cordinates of the north eastern corner.
How to calculate the southest western corner with the above data?
I need this info to calibrate the image in SbuilderX.

Moreover is there any way to get a txt file for each bmp extracted fro bgl that ha the same instruction format to be used to compile them again by mean of SbuilderX?

Thanks!