P3D v4 - different encodings & configuration file list

[CitMusJoe]

In FSX I never had to worry about encodings and the like. But now I changed to P3D v4 due to the advantages of a 64bit architecture.

As "n4gix" already pointed out in an earlier thread ( http://www.fsdeveloper.com/forum/threads/beware-p3dv4-scenery-cfg-encoding-changes.440551/ ) the correct file-encoding at least for the scenery.cfg file after it's first use is UNICODE.

As I'm a casual developer only I don't know as much as I obviously should about encodings. I searched the web, but there are still some issues re. encoding I'd like/need to clear up.

Can someone please explain following questions/issues about file encodings used in P3D v4?

1. Is unicode the same as UTF-8 in P3D v4? There are so many differing explanations out there...
(e.g.: https://stackoverflow.com/questions/643694/utf-8-vs-unicode )
2. Is ANSI really identical to windows-1252?
3. And many other obviously duplicated namings for the same encoding?
4. Has anyone a list of all names of encodings I need in P3D 4?
5. why do some text-editors show different encodings for the same file?
6. Which text-editor do _you_ use?
7. When saving a confiruration file - besides an *.xml file, where the encoding is defined in the first line - do I have to save it always with the same encoding as when I opened it before editing?
7. Is there a list of all configuration files for P3D v4 with it's corresponding path _and_ encoding for P3D v4? Could'nt find any in P3D v4 SDK

Any hints are highly appreciated!

 

[Paul Domingue]

1. UTF-8 is the most commonly used version of UNICODE.
2. Windows-1252 is another name for ANSI

That is the extent of my knowledge for now. I suggest you use notepad++ which will allow you to set any type of encoding you wish to use.

 

[CitMusJoe]

Thanks for clearing up some of my questions, Paul.

 

[lmoelleb]

A bit late, I know - but now I am here I'll try to clarify a few things.

1. Is unicode the same as UTF-8 in P3D v4? There are so many differing explanations out there...
(e.g.: https://stackoverflow.com/questions/643694/utf-8-vs-unicode )

No, it is not the same. Unicode is basically just a list that assigns a numerical value to a character. Unlike other such "code pages" Unicode attempts to address all languages (including things like Runes etc).

UTF-8 is a way to store these numbers as an array of bytes (this is called an encoding as it encodes the numbers to bytes). Other encodings like UTF-16 and UTF-32 will store the exact same numbers using different byte patterns.

For example, take the character ɸ - Unicode 0x0278 (decimal 632).

Obviously this character can't be stored in a single byte. Instead multiple bytes are used. Based on the encoding, you will get the following bytes (listed as hex values) in the file to represent this character:
UTF-8: C9 B8
UTF-16 (big endian): 02 78
UTF-16 (little endian): 78 02
UTF-32 (big endian): 00 00 02 78
UTF-32 (little endian): 78 02 00 00

In memory all modern computers used UTF-16 (either little or big endian) and they will convert to/from this no matter what encoding is used in the file.
UTF-8 appears to be the most common encoding used with files and network traffic.

2. Is ANSI really identical to windows-1252?

No. ANSI is not a single "code page" - which is one of the things that makes it so problematic to use.

Windows-1252 is the default ANSI code page used by default on Windows systems in most (all?) western languages. A Japanese, Russian, Greek, ... system would have another default ANSI code page. Which basically means they do not agree on what character is represented in a file. To make it more complicated you can change the ANSI code page on your computer from the settings. So you can basically save a text file, change a setting, and then be unable to load the file correctly if you use ANSI.

3. And many other obviously duplicated namings for the same encoding?

Not sure which duplicates you are referring to - UTF-7,16,32?.

4. Has anyone a list of all names of encodings I need in P3D 4?

I don't. But so far I have only seen UTF-16 and ANSI as far as I remember. I just open it in Notepad, select Save as and see which encoding it is set to. You just need to know that it calls UTF-16 "Unicode" - which is somewhat confusing at UTF-8 is obviously also Unicode.

5. why do some text-editors show different encodings for the same file?

Hard to say. They should all be able to recognize UTF-16 as it is supposed to have two bytes identifying it. It is however not always possible to tell if a file is UTF-8 or not. Typically the editor can guess it if it contains characters outside the ASCII range as the byte patterns in UTF-8 follows some strict rules - but still it would be a guess. Like UTF-16 there might be a few bytes at the start of the file indicating it is UTF-8, but this is not required.

6. Which text-editor do _you_ use?

Notepad. And Visual Studio if I need to read/write encodings outside UTF-8, UTF-16... but that hardly ever happens. Years back I worked in software localization. Back then UltraEdit, Notepad++ where popular but unlike Notepad they did not handle Unicode. I am sure it has been corrected, but I never bothered to check again. Correct handling of text was more important to me than 10000 features, and if they did not agree, well... bye....

7. When saving a configuration file - besides an *.xml file, where the encoding is defined in the first line - do I have to save it always with the same encoding as when I opened it before editing?

I would recommend it, but it really is up to the program reading. Even if your flight simulator handles it, you have to consider if it will work on a flight simulator running on for example a Japanese system. And as you share configuration files with other add-ons, you have to consider some of these can fail even if FS still works.

7. Is there a list of all configuration files for P3D v4 with it's corresponding path _and_ encoding for P3D v4? Could'nt find any in P3D v4 SDK

I do not have it. I just checked the files I needed to generate and they where UTF-16, so that is what I generate them with.

There has been Unicode content in FS for quite a while by the way - but it has only been me dealing with it so far.
The text used by the ATC system (so everything you see written in the ATC window as well as pilot voice names) are stored in UTF-16 - and has been since it was introduced in FS2002.

 

[tsgucci]

I ran into this issue just recently. Our SceneryConfigurator saved the new scenery.cfg in UTF-8 format which was never an issue to any simulator.
Now Prepar3D expects the scenery.cfg in UCS-2 LE BOM (at least the original is saved in that format). Basically the simulator was working but our DLL (which places the simobjects) could not work with it.
Our Scenery Configurator is written in Visual Basic I use this code (in case of P3D v4):

Using sw As New IO.StreamWriter(CFGDataRoot & "scenery.cfg", False, System.Text.Encoding.GetEncoding("ISO-10646-UCS-2"))

I've found this on Microsoft's site:
https://msdn.microsoft.com/en-us/library/aa332097(v=vs.71).aspx
See the code page.

My question is: what format is expected in dll.xml, fsx.cfg?