Installing custom autogen configuration

[GaryGB]

Hi Folks

Gary -
Previous versions of NPP displayed the BOM character.
NPP now suppresses its display through applied styles, (Encoding Menu).
Look in a hex-editor, its still present.

IIRC, deleting the BOM,
then attempting loading into FSX caused the file's processing to fail.

Thanks Paul.

So, IIUC one need only save as "eXtensible Markup Language (*.XML)" output file type in the NotePad++ 'Save As Type' pick list ?

Everyone -
Regarding the XML merging -

Are any developers overwriting/reassigning any of MS's original entries ?
Or are they just appending their own extra entries ?

HTH
ATB
Paul

FYI: Jim Robinson was testing an interesting alternative workflow to "edit" autogen object GUID entries in custom local *.AGN files using a non-edited default AutogenDescriptions.spb ...here:

http://www.fsdeveloper.com/forum/th...d-rather-than-grouping-id.429331/#post-664781


Perhaps Jim might share an update on how this methodology is working ? (thanks, Jim ! )

GaryGB

 

[stonelance]

Hi Folks
Steve -
Between FSX betas and RTM, weren't some XML's encoding changed ?

I have no idea. I joined ACES right after FSX shipped. It makes sense if they were having problems with localization though. The weird thing is those files have text, but they are all just descriptions to make editing easier and as far as I remember were never even used at runtime.

 

[stonelance]

Thanks Paul.
FYI: Jim Robinson was testing an interesting alternative workflow to "edit" autogen object GUID entries in custom local *.AGN files using a non-edited default AutogenDescriptions.spb ...here:

It sounds like he just changed the annotations to point to specific vegetation entries instead of a group. This is totally supported as Arno said. Did the FSX annotator only let you annotate with a group? I think that probably got changed in the new annotator for Flight (which is way better than the FSX one).

 

[GaryGB]

It sounds like he (Jim Robinson) just changed the annotations to point to specific vegetation entries instead of a group. This is totally supported as Arno said.

Did the FSX annotator only let you annotate with a group? I think that probably got changed in the new annotator for Flight (which is way better than the FSX one).

IIRC, Arno had previously posted that one can increase the likelihood of placing "individual" trees in FS SDK Annotator by using very small Vegetation Polygons so that effectively when each such tiny vector polygon area is 'rendered to texture', at run time in FS, one might only see 1 (or very few) trees displayed at max Autogen slider density.



BTW: I'm also wondering if this might work (more precisely ?) for placement of individual trees etc. by using Ex: Whisplacer or Instant Scenery:

1.) Place intended "Autogen" objects in FSX at run time

2.) De-compile the scenery object placement BGL into BGLComp XML code

3.) Process the BGLComp XML code via a utility to convert it into autogen placement XML at the specified LOD-13 QMID grid coordinate X-Y offsets

4.) Compile the autogen placement XML code (via Ex: Annotator.exe) to convert it into a *.AGN file

5.) Use in FSX.Cfg:

[TERRAIN]
IMAGE_PIXELS_FOR_AUTOGEN_POLYGONS=1024



Some additional considerations cited by Thorsten Reichert:

"FSX does not use vectors to place vegetation polys, it processes those vectors into 1-bit images. Obviously, a 16x16 pixels image will not show a precise result of complex vegetations polys on a 1024x1024 landclass texture.
The higher the image_pixels value, the more precise the result, but of course this will hit your framerate. I found 256 to be a good compromise between precision and performance."

http://www.fsdeveloper.com/forum/threads/shapefile-with-holes-islands-in-it.351937/

http://www.fsdeveloper.com/forum/threads/accuracy-of-generating-autogen-trees.426360/


NOTE: I have found IMAGE_PIXELS_FOR_AUTOGEN_POLYGONS=1024 to work with no perceptible performance hit on my system.

According to ACES:

"Polyline vegetation is rasterized into a sampling bitmap.

The granularity of the rasterization is determined by the following setting in the ESP.cfg file:

[TERRAIN]
IMAGE_PIXELS_FOR_AUTOGEN_POLYGONS=16


The value of this setting is set at the minimum, 16, and can be any value up to 2048. The higher the number the more accurately a curved polygon will be rasterized. The value 16 indicates a rasterization of approximately 80 meters down to approximately 0.5 meter accuracy for the 2048 setting. There is a performance penalty with the time it takes to rasterize a region, and the memory required for the resulting sampling bitmap."

http://msdn.microsoft.com/en-us/library/cc526979.aspx#AnnotatingTextures

http://msdn.microsoft.com/en-us/library/cc526979.aspx#AutogenValidationTools

http://msdn.microsoft.com/en-us/library/cc526979.aspx#Editing_the_AutogenDescriptions.xml_File

http://msdn.microsoft.com/en-us/library/cc526979.aspx#CreatinganSPBfile


AFAIK, "IMAGE_PIXELS_FOR_AUTOGEN_POLYGONS=1024" (aka "IPFAP") correlates with a 1.2-Meter per pixel ground texture resolution (FSX default land class texture resolution), so one might extrapolate from the above Autogen SDK doc statement using typical QMID grid quad sizes:

IPFAP = Meters on ground per Pixel
   16 = 76.4 Meters
   32 = 38.2 Meters
   64 = 19.1 Meters
  128 =  9.6 Meters
  256 =  4.8 Meters
  512 =  2.4 Meters
 1024 =  1.2 Meters
 2048 =  0.6 Meters

However, it would also be very interesting to see a freeware "re-interpretation" version of the MS-Flight Autogen Annotator ...for FSX !

GaryGB

 

[JRobinson]

Perhaps Jim might share an update on how this methodology is working ? (thanks, Jim ! )

Sort of off topic for this thread but yes I've been using this approach a lot, most of what's seen around houses in my Inverness thread have been annotated this way while I still use the groupings and larger veg rectangles for the outlying areas without houses. One complaint we often hear is "the trees are too big and they dwarf the houses" so this is a good solution for that problem too. I haven't found a way in the annotator to use a ModelEntry GUID directly, only regionalizations or groupings. Once you've hacked at least one autogen tile however, it's possible to copy & paste these specific vegetation objects in the annotator just like anything else (I use tiny veg rectangles BTW which represent just a single tree as opposed to polys).

Additionally it's possible to select/copy the veg rectangles in the annotator and paste them into Notepad as text. This can be saved and used on other PRs although sometimes depending on the resolution and size of the PR those annotations don't necessarily retain their size and spatial relationship with one another. It's possible to set the GUID in "S" (select) mode by simply selecting one of the individual trees and then switching to vegetation mode (V) and drawing some new veg rectangles to copy & paste - the new veg rectangles will retain the GUID. This is a way around the size/spatial relationship problem with saving as text although not exactly elegant, lol.

I've done most of my recent annotations with the P3D2 annotator incidentally, it seems to be no different than the FSX annotator with regard to the ModelEntrys (or anything else for that matter) and the resulting autogen seems to work fine in FSX.

Jim

 

[ollyau]

(I use tiny veg rectangles BTW which represent just a single tree as opposed to polys).

I recall you (or someone else) mentioning that previously, but couldn't find an exact source. This is even more off topic, but I was wondering if Arno could make the DETECTFEATURES step in scenProc output small vegetation rectangles for each tree instead of larger irregular polygon areas. The small rectangles seem more precise than the polygons, and it seems appropriate since scenProc seems to detect each tree individually (based on the texture filter editor display).

 

[arno]

Everyone -
Regarding the XML merging -

Are any developers overwriting/reassigning any of MS's original entries ?
Or are they just appending their own extra entries ?

It would be a bad idea to change MS entries. So I hope that developers only add their own. But from the MergES issues I have been looking at for now, I learned that sometimes developers update their own entries, so there are different versions floating around with the same GUIDs in different products. That's the only situation to handle, but that's the developers problem

 

[arno]

I recall you (or someone else) mentioning that previously, but couldn't find an exact source. This is even more off topic, but I was wondering if Arno could make the DETECTFEATURES step in scenProc output small vegetation rectangles for each tree instead of larger irregular polygon areas. The small rectangles seem more precise than the polygons, and it seems appropriate since scenProc seems to detect each tree individually (based on the texture filter editor display).

We are going a bit off-topic here, maybe we should do the vegetation discussion in a new thread and keep this one for merging configurations?

From my experience the detect feature logic doesn't really detect individual trees. Especially for a forest area you won't get each tree detected, you just get the area where the trees are. So that's why it outputs polygons now.

But I guess you could try to filter out the polygons that have only 4 vertices and have a small area (exact area depends on your border setting). These could then be represented by a rectangular vegetation object and the others with a polygonal vegetation object.

 

[GaryGB]

Sort of off topic for this thread but yes I've been using this approach a lot, most of what's seen around houses in my Inverness thread have been annotated this way while I still use the groupings and larger veg rectangles for the outlying areas without houses. One complaint we often hear is "the trees are too big and they dwarf the houses" so this is a good solution for that problem too. I haven't found a way in the annotator to use a ModelEntry GUID directly, only regionalizations or groupings. Once you've hacked at least one autogen tile however, it's possible to copy & paste these specific vegetation objects in the annotator just like anything else (I use tiny veg rectangles BTW which represent just a single tree as opposed to polys).

Additionally it's possible to select/copy the veg rectangles in the annotator and paste them into Notepad as text. This can be saved and used on other PRs although sometimes depending on the resolution and size of the PR those annotations don't necessarily retain their size and spatial relationship with one another. It's possible to set the GUID in "S" (select) mode by simply selecting one of the individual trees and then switching to vegetation mode (V) and drawing some new veg rectangles to copy & paste - the new veg rectangles will retain the GUID. This is a way around the size/spatial relationship problem with saving as text although not exactly elegant, lol.

I've done most of my recent annotations with the P3D2 annotator incidentally, it seems to be no different than the FSX annotator with regard to the ModelEntrys (or anything else for that matter) and the resulting autogen seems to work fine in FSX.

Jim

Thanks for that update, Jim.

I have replied to the above quoted post, and to the original thread on this related topic of vegetation placement options ...in your thread here:

http://www.fsdeveloper.com/forum/th...delentry-guid-rather-than-grouping-id.429331/

GaryGB

 

[stonelance]

to work around the annotator not being able to annotate single trees, you could create groupings that only contain a single tree, but it would be slight less efficient. You would also have to deal with the file merging issue.

 

[GaryGB]

to work around the annotator not being able to annotate single trees, you could create groupings that only contain a single tree, but it would be slight less efficient. You would also have to deal with the file merging issue.

Hi Steve:

When you state that autogen groupings containing only a single tree would be slightly less efficient, do you mean run time performance in FS ?

Is there a limit on how many autogen groupings that only contain a single tree which can be created in [FS install path]\Autogen\Default.xml ?

If there is not a absolute numeric limit on same, is there a "practical" limit, which if exceeded, adversely affects FS run time autogen rendering ?

That is, of course, only referring to "availability" of the groupings as choices to pick from within the Default.xml file, aside from whether any are actually placed via a local AGN file.

I'm wondering whether there would be an undesirable consequence to creating additional "single" tree groupings in the Default.xml file for each FS default tree type and size, so that one could place individual trees for various scenery environments where single trees might be appropriate as landmarks which exist in the real world, and/or in high resolution aerial imagery where seen visibly casting shadows on the photo-real ground texture etc..

The reason I ask is, questions have been raised over the years in the FS Community as to whether there are un-documented absolute and/or practical limits on how many total entries may be created for various FS files / folders such as:

* [FS install path]\Autogen\Default.xml

* [FS install path]\Autogen\*.spb files

* [FS install path]\Terrain.Cfg

* 'active' User copies of Scenery.Cfg

* 'active' User copies of DLL.xml

* Total Gauges / Modules listed in "active' User copies of FSX.Cfg in [Trusted] section ...with "Permission" to run automatically

* Total Gauges in [FS install path]\Gauges

* Total "Liveries" within various Aircraft.Cfg's in [FS install path]\SimObjects\Airplanes folder chain

* Total Aircraft.Cfg's and/or Panel.Cfg's within [FS install path]\SimObjects\Airplanes folder chain



Thanks in advance for any additional clarification and insight you might offer on this.

GaryGB

 

[stonelance]

I don't know about limits, but the slight performance hit would probably manifest mostly as slightly increased load time and memory usage.

Anyway, we are still getting a little off topic

 

[arno]

Hi,

Would the vegetation groups not go in the spb file for vegetation instead of default.xml? From default.xml you could only place them as library objects, that would be even worse for performance I think.

 

[GaryGB]

Hi all,

The more I'm exploring autogen, the more possibilities I see. But there is a catch and that is the problem of updating the autogen configuration when distributing scenery. To make sure you don't conflict with other developers also making changes to these files some kind of merge action would have to be done upon install of the addon.

Are there any easy to use tools for that already? I would expect that the big publisher that release such addons have such tools. But are there also tools widely available to everybody?

What I have found until now is the sdk tool to turn XML into spb and the freeware/open source spb2xml tool to go the reverse way. With those tools it should be possible to turn the autogen configuration on the users system into XML, merge in your new XML elements and classes and then compile back to spb.

Is this how others do it as well? And can we even include the sdk tool to make the spb files in our installers?

Any ideas would be welcome...

Yes, Earth Sims developed a utility called MergES which does indeed merge autogen files. It is included in all their sceneries and run on install to avoid their sceneries overwriting existing autogen files. They also offer it separately as a free utility for anyone else (including devs) to use, just acknowledgement required

I've used it frequently whenever I install something new.

Hi,

Great, I didn't know that tool. Only problem is that it doesn't seem you can find it easily . I only found the manual:

http://earthsimulations.com/MergES.pdf

It seems you first need to download some other installer tool to get it. I'm now on the iPad so can't even try that. If such a tool is to be widely used by developers, I think it should be available more easily.

Hi Arno

Yes it does. I've had a look myself but couldn't see why it had stopped working. So I'm trying to get Lars ( who made it ) to have another look, but he is very busy just lately. If he can't find time I'll be happy to send you the source code, but had better see if Lars can help first though. It's in c#

Definitely not! We've only made the compatibility files because MergES stopped working. MergES is a much better solution. It's described HERE if you want to see how it works ( or did work should I say )

PS: ..Like your new forum.

Hi Darren,

Yes, I had found the MergES manual already. Looks like a nice tool, exactly what we need.

Let me know if I can help debugging. Would be happy to help. Much better to get this tool working again then making another one.

Hi Arno:

As I am now reviewing and researching issues related to the OP (and 'evolving sub-topics') of this thread from the very beginning, I encountered another version of the MergeES manual ("Printer-Friendly" ...and IMHO, more viewer friendly !) which you and others here may wish to download as an alternative to the above quoted link:

http://earthsimulations.com/MergES-Printer-Friendly.pdf

"MergES will handle all the different types of autogen file as listed here:

* Default.xml

* Autogendescriptions.spb

* Roofdescriptions.spb

* Materials.spb

* Extrusions.spb

* TerrainAutogenClassDescriptions.spb"



BTW: Arno, as OP, would you prefer the future direction of this particular thread be 'limited' to only *.SPB files rather than anything involving the Default.XML file or Terrain.Cfg file, which IIUC, are also involved in "Autogen configuration" for display (...or non-display) at run time ?

Hope this helps !

GaryGB

 

[Tatu]

Hi.

Thank you for working in this matter but I still have to ask if you have found any working solution?

I have a huge photo real area around the airport EFRO (Rovaniemi in Finland) with my own forest. That means modified AutogenDescriptions file and also few entries in Default.xml.
The scenery is not released yet except for a limited amount of people but swapping those autogen files is pain in the "you know where". ;-)

Work in progress but you can take a look here: www.tatukantomaa.net/EFRO

Also I tried to find that earthsimulations MergES program but did not find it.

Thanks.

Tatu

 

[stonelance]

I have been working on my SimProp library. I have decided to completely remove its dependency on zip libraries and my other Flight related libraries. It will now only depend on assemblies from the .NET runtime. I will be releasing it soon, and it should be possible to use it in installers to easily merge the spb files from FSX (and possibly P3D if they haven't changed the format). I may consider making it open source as well, but the assembly should be easy enough to use by itself.

 

[ollyau]

Sounds good; looking forward to using it. Open source would be especially cool!

 

[stonelance]

I added a project to CodePlex. I also added debug and release binaries in the download section if someone wants to try it without playing with the code: https://simprop.codeplex.com/
This is my first time creating a project on there, so if you see anything wrong, please let me know.

There is some sample code in the sources section for merging documents generically, but it is incomplete. There may be special code required to merge autogen description docs, in order to combine the groupings together in a way that FSX can read them.

 

[ollyau]

Added issue #1. Initializing SimProp throws a DirectoryNotFoundException because Symbols is instantiated with null as the argument, which sets symbolPath to [executing directory]\SimPropSymbols\Flight. Line 25 in Symbols.cs then tries to enumerate the files in a directory that doesn't exist.

Edit: Forked the project with the intention to fix it, but I ran into another couple issues.

When loading the FSX propdefs, the Symbols constructor throws an exception when loading propautogen.xml with an inner exception message of "An item with the same key has already been added."

When loading the Flight propdefs and using the TestBinaryRead unit test, symbols.GetSymbolDef(idToObjectSizeTable.GetKeyByIndex(guidIndex)) (line 103 in SPB.cs) returns null, causing a NullReferenceException on line 106 of SPB.cs.

Edit 2:

Did some additional debugging and found the problem loading propautogen.xml:

includeGuid for both propworldbase.xml and propgroups.xml are {63370acd-3f33-446a-bb06-4c96dd56396a}. After checking propautogen.xml that seems to be the case. I think you mentioned there were some errors in the default FSX propdefs before.

 

[stonelance]

Yeah, that definitely seems wrong. I guess the question is what is the FSX simprop loader doing in that case? Is it just ignoring guids, and only going off filenames? I believe I did have it working at some point so it could be a regression in my other refactoring. I did change a bunch how the includes are processed.

My guess is the Ids are unneeded, and can just safely be ignored.

Edit: I changed the code to ignore the guids in the includes and got past that part. I ran into a couple other issues:

In propui.xml:
- BkgndTransRectType is an enum type but doesn't actually specify any enum values. I don't know what this behavior would result in.
- ListviewColumn element has a typo ("SetDev" instead of "SetDef"), and results in my validation code failing because one of the referenced properties didn't exist.

Do you think I should put hacks in the code to work around these issues, or should users be required to fix up the simprop docs before using with the tool?

Edit: For the 3rd issue you ran into what was the specific scenario you tested? That error is expected if you aren't using the correct symbols for the file that is being loaded. How did you create the symbol folder for Flight? Make sure you follow the instructions on the document page on CodePlex. You need to unzip the propdefs from the pak files in a specific order.