1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

We NEEEED a multithreaded resample!!! ARRRGHHH!!!

Discussion in 'Terrain Design General' started by MOUSY, 20 Mar 2017.

  1. MOUSY

    MOUSY Resource contributor

    Joined:
    14 Aug 2008
    Messages:
    292
    Country:
    dominica
    Complete shot in the dark... but how hard it would it be to create a tool to somehow magically enable multiple resample instances to run simultaneously and compile into 1 BGL?

    Or an even longer shot in the dark, to somehow get the source code for resample from Lockheed developers and rewrite it so that is multithreaded?

    Or even fuuurther... start from scratch, figure out how resample does what it does and create an entire new and improved multithreaded resample 2.0?

    I'm currently sitting at my PC watching 3 instances of resample run on an 8 core processor and only 39% of the total avaialble processing power is being used. One of them sits at 5% after 2 and a half hours and has a further 46 hours to go... **pulls hair out**
    I'm running 3 of them just so that I can experiment with different terrain nightmaps, and, in the event that the one which finishes first produces unsatisfactory results, I can cancel the other 2 and start over. There has GOT to be a way to speed things up for large scale outputs. (Yes I know I can work on a smaller area first and then go full scale later, but i'd still have to wait for 2 straight days eventually!)
     
  2. GaryGB

    GaryGB

    Joined:
    23 Dec 2005
    Messages:
    3,026
    Country:
    us-illinois
    Hi MOUSY:

    I agree that it would be helpful to have a faster-working version of FS / P3D SDK Resample.


    Until we have one, it merits repeating a caveat regarding avoidance of LZW compression in Resample TIF source files:

    http://www.fsdeveloper.com/forum/threads/blend-water-masks.435509/#post-726137


    PS: I edited today another of my posts regarding Resample (one from 2011) that did not yet mention this caveat. :pushpin:


    Hope this info helps anyone still using LZW Compression in aerial imagery and/or elevation data Resample TIF source files. :)

    GaryGB
     
    MOUSY likes this.
  3. MOUSY

    MOUSY Resource contributor

    Joined:
    14 Aug 2008
    Messages:
    292
    Country:
    dominica
    Wow. Gary I was TOTALLY unaware that LZW compression TRIPLES (TRIPLES!!!) compile time! Thanks for that huge tip!
    (I must also remember to work off my SSD, but its already so cramped!)

    Now back to the original topic. Does anyone have any ideas on how we can go forward with this. I would put all my programming know how into this if I learnt there was a way.
     
  4. Clutch Cargo

    Clutch Cargo

    Joined:
    7 Mar 2006
    Messages:
    827
    Country:
    unitedstates
    I don't have an answer for you but what could you possibly be compiling that would take 46 hours!? An entire country into one bgl :p. I usually perform a batch run of 4-5 resamples running at once on my I-7 CPU at 90%. I can create 25 bgls in around 20 minutes covering around 50 square miles (about 130 square kilometers).
     
  5. MOUSY

    MOUSY Resource contributor

    Joined:
    14 Aug 2008
    Messages:
    292
    Country:
    dominica
    An entire country into 1 BGL is correct.
    4 TIFS for 3 different seasons and a night map, each covering about 1200 square kilometers at 0.5m/px, each LZW compressed to 3.4GB (would otherwise be 11GB BigTiffs and thus unusable by resample).
    The resulting BGL is 400MBs at 85% compression.

    I'm considering using multiple BGLs instead of a single large one, however I have not yet fully investigated the effects on VAS usage in SIM.
     
  6. Golf-HotelDelta

    Golf-HotelDelta

    Joined:
    20 Dec 2004
    Messages:
    11,149
    Country:
    england
    I use LZW compression for all water and blend masks and have no problems.
     
  7. GaryGB

    GaryGB

    Joined:
    23 Dec 2005
    Messages:
    3,026
    Country:
    us-illinois
    Thanks for sharing that practical insight, George; apparently 1-Bit B+W and/or 8-Bit Gray Scale TIF Terrain Mask files using LZW compression do not take as much time to process as 16-Bit, 24-Bit, or 32-Bit "color" source files. :)


    Indeed, I have edited my suggestions on "preferred" settings to use with TIF source files for aerial imagery and terrain mesh on the assumption, IMHO, that most would-be users may want to forego LZW compression in favor of saving processing time.

    However, those with special requirements involving very large data sets wishing to conserve USERVA in a FS / P3D SDK Resample command mode Windows task session, and/or those having a goal of minimizing size or number of output BGLs, certainly might want to choose the (non-lossy) LZW compression option to preserve data fidelity, rather than utilizing the (always lossy) compression options offered by Resample itself ...with the understanding that Resample processing of all source files may take upwards of 3 times longer. :idea:


    One might also wonder whether another option might be faster processing by Resample of source files located within folder chains compressed by Windows NTFS folder compression (...which at one time was promoted by MS due to reportedly minimal impact on system throughput speed due to its background use of 'unused' CPU clock cycles on 1 GHz and faster CPUs) ? :scratchch


    Regardless of the processing scenario, one may also speed things up by setting the FS / P3D SDK Resample command mode CPU task priority in Windows Task Manager to "Realtime". ;)


    GaryGB
     
    Last edited: 25 Mar 2017 at 11:20
  8. arno

    arno Administrator Staff Member FSDevConf team Resource contributor

    Joined:
    28 May 2004
    Messages:
    24,076
    Country:
    netherlands
    Hi,

    I doubt we can make a new resample. The file format is not fully known.

    What I do is run multiple resample instances in parallel. And I'm planning to stretch that to control multiple instances on multiple machines controlled from one place.

    I'm actually running a job now, I'll post the performance here when it's done.
     
  9. arno

    arno Administrator Staff Member FSDevConf team Resource contributor

    Joined:
    28 May 2004
    Messages:
    24,076
    Country:
    netherlands
    Hi,

    Just for your information, here are the stats of my last run. It's an area of 2500 kmsq covered by LOD17 photo scenery (25 cm input imagery). It took 12 hours and 30 minutes to generate the whole area. The BGL files are divided by LOD11 (typically around 300 MB in size).

    I was running 4 resamples in parallel for this, on my build machine with 8 GB of RAM and an i5 processor. Most of the time is running resample, but generating the watermasks from vector data is part of the process as well.
     
  10. MOUSY

    MOUSY Resource contributor

    Joined:
    14 Aug 2008
    Messages:
    292
    Country:
    dominica
    Good info Arno.
    Any particular reason for splitting the BGLs into LOD11 areas?
    And were your TIFFs uncompressed?
     
  11. arno

    arno Administrator Staff Member FSDevConf team Resource contributor

    Joined:
    28 May 2004
    Messages:
    24,076
    Country:
    netherlands
    Yes, the input files are uncompressed GeoTIFF files.

    I guess I might get away with LOD10 for the BGL size as well, but else I go over the 2 GB limit. That's something I would still have to experiment with a bit.
     
  12. MOUSY

    MOUSY Resource contributor

    Joined:
    14 Aug 2008
    Messages:
    292
    Country:
    dominica
    Ok. I thought that splitting the BGLs into LOD 11 or 10 coverage areas was to improve in sim perforamce, example, decreased VAS usage.
     
  13. arno

    arno Administrator Staff Member FSDevConf team Resource contributor

    Joined:
    28 May 2004
    Messages:
    24,076
    Country:
    netherlands
    I'm sure it helps performance to organize the files by LOD, but I only do it when the area doesn't fit in one file due to the 2 GB limit. This 2500 sqkm area results in 40 GB of BGL files right now.
     

Share This Page