The NullValue only affects the Source1 image... it has nothing to do with the Channel_BlendMask ( Source2 ).
There are 2 separate ways to mask off an area, using either the NullValue or a Channel_BlendMask/MultiSource. You can use both in a photoreal.
The use of a Channel_BlendMask only is of real use when you wish to blend the edges of the photoreal into the underlying landclass texture. That's why it uses 256 greyscale values... to achieve the increasingly transparent blend.
It's much easier to NOT use a Channel_BlendMask for water masking...just use the NullValue for water, making it whatever RGB color you would like. Why have 2 images when 1 will do? 255,0,0 is red. 0,0,255 is blue. Just paint the color where you want the mask to go. Then adjust the INF if needed.
In the INF you showed, you used a Channel_BlendMask, so the second image will be used as a mask, as well as the NullValue. The convention for that is white is no-mask, and black is full-mask, greys are transparent in-between.
The NullValue line can be deleted from the INF if you don't need to use it.
SBuilderX just includes it because the SDK used it in their example... but it works fine as it eliminates anything pure white.
If the SDK had used 0,0,0, then we'd all be using that, and pure black would be voided ( actually a better idea, as this is similar to the old FS9 methods of using black in an alpha-channel; and black is used in the FSX Channel_BlendMask as a void ).
Because SBuilderX, and I think all other INF-building programs, use 255,255,255... then just paint masked areas white, and skip the use of a Channel_BlendMask, unless you want a blended ( degrees of transparency ) edge to your photoreal.
If you look at the millenniumimage3.tif in the SDK, you'll see there are white areas at the edges, as if the image were reprojected to WGS84 lat-long projection from UTM projection, and the image doesn't line up quite right at the edges. The NullValue=255,255,255 voids those areas from the photoreal. That's why they used that value, for that example.
Dick