This site is supported by the advertisements on it, please disable your AdBlocker so we can continue to provide you with the quality content you expect.

Dynamic Range (DR) setting effect on RAW data with X-Trans III sensors (X-T2, X-P2, etc)

Discussion in 'General X Camera Forum' started by Zurubi, Jan 7, 2018.

  1. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    Are you looking at the JPEG generated by the camera? Look at the cypress trees example I just posted above and compare the sky in the camera JPEG with the sky in my processed photo -- both from the same sensor exposure.
     
  2. Dr Settings

    Dr Settings Member

    Joined:
    Jan 6, 2018
    Messages:
    48
    Likes Received:
    23

    -Return to Top-

    Yes, I realise it was a bit of a loaded question! Thanks for the answer, very well expressed. I generally take default setting JPGs and, as you say, lightly PP. A touch of levels, white point, maybe some contrast. I'm usually happy enough with that.
    I think I'll give what you suggest a try, see how I get on with it. Thanks.

    Not read the link yet, but will do.
     
  3. FujiMongol

    FujiMongol Premium Member

    Joined:
    Oct 13, 2017
    Messages:
    280
    Likes Received:
    135

    -Return to Top-

    Both actually, jpg or raw. But I never work with jpg's (I use then to quick select only), so in general I speak in PP mode of RAW.
    I sense the loss of deep blue in the sky the more I do ETTR, even without clipping I've noticed the colors gets affected (towards pale) before any clipping starts.
    For having better dymanic range in the sky I've experimented with the opposite namely ETTL. Then Lightroom and the "ISO-independent" character of X-T2 could effortlessly bump up the dark parts unpunished. It led me to a habit of underexposing dark parts -1 and have all my light parts with max dynamic range. Then get my dark parts +1 in LR, and I sensed LR to be slightly better than C1 on that part. I end with a pic that looks like a HDR.

    For example this picture, the lake, it's shot heavily ETTL instead of ETTR, the bushes were very very dark. Then LR pumped up the shadows. But I like to keep that limited to max +2 in LR shadows. If I shot the pic ETTR then my details in the sky would have been lost.
    2nd Advantage is that due ETTL I have one stop more at disposal, I could half shutter time for example.
    I'm still learning but checking out things. I'm not suggesting the best here, just exercises and their results.
    ettl.01.jpg

    In the second pic, the stroll path, was also shot ETTL-ish because otherwise the sky would have blown away finer details where the finer twigs meet sky.
    Path.jpg
    (PS: pics are sharper in full format, these uploads are strongly reduced)

    I do agree to aim for max signal!! But I do that (for now) with respect of dynamic range in the light parts.
    Maybe I've been doing things wrong in PP but for now ...

    I understand the reasons for ETTR and max signal, but I've found (so far) it does have ramifications on lighter parts despite no-clipping, but probably it depends on the kind of shot, like in that example picture of the candy pots there's little fear to loose details when doing ETTR, there's no sun blazing trough twigs.

    Eventually what I do is exactly the DR200 and DR400 trick (which is meant for the SOOC jpg's) but on my RAW's + PP.
    Fuji must also have seen loss in lighter parts when the contrast bumps up, and they found solution for their jpg's to move away from ETTR and more to the left by doing DR 200 or 400.
    But for RAW there was no solution and therefore I moved to the LEFT myself, I did not wanted to mimic Fuji DR jpg solution, it just seemed to co-ax in logic.

    Maybe there are ways to preserve all DR and detail and color in lighter ETTR (not clipping) parts of the RAW in PP, but I haven't found them yet.
     
    Last edited: Jan 10, 2018
  4. Dr Settings

    Dr Settings Member

    Joined:
    Jan 6, 2018
    Messages:
    48
    Likes Received:
    23

    -Return to Top-

    ETTR is to maximise exposure but not to the detriment of blowing out wanted highlights, such as in the sky.

    So perversely your ETTL shots are actually ETTR, because you've done just that - exposed to capture the sky without blow out.
     
    lawsofphysics likes this.
  5. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    A digital sensor has a maximum data recording capacity. Maximum dynamic range (tones from dark to light) is achieved when the full recording capacity of the sensor is utilized -- diffuse highlights placed at the sensor clipping threshold. Any reduction in exposure such that the diffuse highlights of a scene are recorded below the sensor clipping threshold reduces DR including DR in the highlights as the sensor records data linearly.

    You're suggesting that an exposure reduction is producing better highlight dynamic range. Unless you're pulling back from already clipped diffuse highlights that's not how a digital sensor works. It may be something LR is doing but it's not something your sensor is doing.

    ISO invariance in no way ameliorates the loss due to reduced exposure. Any unnecessary exposure reduction is always punished.

    Dynamic range in the highlights increases with increased exposure up to the point of diffuse highlight clipping. Any reduction of exposure that places the diffuse highlights below the sensor clipping threshold reduces dynamic range in the highlights along with the rest of the photo.

    As long as the sensor is not clipped the data in the highlights improves in all ways with increased exposure. Unless there's sensor clipping there is no down side.

    A full sensor exposure that places the diffuse highlights at the sensor threshold (not clipping) will have max DR in the highlights and record best possible color and detail. If you're having difficulty it's not your hardware -- I'd suspect software.

    You note multiple times you've tried for increased signal (exposure) but qualified that saying no clipping. How do you determine the state of your raw file? How do you know there is no sensor clipping?
     
    lawsofphysics likes this.
  6. johant

    johant Premium Member

    Joined:
    Apr 25, 2015
    Messages:
    2,082
    Likes Received:
    1,048
    Location:
    Netherlands

    -Return to Top-

    Very interesting posts, thanks for taking the time!

    When I expose to the right (and deliberately ignore the JPEG output the camera is showing me), my issue often is adjusting the tone curve in RawTherapee or Silkypix (I don't have LR) in such a way that the output looks balanced. I realise that is my own limitation and not something wrong with the RAW converters. But I definitely need more practice in this area (and PP in general).

    (I use a X-T1, but that doesn't change the story really of course)
     
  7. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    How do you evaluate your RAF files for exposure -- highlight clipping or not?

    RawTherapee and SilkyPix have limited to no capacity to apply local adjustments. I assume when you say SilkyPix you're referring to the version supplied with the camera. That complicates the processing and you'll often have to rely on a second app like Photoshop to provide those local adjustments and finish up.

    RawTherapee does provide a gradient filter that can take you a long way with some landscapes.

    Might want to look at Affinity Photo which is inexpensive and would complement both RawTherapee & SilkyPix. Another option would be to consider a different raw converter. DarkTable like RawTherapee is open source and does have the local adjustment capacity.
     
  8. lawsofphysics

    lawsofphysics Premium Member

    Joined:
    Apr 17, 2013
    Messages:
    2,601
    Likes Received:
    1,010
    Location:
    usa

    -Return to Top-

    A +2 shadow push in LR is conservative. I like to keep to that too. However, consider that ISO 1600 requires a 3 stop global push of some sort. It could be in-camera DC voltage amplification (actually setting ISO to 1600) or increasing global brightness during raw rendering using digital multiplication (+3 LR Exposure slider).

    So, when you underexpose using ISO 200 with raw, a 3 stop selective push for shadow regions is not excessive.
     
  9. FujiMongol

    FujiMongol Premium Member

    Joined:
    Oct 13, 2017
    Messages:
    280
    Likes Received:
    135

    -Return to Top-

    Thank you ysarex for your time and effort.
    The technicality about max exposure for best S/N ratio is well known to me, and your question how I decide I am not clipping (when trying ETTR) was based on the little histogram on the X-T2, which might be not the best of guidelines, but now since firmware 3.0 we have blinkies finally.
    I think to be a bit in johant's boat, that I have more of a problem getting my image to liking in PP when it was shot very bright (not clipping). This might boil down to experience in PP. An image might have the best S/N ratio but when it cost me double the time to get an image right I'd rather stop down a notch and have it nearly on a plate as long as pixel peeping makes me happy.
    From here I think the only way to continue my topic is to get examples in here with their histograms.
     
  10. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    They're not the best of guidelines and the blinkies are no improvement except that they're maybe easier to see. The histogram/blinkies feedback from the camera is not accurate to the sensor exposure -- raw file. The camera histogram/blinkies reflects the video feed to the camera display which is designed to reflect the camera JPEG. Typically they will indicate clipping before it actually occurs for the sensor. You can improve their accuracy relative to the raw file by setting the camera white balance to unity, but that's a pretty extreme choice to make. The example file below has the WB set to unity.

    To accurately assess a raw file exposure you have to be able to examine the raw file directly. Few processing tools permit this. When we load a raw file into software like LR or C1 the histograms we see are derived from the processed RGB images those program immediately display. RawDigger and it's sister tool FastRawViewer allow us to examine the actual raw file and determine how it was exposed.

    That makes complete sense. Our efforts are constantly being moderated by software and all of that software is engineered with assumptions in place. To a greater or lesser degree we often have limited access to disengage those up front assumptions that are applied to our images. Typically their existence isn't even acknowledged. You noted that you're using LR. It's great software (I make my living teaching it) but in that class of software (raw converter) LR is less likely to allow you the option to disengage all the helpful things it does for you up front. In this case specifically I mean the tone curve applied to your file when it's first presented to you. Sensor data is strictly linear and it looks pretty awful in that linear state. So every raw converter applies a default tone curve to your image before you attempt any further adjustments. Some raw converters for example LR, SilkyPix, RawTherapee don't contain an option to turn that default tone curve off. Other raw converters do allow you to shut it off. LR, SP, and RT make that assumption up front that their tone curve is appropriate for your image and further assume you have an exposure that's "normal" for your camera.

    So faced with PP an image let's consider three options:

    A) Take the photo exposing to produce a basically good camera JPEG -- the camera engineer's expectation. Raw file opened in LR with the default tone curve applied and the photo already looks pretty good! A few more tweaks and it looks great!

    B) Take the photo exposing for a full sensor exposure -- camera JPEG looks blown out. Raw file opened in LR with the default tone curve applied and the photo looks awful. It's overexposed with washed out colors and it's going to take more work to process.

    C) Take the photo exposing for a full sensor exposure -- camera JPEG looks blown out. Raw file opened in (C1, DarkTable, PN) with the base tone curve turned off and the photo looks awful and it's going to take more work to process.

    Options B & C are understandably less attractive.

    So here's an Please login or register to view links of a full sensor exposure. The green channel has reached the sensor clipping threshold -- slight amount of green channel clipping. Any decent raw converter will be able to reconstruct those minor clips from the unclipped red and blue channels. From the standpoint of S/N and data collection this is a textbook example of "nailed exposure." Open that in LR and it will by default look absolutely horrible. You would have to assume it couldn't be fixed. Part of that problem will be WB. Set the WB temp to 5600 and the tint to 16 and it'll look better but still look badly overexposed -- you're seeing LR's default tone curve applied. The engineers assumed I wouldn't take that photo with a +1.67 EC, and there's no switch in LR to turn off that tone curve.

    Here's the histogram for that raw file.

    [​IMG]

    And here's that photo as I saw it and took it.

    [​IMG]

    (Yes I flipped it). The data I need is in the raw file and it's the best data possible given the camera I used. But yes I absolutely had more work to do post processing the data. Given the sensor exposure I chose I forced myself into either option B or C above when I opened the RAF file in a raw converter.

    I want to post another visual for this but thinking I'll do that with a second post -- little later.
     
  11. Zurubi

    Zurubi Premium Member

    Joined:
    Apr 19, 2014
    Messages:
    304
    Likes Received:
    197
    Location:
    Usa

    -Return to Top-

    i was actually going to ask you about this, but you gave the answer, that you can't trust the new histograms in FW 3.0 and the blinkies. Do you have a rough idea of how much more room you have once the blinkies start? Or will that be totally thrown off by the jpg engine?

    I am starting to feel like @FujiMongol, if this will create a huge new burden in PP, then it won't be fun anymore!

    thanks
     
  12. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    Seriously, I'm a nut-case. I'm retired (I teach some college classes part-time), I no longer do any work for hire, I just shoot for myself. I only have myself to please. If I took a job to take photos for $$$ I'd immediately adjust my practice to conform with mainstream expectations and shoot appropriate exposures that look good default opened in LR. I teach my students to work the same way -- do as I say not as I do! What if you have a client standing over your shoulder? Are you going to try and explain why there's a difference between a sensor exposure versus a JPEG exposure? Modern sensors are so good that in general the difference just isn't worth the effort and risk.

    Except.................... in those few cases where you have extreme lighting contrast and want all the tonal data you can get. I enjoy taking photos like that. So I like to be in practice and know I have everything tested to be able to do it. Best way to stay in practice is to practice so I just eventually drifted into shooting for a full sensor exposure. But yes, it's risky; it really hurts when you screw-up and go too far and it makes the PP job certainly more difficult for someone who isn't very capable with the PP software.

    You're blinkies/histogram question: First it's important to be sure you're discriminating between specular and diffuse highlights. The blinkies will come on way early if the scene contains specular highlights (reflections from shiny objects). E.g. if you photograph a motorcycle in the sun the chrome will contain specular highlights; you want those to clip.

    You can make the blinkies/histograms more faithful to your sensor exposure by setting the camera WB to unity. You're JPEGs will all come out green (see landscape RAF in my last post). You can use RawDigger to verify how close you are between the sensor exposure and the viewfinder feedback. You can also switch back and forth between unity WB and auto and get an idea of that variance.

    The trick is setting your X-T2 to unity WB. Here's a link: Please login or register to view links but that's a lot of work. So I've attached a file to this post. It should get you pretty darn close. Just load it up in your image viewer, fill the screen and set a custom WB on your camera.

    XT2_uniwb_reference.jpg
     
  13. AnthonyM

    AnthonyM Premium Member

    Joined:
    Mar 26, 2014
    Messages:
    1,521
    Likes Received:
    746
    Location:
    London, UK

    -Return to Top-

    Thanks for this.
     
  14. FujiMongol

    FujiMongol Premium Member

    Joined:
    Oct 13, 2017
    Messages:
    280
    Likes Received:
    135

    -Return to Top-

    Thank you ysarex for the explanation again, and the study material RAF.

    The main question left is ... why?
    Why is Fuji forcing us to not use the complete DR at hand? They do this by feeding us a manipulated histogram/jpg/display letting us think too soon ETTR.

    I am amazed how (alleged) overexposed this ysarex-RAF was, yet all inside DR. Also amazed how much can be dragged out of the white, while at first sight considered lost.
    The tonality in the clouds is there, probably thanks to the used DR.
    When testing out the iso-invariability before, I was amazed how much the dark could be bumped up, and now amazed again how much detail can be in the light parts too.

    This JPG is what I made of it using Capture One 11, I was carefully with the blue, I've no idea how it looked so I was guessing. Tried my best as an amateur.

    field.5.jpg
     
  15. FujiMongol

    FujiMongol Premium Member

    Joined:
    Oct 13, 2017
    Messages:
    280
    Likes Received:
    135

    -Return to Top-

    And this one came out of Lightroom, little bit more blue allowed.

    field.6.lr.jpg

    I was also careful with the contrast, I realize many people like more punch o_O
     
    Last edited: Jan 13, 2018
  16. FujiMongol

    FujiMongol Premium Member

    Joined:
    Oct 13, 2017
    Messages:
    280
    Likes Received:
    135

    -Return to Top-

    This is my wild guess about the "why" they force us to shoot less than max DR/SNratio:

    I've the impression that true ETTR's leave us with not a deep color set, and that shooting at a lesser than ideal DR/SNratio leaves us a more pronounced color set so it's easier with those bigger color handles to PP. For example: at lesser than ETTR then I have also lesser guesswork what kind of blue that sky actually had.
    Maybe it's simply me needing more experience how to extract the original blue in PP.
    But for now I've the impression that also Fuji wanted to feed us a more powerful colorset at the expense of some DR/SNratio, like a trade off?
    I'm not sure, these are just thoughts.
     
  17. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    Great question. To begin with they all do it to a greater-lesser degree. An advantage of my continuing to teach is that I eventually get my hands on most of the camera brands out there. My students come to class with mostly entry level Canon and Nikon cameras but I usually get an Olympus and Panny etc. now and again. Last semester I had a student walk in with a Nikon D800.

    Forced is a little too strong. I'm not forced. Fuji has provided me with a camera that allows full override exposure control and that saves raw files. I am free to do as I please and very happy with the camera. The "forced" condition only applies to the JPEG side of the camera. And all of the major camera manufacturers are engineering the exposure systems and JPEG processing software in their cameras to hedge back anywhere from a 1/2 to more than a full stop from the sensor clipping threshold. To know for sure we'd have to ask them and I suspect they won't tell but if I were them I'd do exactly the same thing. Reflected light metering is imprecise and is going to stay that way. Take the example photo I supplied -- the side to slightly back light increases contrast and the sky is too light compared with the foreground -- that exposure will always be a judgment call. Take that context and add two more: 1. Reaching sensor saturation is like hitting your head into a concrete wall -- you're going to crack your skull open. 2. Modern sensors are now capable of 11+ stops of recorded tonal data; normally more than needed.

    So add those three contexts together:
    1. Imprecise reflected light metering.
    2. Sensor clipping threshold = head busted open.
    3. Sensors now record more data than normally needed.
    and what are the engineers going to do in response to those three facts? I know what I'd do. I'd adjust the metering system and JPEG software in our cameras to stay at least 1/2 stop clear of that sensor clipping threshold. Remember who are our customers buying these cameras.

    Once the camera manufacturers have finalized the hardware with that built-in hedge the raw file converter software engineers build in appropriate support on their end so that the input profile and tone curve, in for example LR, is adjusted to expect that the camera was used the way the camera engineers adjusted the JPEG system.

    I think that's what we have here industry wide and I think it's the way it should be. I wouldn't change it. For me, I'm extra happy if the processing software will let me shut off the tone curve. I'd rather start from there than wrestle back the inappropriate curve -- but no big deal.

    You have C1 access. Do this: Load that landscape RAF back into C1. Clear all adjustments and start from scratch.
    Set WB temp to 5300 and tint to 4.

    What you then see is approx. good WB but a way overexposed image. Find the C1 tool BASE CHARACTERISTICS and change the Curve dropbox to Linear Response and watch what happens. It's easier to continue from there.

     
  18. ysarex

    ysarex Premium Member

    Joined:
    Mar 21, 2014
    Messages:
    1,013
    Likes Received:
    633
    Location:
    St. Louis

    -Return to Top-

    I think you're being influenced by the experience using post processing software. If the sensor records data (not clipped) we can do anything we want with it. At the top end the sensor data is cleanest, best S/N and no color problems. I think the camera makers are just designing the cameras with a safety hedge built in so the average user experiences less loss due to clipping errors. There's no down side to a full sensor exposure. The data just keeps getting better until you reach sensor saturation. The post processing software you're using has been engineered to assume you're using the camera the way it's JPEG system was designed.
     
  19. Zurubi

    Zurubi Premium Member

    Joined:
    Apr 19, 2014
    Messages:
    304
    Likes Received:
    197
    Location:
    Usa

    -Return to Top-

    I've been thinking about what you wrote for a few days since I once took a Luminosity mask from one of the true pros, and I found my notes, so one issue could be that in LR (and most PP softwares), the curves and all the light adjustments are done without any blending modes. That leads to shifts in colors and contrast (you can check that very easily, by applying a curve in LR and PS with a different blending mode). If you take your image, load it into PS and apply the same curves but through a Luminosity blending mode, then the colors don't shift. That was quite an eye opener for me. Of course, by using LR and the likes, we just need to apply further adjustments and it becomes a trial and error process.
     
  20. FujiMongol

    FujiMongol Premium Member

    Joined:
    Oct 13, 2017
    Messages:
    280
    Likes Received:
    135

    -Return to Top-

    Thank you all,
    now I must get rawdigger :D
     

Share This Page

  1. fujix-forum.com uses cookies to help personalize content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice