That said, my personal contributions to the latest release are mainly related to the basecurves… The basecurves allow us to have decent default output of camera RAW files. With proper basecurves Darktable’s output is roughly similar to the camera JPEG output. We now have basecurves for Canon, Nikon, Sony, Pentax, Olympus, Panasonic and Leica. We have a basecurve for Kodak as well, though that didn’t make it into the 0.6 release.

With the new 0.6 release I also took the oppertunity to cleanup my PPAs. I now have two seperate PPAs exclusively for Darktable. For “stable” releases add this PPA to your sources:

https://launchpad.net/~pmjdebruijn/+archive/darktable-release

Or if you’re feeling adventurous, and don’t mind the occasional problem, you can use our regular development snapshots:

https://launchpad.net/~pmjdebruijn/+archive/darktable-unstable

We’re now working toward a 0.6.1 release with amongst other things some user interface improvements.

Screencasting has always been a great medium for users to learn about applications, it’s so much more efficient than documentation. Especially if we take a look at the huge success of Meet The GIMP, which was well earned.

I took a look at doing screencasts a while back, but back then (Karmic) recordmydesktop was semi-broken. On Lucid it seems to work pretty much perfectly, so I did a first attempt at doing a screencast for Darktable. At first this was intended as a testing-only video for me, but in the end turned out well enough to publish it. It’s still far from perfect, I recorded the audio with the amplification set slightly too high on an otherwise not so great microphone, so I had to do some postprocessing to clean it up (oggSplit, Audacity (Noise Removal & Compression), oggJoin).

The end result was a video that lasts for almost 50 minutes and covers pretty much all important aspects of Darktable, so it’s a great way to learn about Darktable if you can bear to listen to me for the duration of the video. For the short run this will most likely stay a one-off project. In the long run I’ll consider doing more shorter video’s detailing certain aspects of Darktable, taking a more structured approach to the content.

The video below is licensed under the terms of the Creative Commons Attribution ShareAlike 3.0 license.

The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or you can download the screencast here.

Why?

The pretty images our camera’s output aren’t literally what the camera sensors see, there is a lot of proprietary postprocessing involved. Now when working in the RAW format we ditch all that postprocessing in favor of RAW sensor data. Which is good if we want maximum control and flexibility. The problem is that camera vendors don’t publish their proprietary postprocessing methods, which leaves us with a problem, how do we postprocess then?

In the past some good work on this topic has been done by Adobe, who published their DNG specification including color matrices. Color matrices are specifications on how camera native color is transformed into something that an end user might like, and ideally will be correct when viewed on a calibrated display.

Historically there have been some problems with the Adobe color matrices, the biggest of these is the rendition of red colors. To this day I still have no clue why this really is a problem. The only explanation I can think of is that this has been a deliberate compromise on Adobe’s part to safely render skin tones, at the cost of red rendition elsewhere. This is speculation at best…

So ideally we want an alternate enhanced color matrix that will render red as it’s supposed to, possibly at the cost of some skin tones rendition (rendition of skin tones is often a matter of taste, so we can fix that using the color zones “natural skin tones” preset).

Types of profiles

Without going into all the nitty gritty details, there are basically two kinds of profiles, the first are XYZ matrices (these are often called color matrices) and are typically combined with a gamma curve, the second are LUT profiles. What’s a LUT, it’s a lookup table… The big difference is that with a XYZ matrix all color transformations are calculated on the fly, while LUTs are precalculated, so transforming color via a LUT is simply looking up an input color and it’s matching output color. The nice thing about LUTs is that they can deal with slight (nonlinear) deviations and can even be tweaked for creative purposes. So when generating a LUT profile, the profile is likely to pick up some of the peculiarities of your particular camera. XYZ matrices don’t have that problem since they are defined by only 3×3 coordinates in XYZ colorspace, and thus are quite generic by their very nature. This is also the reason why we are sticking to color matrices instead of supplying more detailed LUT profiles (besides diskspace usage).

What digital camera’s can be color profiled

All camera’s that can output a digital RAW format supported by LibRaw can be properly profiled. This effectively means that most compact camera’s can not be profiled (yes, some could using the CHDK firmware hack). Probably half of all digital bridge camera’s can output RAW and thus can be profiled as well. Pretty much all digital SLR camera’s are covered as well…

What color target do I need?

There are currently lots of color targets available on the market produced using different techniques and sold at very different prices. Here are some of the things you need to keep in mind…

First we can classify all targets into two groups, matte and semigloss… In theory semigloss targets can cover a larger gamut (range of colors) than matte targets, but most semigloss targets tend to glare, which is a big inconvenience when shooting the target. Most IT8 targets are semigloss, CMP’s Digital Target for example is matte… I personally prefer matte targets for their convenience since they are easily shot using a decent strobe, which is often hard to do with semigloss target. Semigloss targets are often shot best outdoors on a sunny non-cloudy day at an angle to prevent glare.

Second there’s the patch count, Gretag’s Classic ColorChecker only has 24 patches, while most IT8 targets have up to 288 patches, and CMP’s Digital Target even has 570 patches. When one only want to generate a color matrix this is not of particularly great relevance, the ColorChecker’s 24 patches are enough. But if one wants to do LUT profiles more patches tends to be better.

Third there’s the production technique, some charts like CMP’s Digital Target are produced on highend inkjets, which implies the target is very vulnerable to moisture making it more fragile than other targets. Most IT8 targets seems to be produced using traditional minilabs, and last but not least some (typically Gretag’s) targets are painted. In theory painted target should be superior in the sense that the manufacturer has greatest spectral control over the patches, which means you’re less vulnerable to metamerism when shooting the target under different lighting conditions.

Fourth there’s reference measurements, some targets like Gretag’s ColorChecker are very accurately produced to a single reference specification.  Most IT8 targets are produced in batches, and will include per batch average reference data. The last and most accurate approach are individually measured charts, which means each chart has a reference which is specific to that chart alone, CMP does this for all it’s Digital Target’s and some IT8 vendors will also offer such services on request.

And last and least, don’t try to print your own target, even if you have a color calibrated printer, it will not be accurate enough!

Buying Advice

If you want a target that you can easily use with commercial software as well you should probably buy a (relatively expensive) ColorChecker Passport.

If you want a convenient and highly accurate target you should probably buy an (expensive) CMP Digital Target. This is my current personal preference.

If you want an affordable high quality target, and don’t mind some extra fiddling to get the shot right, you could probably buy an IT8 target from Wolf Faust.

Small note, Wolf Faust really kicks ass, since he relicensed his reference files under the terms of the Creative Commons Attribution-Share Alike license.

Shooting the color target

If you already have a target, or just purchased or borrowed one, we can try to shoot the target. First check if the target is clean and has no damage to the color patches. Then depending on the type of target you have available we can determine the shooting conditions:

Shooting a matte target

If you’re lucky enough to have a matte target available shooting it will be a breeze, since you can shoot the target head-on using a decent strobe. When possible put a dark grey or black piece of card behind the target. When I said decent strobe I meant something like a proper branded strobe like Canon, Nikon, Metz, etc. Please don’t do this with your generic asian no-brand flashes, since we don’t really know the quality of the light it produces, it might be fine for your photos, but I’d rather not use it for profiling purposes…

Shooting a semigloss target

If you have semigloss target the current recommended way to shoot it, is to wait for a sunny day with little or no clouds. Then usually at noon when the sun is highest in the sky shoot the target when there are no clouds obscuring the sun. When possible put a dark grey or black piece of card or cloth behind the target. You probably need to position the target at an angle (with regard to the camera’s position) to prevent any glare…

… Shoot the target using a modern normal lens (no antiques (since lens coating can differ) or fisheye folks) at about 50mm or so, at a distance of about roughly 1 meter or 3 feet. The target should cover about 50-75% of the image surface. The dark piece of board behind the target should be visible on all four sides of the target.

Now comes the hard part, getting the exposure right… Unless your camera has a true RAW histogram, the histogram will be useless to you. The best exposure will likely look slightly overexposed as a preview image. The best way to approach this is to take a boatload of exposures from severely underexposed to severely overexposed and select the best exposure later on using Darktable.

So, if applicable set your strobe to full manual (1/2 strength for GN36, 1/4 strength for GN48, 1/8 strength for higher GNs). Then set your camera to full manual mode as well, set Exposure to 1/200 sec, and set the sensor sensitivity to it’s normal lowest possible setting (without pushing), which is usually ISO 100 or ISO 200. Don’t use any special modes which allow extra low or extra high ISO, since this is often digital trickery… Then close down the aperture all the way (usually f32 or something) and take a shot, open up the aperture 1/3rd stop and take another stop, keep opening up the aperture and taking shots all the way, until the aperture is fully open. Usually this means you have taken about 20-30 shots by now.

Selecting and Processing a target image in Darktable

Now import the images you shot into Darktable, pick a target image which looks slightly overexposed, and open it in darkroom (develop) mode. Then disable all plugins except for these:

• raw import
• exposure
• white balance
• highlight reconstruction
• input color profile (set to linear rgb/absolute colorimetric)
• output color profile (set both to linear rgb/absolute colorimetric)

Make really sure you have the basecurve and sharpening disabled (which are enabled by default). You’ll notice the image getting darker… Then do spot white balance on the target’s grey patches. Then use the color picker (bottom panel) to check the Lightness value (the L from LAB), of the brighest white patch. You need to check you’re charts reference file for the exact Lightness you should match. It’s usually L 96 for ColorCheckers, L 97 for most CMP Digital Targets and L 92 for most IT8 targets. If the Lightness of the brightest white patch is L 99 or L 100 straight away, the image is probably overexposed (and useless), move to an image shot a 1/3rd stop darker. Now open up the exposure plugin, and start increasing the exposure until the Lightness value of the white patch in the image matches the Lightness specified in your charts reference file. If you need to apply more than +0.3 or +0.4 of digital exposure, you probably need to check an image that was shot a 1/3rd stop brighter.

Once you find the proper image, and equalize the Lightness of the brightest white patch to the chart’s reference value, you need to inspect the image for glare, if there is any glare whatsoever you need to redo everything again. This is once again why I prefer matte targets, they’ll save you some aggravation…

If everything is dandy exit darkroom mode, and export a 16bit TIFF image.

Generating the color matrix using Argyll

To do the actual calculations we need Graeme Gill’s excellent ArgyllCMS, particularly version 1.1.1 or higher. First we need to read the TIFF file for it’s color patch values, we do this using Argyll’s command line utilities:

# scanin -v -p -a -dipn IMG_1234.tiff /usr/share/color/argyll/ref/CMP_Digital_Target-3.cht CMP_DT3.txt

This command will output a diag.tif which you can use to see if the chart was properly recognized. Using the previously read values we can calculate the actual matrix:

# colprof -v -A "Canon" -M "Canon EOS 400D DIGITAL" -D "canon eos 400d" -C "Copyright (c) 2010 Pascal de Bruijn. Some rights reserved." -q l -a m IMG_1234

This command will output a IMG_1234.icc file which is a standard ICC color profile. To directly use it with Darktable you can copy it for Darktable to find:

# sudo cp IMG_1234.icc /usr/share/darktable/color/in/canon_eos400d.icc

There are other options to generate profiles for personal use, for this please read up on color management in general and the ArgyllCMS documentation.

Submitting results

Since there is no direct way to check if a matrix’ values are correct, I’m hesitant to directly accept profiles/matrix values from third parties. At this point I highly prefer for users to make properly exposed RAW chart images available to us, so I can do the calculations. Proper credit will be given to whoever supplies the chart.

Screencast

The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.

There are a couple of things that interest me in Darktable, the first is that it is a combo of both photo management (lighttable) and raw development (darkroom), it’s really integrated, not two programs (F-Spot + UFRaw) stuck together with some ducktape.

The second would be the fact that Darktable seems extremely well designed, it’s entirely plugin based, even the raw import itself is a plugin. This means the code is well separated into these plugins, this inherently means functionality can be disabled by just disabling a plugin. No need to comment out some code before compiling.

Another strong point of Darktable is it’s imaging pipeline, except for the first and last few steps, the pipeline is entirely in 32bit floating point LAB colorspace, properly integrating color management. Which means it’s calculations can be extremely accurate. A nice extra benefit is the fact that some image operations just work better in LAB than other colorspaces, the Unsharp Mask being the principal example of this.

Something that’s also very important to me is the user interface, I really care about this. While Darktable does not always conform to common practice in user interfaces and often prevents novel idea’s for it’s user interface, it’s easy to work with. It works particularly well fullscreen (F11 just like Firefox), it’s dark theme makes sure you’re not distracted from your photograph. And the edge/buttons for expanding or collapsing the sidebars are at the screen’s edges, which means you can just quickly drag your mouse to the edge (without any particularly aim), and click to expand or collapse a sidebar, this works really fast!

Darktable is still a very young project, but it’s well worth taking a look at. Rolf Steinort from Meet The GIMP recently did a show on Darktable.

If you want to try Darktable yourself can get nicely packaged git checkouts (development versions) from my PPA for Ubuntu.

The above picture is a first version of my homebrew ColorChecker. I made it by first looking for a good neutral matte white paper, which became Tetenal Photo Archival Matte, for which I then made a printer profile using ArgyllCMS. Generating a printer profile (actually it’s profile of the paper+ink+printer+driver configuration combo) means I can print images with a reasonable level of confidence the print will be color accurate. Then I made a custom sRGB version of the ColorChecker target (look for inspiration here). Then I printed the custom sRGB image of the ColorChecker with the printer profile applied. Then I measured the target using my spectrophotometer, so I know what colors the patches actually are. I scaled the print so that the target could fit inside the manual area of a traditional 14mm DVD case. The DVD case has a dual purpose here, first it keeps the target straight/upright and when closed it protects the target from dust/moist. To stick the target to the inside of the case I used 3M Photo Mount, which is Ph-neutral to prevent the glue from deteriorating the colors.

Initial tests are very encouraging, camera profiles generated from it work quite well, especially when shot using a decent hot-shoe flash.

Open up your noisy phone camera image in GIMP, then go to Filters, Enhance, Despeckle… And obviously the Despeckle dialog turns up… Enable the adaptive median (but leave recursion off). Set the radius to 1, and leave the black and white levels at their defaults (respectively 7 & 248), and behold the results:

Having said that, I am still slightly sad about this, GIMP is one of those iconic open source applications that have been around for more than a decade. Luckily we’re not saying farewell, just until we visit next time.

Another “problem” of using the commandline utilities of ArgyllCMS was, that display profiles consist of two parts. A VideoLUT which has to be applied by X11 or loaded into the video card, and a color matrix with gamma shaper which has to be applied by the color management aware applications like GIMP and UFRaw.

In the past, I’ve always built simpeler (and less accurate) profiles, to prevent me from having to load the VideoLUT every time I logged in. GNOME Color Manager solves this by introducing a service which does this fully automatically. Making it easy to use more accurate profiles.

Last but not least, GNOME Color manager also adheres to the XICC specification, and makes sure color management aware applications like GIMP and UFRaw can automatically see for which profile the VideoLUT had been applied, and thus dictating which profile to load in the application.

As could be expected, I’ve been building Ubuntu (Karmic) packages from regular git checkouts, which are available at my PPA.

You can download the profiles here, or get them nicely packaged for Ubuntu Karmic from my PPA.