Nonsense!
I just suplemented my Darktable Camera Color Profiling article with a screencast:
The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.
After 5 month’s of hard work, we finally got around to releasing Darktable version 0.6. The new version has boatloads of new features. For example we added a lot of new image processing plugins. Other notable features are Picasaweb export and we can now storage password (like your Picasaweb login) in GNOME Keyring or KWallet (as any decent application should). Beside the new bells and whistles a lot of crashes have been fixed in the last month. Hats off to Johannes, Henrik, Tobias and the other contributers.
That said, my personal contributions to the latest release are mainly related to the basecurves… The basecurves allow us to have decent default output of camera RAW files. With proper basecurves Darktable’s output is roughly similar to the camera JPEG output. We now have basecurves for Canon, Nikon, Sony, Pentax, Olympus, Panasonic and Leica. We have a basecurve for Kodak as well, though that didn’t make it into the 0.6 release.
With the new 0.6 release I also took the oppertunity to cleanup my PPAs. I now have two seperate PPAs exclusively for Darktable. For “stable” releases add this PPA to your sources:
https://launchpad.net/~pmjdebruijn/+archive/darktable-release
Or if you’re feeling adventurous, and don’t mind the occasional problem, you can use our regular development snapshots:
https://launchpad.net/~pmjdebruijn/+archive/darktable-unstable
We’re now working toward a 0.6.1 release with amongst other things some user interface improvements.
Something universal about pretty much all software is the fact that it occasionally crashes. The nice thing about open source software is that you can actually do something about it even if you’re not a programmer yourself. You can generate a so-called backtrace, with which you can help developers figure out why their program is crashing.
The following video tutorial demonstrates a crash with Darktable but the tutorial should be applicable to most applications. And don’t worry the crash in the video got fixed minutes after I made the video.
The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.
Since a few days I’m the owner of a Toshiba Camileo BW10 pocket camcorder. It’s a very small FullHD (1080p30) watertight camcorder. The watertightness is one feature that sets the BW10 apart from it’s competition, especially at it’s particular price point (one can be had for ~140EUR).
The BW10 produces 1080p video at a 30fps framerate, encoded as H264 (AVC). Please note the BW10 does real 1920×1080 instead of 1440×1080 (anamorphic) video, so it’s true FullHD. It encodes between 13-15MBit/sec.
Now that all sounds fine and dandy, however, the BW10 seems to not deal very well with full screen motion, the recorded visual quality significantly drops when the camera is heavily shook, or the larger part of the screen is in continual motion. However when recording in 720p, it does deal pretty well with full screen motion. I guess Toshiba limited the maximum bitrate to 16mbit (SDHC class 2), no matter which kind of card you use, so all cards are compatible, but 16mbit probably doesn’t provide enough headroom to do 1080p with full screen motion.
Because the BW10 is very very light, it is hard to keep it steady at times, and although it has a digital stabilizer, which does seem to work, it can’t work miracles. So I’m considering purchasing a monopod to use as a simple stabilizer. You can obviously take this as far as you want.
Last and least, true to it’s class, it performs poorly in low light conditions, just like all other (more expensive but still) affordable camcorders.
The BW10′s photo capabilities are horrible and then some less. Some rocket scientist at Toshiba thought it to be a good idea to oversharpen the hell out of everything to the point it produces artifacts. This is obviously a firmware issue, so this could be fixed with a future firmware update, if Toshiba decides to grace us with one.
If Toshiba graces us with a fix, I reckon it’s photo quality could be similar to the better phone camera’s.
Update: when you set the photo camera feature to only to 2MPixel shots, the resulting images are quite decent. I wouldn’t be surprised if the actual sensor is only 2MPixel (which is enough for 1080p) and upscales to the advertised 5MPixel.
Funnily enough the BW10 can also function as a Webcam (640×480), which is nice, but not really that special. They did implement this well, it’s USB Video Class, so it’ll work with any modern operation system, without any driver installation woes.
At first glance I foolishly mistook the BW10′s Micro USB plug for a proprietary one. After some reading up, it seems phone manufacturers settled on Micro USB for charging via USB, and it’s great Toshiba went with that. Thumbs up for Toshiba.
The BW10′s battery is not a sealed internally as part of the unit, so in theory you could buy extra batteries for extended operation. Toshiba lists 70 minute operation for a fully charged battery, while charging an empty battery can take up to three hours. I’ve been able to get 75 minutes of FullHD continous video when shooting directly after charging. Not a particular real-world use-case. I guess it’ll be realistic to expect an hours worth of FullHD video from a single battery with real-life usage.
Some other things that are nice about the BW10 is the pouch Toshiba includes with the camcorder, it won’t protect the unit from shock damage, but it will protect it from getting scratched by your keys when you have it in your pocket. For serious protection a real pouch like Lowepro’s Apex series will be better.
Toshiba also had the insight to fit the BW10 with a standard screw in mount so the BW10 can be used with a generic tripod or monopod.
All in all the BW10 is far from perfect, not forgetting it’s faults, I’d still say it’s a bargain. Especially considering that most competing products in it’s price class have similar deficits.
I’ve been ranting a lot about Darktable lately, some of you might still have no clue what it’s all about. Also Darktable already has a lot of features, and like any complicated application some of it does need some explaining.
Screencasting has always been a great medium for users to learn about applications, it’s so much more efficient than documentation. Especially if we take a look at the huge success of Meet The GIMP, which was well earned.
I took a look at doing screencasts a while back, but back then (Karmic) recordmydesktop was semi-broken. On Lucid it seems to work pretty much perfectly, so I did a first attempt at doing a screencast for Darktable. At first this was intended as a testing-only video for me, but in the end turned out well enough to publish it. It’s still far from perfect, I recorded the audio with the amplification set slightly too high on an otherwise not so great microphone, so I had to do some postprocessing to clean it up (oggSplit, Audacity (Noise Removal & Compression), oggJoin).
The end result was a video that lasts for almost 50 minutes and covers pretty much all important aspects of Darktable, so it’s a great way to learn about Darktable if you can bear to listen to me for the duration of the video. For the short run this will most likely stay a one-off project. In the long run I’ll consider doing more shorter video’s detailing certain aspects of Darktable, taking a more structured approach to the content.
The video below is licensed under the terms of the Creative Commons Attribution ShareAlike 3.0 license.
The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or you can download the screencast here.
I have proper ioQuake3 (1.36) packages for Ubuntu Lucid available in my PPA now. The package includes an important bugfix to make bots work again on 64bit systems, the new Tango icon is used as well…
These are just the open sources built for Ubuntu Lucid, the packages do not include any game data, you still need the original pak0.pk3 .. pak8.pk3 files from Id.
Are you highly annoyed that generating GPG keys takes ages? Well the problem is reasonably obvious, you don’t have enough ready to go entropy, so you actually have to generate some while GPG generates the key… While GPG is one of the most common and visible use cases, entropy (randomness) is used for lots of things, particularly in cryptography.
So ideally we always want our entropy pool topped up, sourced from a reliable source of true entropy. Some processors and chipsets have a built-in entropy generator, however there aren’t many yet, so chances are your processor or chipset won’t have a built-in entropy generator. So meet the SimTec EntropyKey.
The nice thing about the entropy key is that all of the supporting software is open source, and it does not require weirdo (vendor) ill-maintained kernel modules. And the best part is that it is quite affordable, I think it cost me about 50EUR including shipping.
If you order one of these USB keys (they really look like a clumsy USB memory stick) and want to setup everything, you need to install the ekeyd package from the Ubuntu (Lucid) repositories, no need for third party packages. The ekeyd package has a nice udev rules included to automatically recognize EntropyKeys and give them human readable device nodes (you do need to reboot or reload udev for this to work):
/dev/entropykey/M_3lUoWERuOdXDFR -> ../ttyACM0
You’ll note that the EntropyKey actually uses the generic ACM USB serial driver to talk to it’s userland tools. The default configuration file in the ekeyd package automatically recognizes all EntropyKeys setup by udev, so in theory multiple keys should actually work without changing the configuration file.
Now you can use the following commands to setup and initialize the EntropyKey:
cat /proc/sys/kernel/random/entropy_avail
432
# sudo ekeydctl list
NR,OK,Status,Path,SerialNo
1,NO,Long-Term-Key is bad,/dev/entropykey/M_3lUoWERuOdXDFR,M/3lUoWERuOdXDFR
So we see that the device is not yet running properly, it still needs it’s Long-Term-Key, which is supplied in an envelope included with the EntropyKey. We tell ekeyd it’s Long-Term-Key like so:
# sudo ekey-rekey M/3lUoWERuOdXDFR AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
So finally we can check the EntropyKey’s status again:
# sudo ekeydctl list
NR,OK,Status,Path,SerialNo
1,YES,Running OK,/dev/entropykey/M_3lUoWERuOdXDFR,M/3lUoWERuOdXDFR
cat /proc/sys/kernel/random/entropy_avail
3968
You’ll note that the available entropy has significantly increased, and the EntropyKey should constantly replenish this pool, making sure you never run out of entropy.
A long long time ago I bought Ultimate Quake, which is a collection of Quake I, II & III (without expansions). Back then it cost about 50EUR, so it was excellent value. I’m providing this info to illustrate I do own a real legal copy of Quake III Arena.
However, I never did receive a CD Key for Quake III Arena, back then I assumed it didn’t matter because I never played online. Still you get that please enter your CD Key nag screen every time you start Quake III Arena, so as time passed by one learns to automatically hit Escape after starting Quake III Arena.
Only recently I learned that you can enter ‘jjjjjjjjjjjjjjjj’ as a valid CD Key, Quake III Arena will then no longer ask for a CD Key. I assume that servers running with sv_strictAuth on will still reject the key, but at least you won’t be nagged anymore…
In the past I wrote about profiling for camera profiling for UFRaw. Since then I’ve pretty much switched to Darktable, and I’ve learned a few things. So here we go again, with a vengeance…
The pretty images our camera’s output aren’t literally what the camera sensors see, there is a lot of proprietary postprocessing involved. Now when working in the RAW format we ditch all that postprocessing in favor of RAW sensor data. Which is good if we want maximum control and flexibility. The problem is that camera vendors don’t publish their proprietary postprocessing methods, which leaves us with a problem, how do we postprocess then?
In the past some good work on this topic has been done by Adobe, who published their DNG specification including color matrices. Color matrices are specifications on how camera native color is transformed into something that an end user might like, and ideally will be correct when viewed on a calibrated display.
Historically there have been some problems with the Adobe color matrices, the biggest of these is the rendition of red colors. To this day I still have no clue why this really is a problem. The only explanation I can think of is that this has been a deliberate compromise on Adobe’s part to safely render skin tones, at the cost of red rendition elsewhere. This is speculation at best…
So ideally we want an alternate enhanced color matrix that will render red as it’s supposed to, possibly at the cost of some skin tones rendition (rendition of skin tones is often a matter of taste, so we can fix that using the color zones “natural skin tones” preset).
Without going into all the nitty gritty details, there are basically two kinds of profiles, the first are XYZ matrices (these are often called color matrices) and are typically combined with a gamma curve, the second are LUT profiles. What’s a LUT, it’s a lookup table… The big difference is that with a XYZ matrix all color transformations are calculated on the fly, while LUTs are precalculated, so transforming color via a LUT is simply looking up an input color and it’s matching output color. The nice thing about LUTs is that they can deal with slight (nonlinear) deviations and can even be tweaked for creative purposes. So when generating a LUT profile, the profile is likely to pick up some of the peculiarities of your particular camera. XYZ matrices don’t have that problem since they are defined by only 3×3 coordinates in XYZ colorspace, and thus are quite generic by their very nature. This is also the reason why we are sticking to color matrices instead of supplying more detailed LUT profiles (besides diskspace usage).
All camera’s that can output a digital RAW format supported by LibRaw can be properly profiled. This effectively means that most compact camera’s can not be profiled (yes, some could using the CHDK firmware hack). Probably half of all digital bridge camera’s can output RAW and thus can be profiled as well. Pretty much all digital SLR camera’s are covered as well…
There are currently lots of color targets available on the market produced using different techniques and sold at very different prices. Here are some of the things you need to keep in mind…
First we can classify all targets into two groups, matte and semigloss… In theory semigloss targets can cover a larger gamut (range of colors) than matte targets, but most semigloss targets tend to glare, which is a big inconvenience when shooting the target. Most IT8 targets are semigloss, CMP’s Digital Target for example is matte… I personally prefer matte targets for their convenience since they are easily shot using a decent strobe, which is often hard to do with semigloss target. Semigloss targets are often shot best outdoors on a sunny non-cloudy day at an angle to prevent glare.
Second there’s the patch count, Gretag’s Classic ColorChecker only has 24 patches, while most IT8 targets have up to 288 patches, and CMP’s Digital Target even has 570 patches. When one only want to generate a color matrix this is not of particularly great relevance, the ColorChecker’s 24 patches are enough. But if one wants to do LUT profiles more patches tends to be better.
Third there’s the production technique, some charts like CMP’s Digital Target are produced on highend inkjets, which implies the target is very vulnerable to moisture making it more fragile than other targets. Most IT8 targets seems to be produced using traditional minilabs, and last but not least some (typically Gretag’s) targets are painted. In theory painted target should be superior in the sense that the manufacturer has greatest spectral control over the patches, which means you’re less vulnerable to metamerism when shooting the target under different lighting conditions.
Fourth there’s reference measurements, some targets like Gretag’s ColorChecker are very accurately produced to a single reference specification. Most IT8 targets are produced in batches, and will include per batch average reference data. The last and most accurate approach are individually measured charts, which means each chart has a reference which is specific to that chart alone, CMP does this for all it’s Digital Target’s and some IT8 vendors will also offer such services on request.
And last and least, don’t try to print your own target, even if you have a color calibrated printer, it will not be accurate enough!
If you want a target that you can easily use with commercial software as well you should probably buy a (relatively expensive) ColorChecker Passport.
If you want a convenient and highly accurate target you should probably buy an (expensive) CMP Digital Target. This is my current personal preference.
If you want an affordable high quality target, and don’t mind some extra fiddling to get the shot right, you could probably buy an IT8 target from Wolf Faust.
Small note, Wolf Faust really kicks ass, since he relicensed his reference files under the terms of the Creative Commons Attribution-Share Alike license.
If you already have a target, or just purchased or borrowed one, we can try to shoot the target. First check if the target is clean and has no damage to the color patches. Then depending on the type of target you have available we can determine the shooting conditions:
If you’re lucky enough to have a matte target available shooting it will be a breeze, since you can shoot the target head-on using a decent strobe. When possible put a dark grey or black piece of card behind the target. When I said decent strobe I meant something like a proper branded strobe like Canon, Nikon, Metz, etc. Please don’t do this with your generic asian no-brand flashes, since we don’t really know the quality of the light it produces, it might be fine for your photos, but I’d rather not use it for profiling purposes…
If you have semigloss target the current recommended way to shoot it, is to wait for a sunny day with little or no clouds. Then usually at noon when the sun is highest in the sky shoot the target when there are no clouds obscuring the sun. When possible put a dark grey or black piece of card or cloth behind the target. You probably need to position the target at an angle (with regard to the camera’s position) to prevent any glare…
… Shoot the target using a modern normal lens (no antiques (since lens coating can differ) or fisheye folks) at about 50mm or so, at a distance of about roughly 1 meter or 3 feet. The target should cover about 50-75% of the image surface. The dark piece of board behind the target should be visible on all four sides of the target.
Now comes the hard part, getting the exposure right… Unless your camera has a true RAW histogram, the histogram will be useless to you. The best exposure will likely look slightly overexposed as a preview image. The best way to approach this is to take a boatload of exposures from severely underexposed to severely overexposed and select the best exposure later on using Darktable.
So, if applicable set your strobe to full manual (1/2 strength for GN36, 1/4 strength for GN48, 1/8 strength for higher GNs). Then set your camera to full manual mode as well, set Exposure to 1/200 sec, and set the sensor sensitivity to it’s normal lowest possible setting (without pushing), which is usually ISO 100 or ISO 200. Don’t use any special modes which allow extra low or extra high ISO, since this is often digital trickery… Then close down the aperture all the way (usually f32 or something) and take a shot, open up the aperture 1/3rd stop and take another stop, keep opening up the aperture and taking shots all the way, until the aperture is fully open. Usually this means you have taken about 20-30 shots by now.
Now import the images you shot into Darktable, pick a target image which looks slightly overexposed, and open it in darkroom (develop) mode. Then disable all plugins except for these:
Make really sure you have the basecurve and sharpening disabled (which are enabled by default). You’ll notice the image getting darker… Then do spot white balance on the target’s grey patches. Then use the color picker (bottom panel) to check the Lightness value (the L from LAB), of the brighest white patch. You need to check you’re charts reference file for the exact Lightness you should match. It’s usually L 96 for ColorCheckers, L 97 for most CMP Digital Targets and L 92 for most IT8 targets. If the Lightness of the brightest white patch is L 99 or L 100 straight away, the image is probably overexposed (and useless), move to an image shot a 1/3rd stop darker. Now open up the exposure plugin, and start increasing the exposure until the Lightness value of the white patch in the image matches the Lightness specified in your charts reference file. If you need to apply more than +0.3 or +0.4 of digital exposure, you probably need to check an image that was shot a 1/3rd stop brighter.
Once you find the proper image, and equalize the Lightness of the brightest white patch to the chart’s reference value, you need to inspect the image for glare, if there is any glare whatsoever you need to redo everything again. This is once again why I prefer matte targets, they’ll save you some aggravation…
If everything is dandy exit darkroom mode, and export a 16bit TIFF image.
To do the actual calculations we need Graeme Gill’s excellent ArgyllCMS, particularly version 1.1.1 or higher. First we need to read the TIFF file for it’s color patch values, we do this using Argyll’s command line utilities:
# scanin -v -p -a -dipn IMG_1234.tiff \
/usr/share/color/argyll/ref/CMP_Digital_Target-3.cht CMP_DT3.txt
This command will output a diag.tif which you can use to see if the chart was properly recognized. Using the previously read values we can calculate the actual matrix:
# colprof -v -A "Canon" -M "Canon EOS 400D DIGITAL" -D "canon eos 400d" \
-C "Copyright (c) 2010 Pascal de Bruijn. Some rights reserved." \
-q l -a m IMG_1234
This command will output a IMG_1234.icc file which is a standard ICC color profile. To directly use it with Darktable you can copy it for Darktable to find:
# sudo cp IMG_1234.icc /usr/share/darktable/color/in/canon_eos400d.icc
There are other options to generate profiles for personal use, for this please read up on color management in general and the ArgyllCMS documentation.
Since there is no direct way to check if a matrix’ values are correct, I’m hesitant to directly accept profiles/matrix values from third parties. At this point I highly prefer for users to make properly exposed RAW chart images available to us, so I can do the calculations. Proper credit will be given to whoever supplies the chart.
The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.
This must have happened in thousands of series/movies, where some bloke makes an outrageously sounding claim about future possibilities. I happened to notice one the other day… Some guy picking up a CD (from the time where CDs were relatively new), and he says: “In the future we’ll be able to storage a hundred of these CDs on a single microchip”.
The silly thing is… actually we already sortof can… affordably… Since encoding 100 CDs x 70 minutes per disc x 60 seconds x 256kbit/sec VBR MP3 x 8 (bit) / 1024 (kilobyte) / 1024 (megabyte) ~ 13GB of data…
And if we play fast and loose with the microchip term… Oh wait… The future is now… It’s called a 16GB SecureDigital memory card, which can be bought at any electronics shop, for about 40EUR… Doh!
