2012
01.29

Color Management (On Linux)

There seems to be a lot of confusion about what color management is, what it is supposed to do, and most particularly how to use it on Linux. While most information below is generically applicable, in cases where I have to be specific I’ll focus on Ubuntu/GNOME/Unity.

The first thing to get out of the way is the simply question what color manament is supposed to do for you. Color management is used to get consistent and reliable results from device to device to device. So if I take an image with my color managed camera, and display it on my color managed display and print it with my color managed printer it should look nearly the same everywhere. This means it doesn’t per-se make your image look better (whatever that may mean for you). Also what color management can’t do for you is make crappy equipment better than it is. Any color management solution always has to work within the bounds of the equipment it is managing. Of course any color management solution tries to compensate for a device’s limits as best it can, but there are inherent limits. When these limits are hit, colors aren’t accurately reproduced anymore.

Now we need to get some terminology straight. Calibration is modifying a device’s characteristics to match a specification (for example changing brightness of a display). Characterization is recording the devices behavior for correction in software. These terms are often incorrectly used interchangeably (even by me, excuse me when I do). The end result of characterization is a (standard) ICC color profile.

While pretty much any device can be color managed , I’ll focus on displays for the rest of this article.

Now to color manage a display you need a device that can “read” (characterize) your displays characteristics. To characterize your display there are two types of devices you can use: colorimeters and spectrophotometers. Colorimeters are the most common device to characterize displays, as they are fairly affordable (100-200 EUR range). Colorimeters do have their limits, they are in essence just a very special purpose digital camera with only a handful of pixels. While I personally never had any issues, I’ve read about older colorimeters having trouble with new kinds of display technology like LED backlit displays, and some entry level colorimeters may not work as well with professional wide gamut displays (more on that later). The other option is a spectrophotometer, these devices are rather pricy, entry level spectrophotometers like the ColorMunki Photo, are priced slightly below 400EUR (if you see any device priced significantly lower, it’s likely that the device is not a true spectrophotometer). Spectrophotometers actually read a full spectrum of the light they receive, which means it produces a lot more detailed information. This means spectrophotometers are unlikely to be fooled by new technologies. Most spectrophotometers also include a reference light source, which means they can illuminate (for example) paper, so they can be used to profile printers (combined with ink & paper) as well.

So now we’ll need to explain some more concepts. So first I’ll tell you how silly talking about (for example) RGB 245/0/0 really is. Imagine owning a car, and you’re stuck with an empty tank. So using your last few drop of petrol you go to a petrol station. Say you live in Europe, and you say to the attendant, I’ll have 40, so the attendant fills up your car with 40 Liters of petrol. If someone living in the US says the same thing to an attendant at a US petrol station, they’ll get 40 Gallons of petrol. So, you’d say, well I did tell you properly, it’s RGB… But RGB really doesn’t mean anything at all. Since RGB just tells you, you are defining colors in three components: red, blue and green. It doesn’t say anything about what the reddest red is, the greenest green is and let not forgot about the bluest blue. To define this, the concept of colorspaces was created, a colorspace defines the range of colors a device can reproduce, this is also called a device’s gamut. RGB colorspaces are defined in CIE XYZ colorspace. They do this, because XYZ colorspace encompasses all colors the average human eye can see. All RGB colorspaces are a defined as a subset of XYZ colorspace. More importantly in the late nineties two of the most important colorspaces were defined: sRGB (by Microsoft and HP), and AdobeRGB (by, erhm… well… Adobe). sRGB was more or less defined as the average common denominator of most affordable displays. These days anything not explicitly defined a in different colorspace is assumed to be in sRGB. On the other hand, AdobeRGB was defined to encompass much more colors, with it’s main goal of covering most colors professional printing solutions could cover.

Next to defining RGB to be in a colorspace, there is still the issue that the human eye does not experience light in a linear fashion, so we need gamma encoding to make images not look like a murky mess. These days gamma 2.2 is universally accepted as a standard for displays. There are some caveats though. I own a cheap netbook, and it’s display for example seems to have a native display gamma of about 1.8, which means it lacks contrast.

And then there is the issue of whitepoints, since there is no such thing a “just” white. For most purposes, a white point of 6500K (this is at least true for both sRGB and AdobeRGB) is good as a standard neutral white. Higher temperatures in Kelvin make a display look blue (common with laptop displays), and lower temperatures in Kelvin more a display look more yellow.

And last there is the question of luminance, which is a snazzy term for brightness. If you work isn’t color critical, just put your display to a comfortable level (usually not too bright), if your work is color critical, it’s common to calibrate your display to 120cd/m2.

That said, there are some common issues to address. As I said the result of characterization is an ICC profile. ICC profiles usually have the file extension of .icc or on Windows .icm. Depending on the software which generated the profile, profiles can either be version 2 or version 4. And at least on Linux (but also true for older proprietary software), many programs may not properly apply version 4 profiles, so it’s best to stick with version 2 profiles for the time being. Luckily ArgyllCMS, the premier open profiling suite generates version 2 profiles by default.

Also, you need to be aware that most web browers aren’t color management aware (Safari & Firefox are the exception when properly configured). The W3C specified that “the web” should be in sRGB. This basically means that you should only upload sRGB images to websites, if you upload images that are not sRGB, they may not look as intended to your potential viewers (depending on which browser they use). The common problem is that people upload AdobeRGB images to the web, and they get complaints that the images look desaturated (since the web browser is assuming them to be sRGB, even though they are not).

Now back to display profiling, there are several ways to accomplish this on Linux. I’ve talked in the past about manually doing this with ArgyllCMS, which is a suite of commandline tools. There are however some front-ends available. The two most important ones are dispcalGUI and GNOME Color Manager. Both tools have their own target audience, dispcalGUI caters to advanced users who really know color management inside out. While GNOME Color Manager caters to entry level users, and tries to make everything as easy as possible. To be blunt, if everything in this article isn’t really obvious to you, your best bet is probably GNOME Color Manager. GNOME Color Manager generally provides sane defaults, and guides you through the process using a Wizard *cough* Druid (or whats-it-called?)…

Next some information on the general anatomy of display profiles. Display profiles in particular have three important components. There is the VCGT, the TRCs and the XYZ matrix. The first bit, the VCGT, also often called the VideoLUT, is a lookup table which is designed to correct your display’s whitepoint and potential aberrations between the R, G and B channels. The VCGT is loaded into your X11 driver, and only works if your driver is in 24 bit mode. When the VCGT is being loaded into X11 (usually in the login manager or just after logging in) you should see the colors of the display shift a bit. The VCGT is the only bit of the profile which is beneficial to all applications (as it’s applied by X11), the other two parts have to be actively applied by the application (if properly configured, more on that later). So next we have the TRCs which basically models your display’s gamma curve. And last the XYZ matrix determines what maximum red, blue and green are for your particular display. It’s possible to have an XYZLUT instead to get more detailed correction, however I’d never recommend this, as not all applications properly apply an XYZLUT.

Since the last two parts (TRC+XYZ) need to be applied by your color management aware applications, they need to be properly configured. To make this easier there is something called the XICC specification, which allows an “active” profile to loaded into X11 as well, this is very rudimentary though, as it just means the file is being loaded into the _ICC_PROFILE atom (which is basically like an environmental variable), so it can be easily picked up by color management aware applications. Applications most commonly use the LittleCMS library on Linux to apply the TRCs and the XYZ matrix.

GNOME Color Manager (via GNOME Settings Daemon) makes sure that a profile’s VCGT gets loaded into the X11 display driver, as well as setting the _ICC_PROFILE atom. You can check if the _ICC_PROFILE atom has been properly set using xprop:

# xprop -display :0.0 -len 14 -root _ICC_PROFILE

It’s known that proprietary (nVidia/ATi) drivers can cause problems, also dual head setups can complicate things.

Now, some applications do color management by default (assuming the _ICC_PROFILE atom has been properly set), this for example includes Eye of GNOME and Darktable. Other applications seem to ignore the _ICC_PROFILE atom by default, like Firefox and GIMP.

To check if a profile is being applied, you need to good test image to evaluate, I can highly recommend SmugMug’s Calibration Print for this. In GIMP’s particular case, load this image into GIMP, and go to Edit and Preferences. Go to the Color Management section. Then check the “Try to use the system monitor profile” box while looking at the image,  in most cases you should see a change (if not use xprop to check the _ICC_PROFILE atom), and more importantly you should be able to distinguish the top gray patches from each other.

Last there is the issue of images that were adjusted on uncalibrated displays, which is true for probably 99% of all images on the web. If the author had a low contrast unmanaged display, it’s likely he might have increased contrast in a particular image. When you take a look at that image on your color managed display (with proper contrast), it may look too contrasty. And on the flipside, if the author had a high contrast unmanaged display, it’s likely he might have decreased contrast in a particular image. When you take a look at that image on your color managed display (with proper contrast), it may look devoid of contrast. So it’s not weird to have discrepancies between managed and unmanaged setups.

With the above text I hope to have shed some light on color management in general and some of the particular issues regarding it’s use on Linux.

2012
01.21

I’ve been posting a fair amount of photography, imaging and color management lately. While colorimetry can be a good solution to display issues, but a lot of people don’t want to take it that far.

So say you’ve just gotten a new notebook, and like many notebooks the display looks a tad blueish, and you don’t want to invest in a full blown color management solution. There is a fairly simple way to address this issue at least to an extent, and it’s called xgamma (please note that xgamma might not work if your X11 setup is in 16bit mode, which is very unlikely on a modern system).

Now before making any changes it’s a good idea to get a good image to evaluate any changes with. I can highly recommend the Smugmug Calibration Print. So open the calibration print in your favorite image viewer, and do:

# xgamma -rgamma 1.0 -ggamma 1.0 -bgamma 0.9

You should see your display shift in color. Lots of notebook display also tend to lack contrast, so in theory you can use xgamma to compensate for that too:

# xgamma -rgamma 0.9 -ggamma 0.9 -bgamma 0.8

Again check the calibration print again, make sure you can clearly distinguish all the grey patches at the top of the image.

Now when you reboot your machine these settings will be lost. The best way I’ve found to automatically apply these settings seem to be via what’s called XDG Autostart, it’s basically a set of .desktop files that are run during session startup. Most big desktop environments (GNOME/XFCE/KDE) support these.

So, put the following into /etc/xdg/autostart/xgamma.desktop

[Desktop Entry]
Encoding=UTF-8
Name=Set display gamma corrections
GenericName=Set display gamma corrections
Comment=Applies display gamma corrections at session startup
Exec=xgamma -rgamma 0.9 -ggamma 0.9 -bgamma 0.8
Terminal=false
Type=Application
Categories=

Now reboot, and see your gamma settings being applied at during each new X11 login.

Please beware that the above correction are ballpark corrections, for real accuracy you really need to do proper color management.

2011
12.24

Darktable Unity Progress

Usually I don’t do a lot of “real” coding for Darktable, but I had some time on my hands today, and I implemented basic Unity integration for Darktable. Since I wasn’t familiar with libunity, nor was I really familiar with the depths of Darktable code let alone CMake, the implementation took me about 2 hours.

That said, have a look at the results:


You can also download the video for offline viewing if you prefer.

By the way, the bug I mentioned at the end of the video has been mitigated, which is a chic way of saying I kludged it so you won’t be bothered by it. But it’s not truely fixed.

2011
11.05

Since I did my last darktable 0.7 screencast library, some things have changed. So at the very least this warranted an update screencast.

The downside is that I was recovering from a cold, so these screencasts sound a bit rough:

Darktable 0.9 Update (download)

Darktable B&W Film Emulation (download)

Darktable Denoising (download)

Darktable Spot Removal (download)

Small mistake, it’s actually possible to remove spots by rightclicking them.

2011
08.20

There are millions of black & white photo plugins available. Some simple, some complex. When I recently got back a batch of real developed black & white film, I tried to investigate my scans to see how to emulate the effect (and possibly how to automate it).

The simplest approach I’ve been able to come up with involves blurring and decreasing contrast (with output levels). It can be automated with ImageMagick like so:

# convert -gaussian-blur 1x1 -filter triangle -resize 3000x2000 -level \!15%,\!95%,1.0 -colorspace gray -gaussian-blur 5x1 input.jpg output.jpg

Please note that doesn’t involve noise simulation yet, which seems to be hard to do with ImageMagick (tips are welcome). Please do note that I’m resampling to 6MPixels for convenience, you can use any resolution assuming you roughly scale along the 5 pixel Gaussian blur.

 

2011
08.11

Recently I’ve been fiddling a bit with Upstart, and in general I’m positive about the experience. Upstart offers a lot of flexibility and simple but very welcome features like real service supervision with respawning capabilities. There are a few downsides:

Relatively few SysV init scripts have been converted to Upstart jobs. This is logical since there are a lot of scripts to convert and all of them need testing before considering them production ready. That said, the progress over time in this area isn’t particularly overwhelming either.

Upstart is much harder to troubleshoot when unexpected things happen, this is to a degree inherent to Upstart’s parallel/event-based nature. Adding the following options to your kernel parameters does help a bit: ‘nosplash INIT_VERBOSE=yes init=/sbin/init noplymouth -v’.

As far as I know there is no way to have SysV init scripts depend on Upstart jobs. While this was to be expected since this is pretty hard to implement sensibly. But considering my first point (the fact that lots of SysV scripts still have be converted), this can be annoying. My advice, don’t fiddle with the the rc job! I did, and it gave me grief (Upstart hung on reboots), causing me to waste a day figuring out what went wrong. If you need a SysV script to depend on an Upstart job, remove the SysV script and convert it to an Upstart job yourself.

2011
08.08

As some of you may know configuring LDAP on Ubuntu Lucid can be a bit of a pain in the ass, especially getting it consistent throughout your infrastructure. So here’s way to relatively easily configure LDAP via Debconf preseeding.

First install debconf (it’s likely already installed, but doublecheck this):

# apt-get install debconf

Then put the following lines into debconf-ldap-preseed.txt so we can feed it to debconf later (obviously you need to adapt this configuration file to your own situation):

ldap-auth-config    ldap-auth-config/binddn    string    cn=proxyuser,dc=yourdomain,dc=com
ldap-auth-config    ldap-auth-config/bindpw    password    
ldap-auth-config    ldap-auth-config/dblogin    boolean    false
ldap-auth-config    ldap-auth-config/dbrootlogin    boolean    true
ldap-auth-config    ldap-auth-config/ldapns/base-dn    string    dc=yourdomain,dc=com
ldap-auth-config    ldap-auth-config/ldapns/ldap-server    string    ldap://10.10.10.10/
ldap-auth-config    ldap-auth-config/ldapns/ldap_version    select    3
ldap-auth-config    ldap-auth-config/move-to-debconf    boolean    true
ldap-auth-config    ldap-auth-config/override    boolean    true
ldap-auth-config    ldap-auth-config/pam_password    select    crypt
ldap-auth-config    ldap-auth-config/rootbinddn    string    cn=manager,dc=yourdomain,dc=com
ldap-auth-config    ldap-auth-config/rootbindpw    password    
libnss-ldap    libnss-ldap/binddn    string    cn=proxyuser,dc=yourdomain,dc=com
libnss-ldap    libnss-ldap/bindpw    password    
libnss-ldap    libnss-ldap/confperm    boolean    false
libnss-ldap    libnss-ldap/dblogin    boolean    false
libnss-ldap    libnss-ldap/dbrootlogin    boolean    true
libnss-ldap    libnss-ldap/nsswitch    note    
libnss-ldap    libnss-ldap/override    boolean    true
libnss-ldap    libnss-ldap/rootbinddn    string    cn=manager,dc=yourdomain,dc=com
libnss-ldap    libnss-ldap/rootbindpw    password    
libnss-ldap    shared/ldapns/base-dn    string    dc=yourdomain,dc=com
libnss-ldap    shared/ldapns/ldap-server    string    ldap://10.10.10.10/
libnss-ldap    shared/ldapns/ldap_version    select    3
libpam-ldap    libpam-ldap/binddn    string    cn=proxyuser,dc=yourdomain,dc=com
libpam-ldap    libpam-ldap/bindpw    password    
libpam-ldap    libpam-ldap/dblogin    boolean    false
libpam-ldap    libpam-ldap/dbrootlogin    boolean    false
libpam-ldap    libpam-ldap/override    boolean    true
libpam-ldap    libpam-ldap/pam_password    select    crypt
libpam-ldap    libpam-ldap/rootbinddn    string    cn=manager,dc=yourdomain,dc=com
libpam-ldap    libpam-ldap/rootbindpw    password    
libpam-ldap    shared/ldapns/base-dn    string    dc=yourdomain,dc=com
libpam-ldap    shared/ldapns/ldap-server    string    ldap://10.10.10.10/
libpam-ldap    shared/ldapns/ldap_version    select    3
libpam-runtime    libpam-runtime/profiles    multiselect    unix, ldap

Now we feed the above file to debconf:

# cat debconf-ldap-preseed.txt | debconf-set-selections

And finally we need to switch nss to include LDAP lookups:

# auth-client-config -p lac_ldap -t nss

Optionally make sure homedirs are created on login:

# echo -e 'session required\t\t\tpam_mkhomedir.so' >> /etc/pam.d/common-session

One of the other advantages is that this may also benefit future upgrades, since all the settings are preseeded through debconf it’s less likely a future update/upgrade will break your setup. This is purely speculative since I haven’t done any actual upgrades yet.

 

2011
07.31

lzma -2

When compressing data we all know we have several options, gzip most commonly used or bzip2 when we need better ratios. Recently we have also seen the introduction of lzma, which is horribly slow by default but gets us amazing ratios.

That said, when compressing your next file, you might want to consider trying lzma -2, it compresses slightly better than bzip2 –best, while still being approximately as fast as gzip –best… Seems like a nice trade off…

2011
07.24

Gnoduino FTW

Recently I’ve been re-entering the world of electronics, and more particularly microcontrollers. Back in the day I had some introductory experience with Parallax’s BASIC Stamp, which wasn’t that great. Only recently I discovered the Arduino project, which is absolutely awesome.

The original Arduino IDE is written in Java however, and that makes for poor integration on the Linux platform, particularly with Ubuntu Unity. Luckily enough someone has been busy to do a simple rewrite of the original IDE in Python and GTK, making a fully Linux native IDE, and it’s called Gnoduino, and I even have a PPA for it.

I did a screencast on the general topic of Arduino and the differences between the original IDE and the Gnoduino IDE:


You can also download the video for offline viewing if you prefer.

2011
05.27

So today I was migrating our HP SSSU utility from an x86 to an x86-64 machine, only to find our I couldn’t execute the utility anymore because it was on a x86-64 machine, I wanted to lookup the version number of the utility to find a matching x86-64 version. So I was too lazy to log onto the old machine, so I used strings to find it’s version number, only to find a crazy surprise:

SSSU for HP StorageWorks Command View EVA
9.2.0
Version: %s
Build: %s
Error closing https connection
Error closing https connection
Total regression differences = %u
Press return to exit
%.6d%c
                           oooo$$$$$$$$$$$$oooo
                      oo$$$$$$$$$$$$$$$$$$$$$$$$o
                   oo$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$o         o$   $$ o$
   o $ oo        o$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$o       $$ $$ $$o$
oo $ $ '$      o$$$$$$$$$    $$$$$$$$$$$$$    $$$$$$$$$o       $$$o$$o$
'$$$$$$o$     o$$$$$$$$$      $$$$$$$$$$$      $$$$$$$$$$o    $$$$$$$$
  $$$$$$$    $$$$$$$$$$$      $$$$$$$$$$$      $$$$$$$$$$$$$$$$$$$$$$$
  $$$$$$$$$$$$$$$$$$$$$$$    $$$$$$$$$$$$$    $$$$$$$$$$$$$$  '''$$$
   '$$$''''$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$     '$$$
    $$$   o$$$$$$$$M$i$c$h$e$l$&$R$o$g$e$r$$$$$$$$$$$$$$$$$$$     '$$$o
   o$$'   $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$       $$$o
   $$$    $$$$$$$$$$$$$$$$$$w$e$r$e$$$$$$$$$$$$$$$$$$$$' '$$$$$$ooooo$$$$o
  o$$$oooo$$$$$  $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$   o$$$$$$$$$$$$$$$$$
  $$$$$$$$'$$$$   $$$$$$$$$$$$$$$h$e$r$e$$$$$$$$$$$$     $$$$''''''''
 ''''       $$$$    '$$$$$$$$$$$$$$$$$$$$$$$$$$$$'      o$$$
            '$$$o     '''$$$$$$$$$$$$$$$$$$'$$'         $$$
              $$$o          '$$''$$$$$$''''           o$$$
               $$$$o                                o$$$'
                '$$$$o      o$$$$$$o'$$$$o        o$$$$
                  '$$$$$oo     ''$$$$o$$$$$o   o$$$$''
                     ''$$$$$oooo  '$$$o$$$$$$$$$'''
                        ''$$$$$$$oo $$$$$$$$$$
                                ''''$$$$$$$$$$$
                                    $$$$$$$$$$$$
                                     $$$$$$$$$$'
                                      '$$$''

This is obviously a reference to The Hitchhiker’s Guide to the Galaxy (why?).