2013
04.14

Often after encoding a video, and playing it back, it seldomly looks as good as I’d expect, especially compared to professionally produced DVDs for example. Now while there are likely a multitude of differences, one of them seems to be acutance (perceived sharpness), which can be enhanced by applying some sharpening after scaling the source material before encoding the video.

The following small script encodes a source video input to a anamorphic widescreen PAL DVD resolution WebM file at a nominal (total) bitrate of 2Mbit/sec, while applying some sharpening:

#!/bin/sh
INPUT="$1"
SCALE_OPTS="-sws_flags lanczos -s 720x576 -aspect 16:9"
SHARP_OPTS="-vf unsharp=3:3:0.5:3:3:0"
VIDEO_OPTS="-vcodec libvpx -g 120 -lag-in-frames 15 -deadline good -profile 0 -qmax 51 -qmin 11 -slices 4 -b 1800000 -maxrate 3800000"
AUDIO_OPTS="-acodec libvorbis -ac 2 -ab 192000"

avconv -i "${INPUT}" ${SCALE_OPTS} ${SHARP_OPTS} ${VIDEO_OPTS}           -an -pass 1 -f webm -y "out-${INPUT}.webm"
avconv -i "${INPUT}" ${SCALE_OPTS} ${SHARP_OPTS} ${VIDEO_OPTS} ${AUDIO_OPTS} -pass 2 -f webm -y "out-${INPUT}.webm"

 

Now what particularly matters are the unsharp parameters, which can be divided into two triplets, the first set of three: luma (brightness (greyscale) information) and the second set of three: chroma (color information). Each of these two sets has three parameters, of which the first two are the horizontal and vertical matrix dimensions (e.g. the evaluation area), in our case a small matrix of 3 by 3 is the only configuration that makes sense. A 5 by 5 matrix is possible but will give a exagerated effect and halo artifacts. The last parameter in the triplets is the strength (respectively 0.5 for luma, and 0 for chroma), which means we’re enhancing acutance for luma, and we’re leaving the chroma unmodified. For luma strength values between 0.5 and 1.0 are likely to be the useful range depending on taste and source material. Typically you’d want to leave chroma be, but in some cases you could possibly use this as a poor mans denoising method by specifying negative values, which effectively turns it into a blur effect.

 

2013
04.14

If you’re running a GNOME or Unity desktop (and probably recent versions of KDE too), you may notice differences in color rendition between different applications. The difference you’re seeing is between applications that apply the system configured display profile and those that don’t. For example Eye of GNOME and Darktable do this by default, GIMP for example doesn’t…

Now, as many people have noticed most displays render color quite differently, and display profiles are a technical means to correct that to some degree. There are several means for obtaining a display profile, one is to buy a measurement device (called a colorimeter) and actually measure your particular display. Some vendors supply a display profile ICC file on a CD that came with the display. And lastly more recent displays apparently provide information which can be used to generate a display profile via EDID (which is a protocol for information exchange via VGA/DVI/HDMI). The respective methods have been listed in order of decreasing accuracy. For a bit more in-depth information you might want to consider reading this.

At least since distributions have been shipping colord and GNOME Color Manager (so I’m guessing since Oneiric for Ubuntu users), colord actually queries your display via the EDID protocol, to extract the required information to generate an automatic display profile, which allow certain applications to correct for the displays behavior.

We Need You

Now, recently we’ve begun to have the impression that some vendors may be shipping bogus information in their displays (possibly under the assumption that it would not be the used anyhow). But currently we have no information to substantiate this.

Please read this and this first before continuing.

I’d like to ask you, to submit your EDID generated profile to the gnome-color-manager-list (you can post to the list without subscribing, your submission will be moderated and thus will take a few days to turn up) including the following:

  • Subject: [EDID] Display Make + Model
  • Attach ~/.local/share/icc/edid-*.icc
  • Display Make (if it’s a laptop, then the laptop make)
  • Display Model (if it’s a laptop, then the laptop model)
  • The Displays Age (approx. how long ago did you buy it)
  • Duty Cycle (light usage on average a few hours a day, heavy usage approx. 8 or more hours a day).
  • The output of  xprop -display :0.0 -len 14 -root _ICC_PROFILE
  • Subjective Impression (download this SmugMug calibration image, and load it into GIMP, then go the GIMP’s Preferences, to go Color Management, and then check/uncheck Try to use system monitor profile while keeping an eye on the image, tell us what looks most realistic to you (checked/unchecked) and why…

After more than 2000 submissions colord-0.1.34 was released which should detect and disable cases where the displays are supplying bogus information via EDID. Based on the current statistics it seems 4% (or thereabouts) of the displays supply bad information.

Working around bad EDID

Assuming some vendors actually provide bad information via EDID, you might need a way to disable this automatically generated profile. In older versions of GNOME Color Manager (3.6 and earlier) there wasn’t an easy way to disable this. There is however a feasible workaround. Install argyll on your system. Then assign /usr/share/color/argyll/ref/sRGB.icm to your display. (Go to the GNOME System Settings, Choose the Color applet, Choose your display, click Add Profile, select Other Profile, and then select Argyll’s sRGB.icm).

2012
07.30

Depending on your type of memory card reader, you may have noticed the following on Ubuntu (and possibly other distributions too). When you connect a USB flashdrive to your system an icon pops up informing you the drive has been mounted. When you insert (for example) an SD card into your cardreader, it may happen that another icon pops up using the same icon as the flashdrive.

While this isn’t the biggest problem in the world, it’s certainly a nuisance, as you’d need to hover over each icon to see the tooltip which explains to you which icon represents what. Ideally you’d want the SD card to show up with an appropriate SD card icon.

Which icon is displayed ultimately depends on disk management done by udisks and more importantly udev. In /lib/udev/rules.d/80-udisks.rules (do NOT modify this file) we find the following rules:

SUBSYSTEMS=="usb", ENV{ID_MODEL}=="*SD_Reader*", ENV{ID_DRIVE_FLASH_SD}="1"
SUBSYSTEMS=="usb", ENV{ID_MODEL}=="*Reader*SD*", ENV{ID_DRIVE_FLASH_SD}="1"
SUBSYSTEMS=="usb", ENV{ID_MODEL}=="*CF_Reader*", ENV{ID_DRIVE_FLASH_CF}="1"
SUBSYSTEMS=="usb", ENV{ID_MODEL}=="*SM_Reader*", ENV{ID_DRIVE_FLASH_SM}="1"
SUBSYSTEMS=="usb", ENV{ID_MODEL}=="*MS_Reader*", ENV{ID_DRIVE_FLASH_MS}="1"

The above rules are matched against the device names which are passed to the kernel. With one of my cardreaders, this sadly doesn’t match:

$ dmesg | grep -i Direct-Access
scsi 12:0:0:0: Direct-Access     Generic  Compact Flash    0.00 PQ: 0 ANSI: 2
scsi 12:0:0:1: Direct-Access     Generic  SM/xD-Picture    0.00 PQ: 0 ANSI: 2
scsi 12:0:0:2: Direct-Access     Generic  SDXC/MMC         0.00 PQ: 0 ANSI: 2
scsi 12:0:0:3: Direct-Access     Generic  MS/MS-Pro/HG     0.00 PQ: 0 ANSI: 2

To create new rules, we first need to figure out what USB vendor/product IDs belong to the cardreader, you can just identify USB devices attached to your computer like so:

$ lsusb
Bus 002 Device 012: ID 048d:1345 Integrated Technology Express, Inc. Multi Cardreader

Just run the command once before attaching the device and once after attaching the device and look for the differences, typically it’ll be the last device in the list. Once we have this information create a new file (replace pmjdebruijn with your own nickname, use exclusively alphanumeric characters):

$ sudo nano -w /etc/udev/rules.d/80-udisks-pmjdebruijn.rules

In this file we put the following lines:

# ITE, Hama 00055348 V4 Cardreader 35 in 1 USB
SUBSYSTEMS=="usb", ATTRS{idVendor}=="048d", ATTRS{idProduct}=="1345", ENV{ID_INSTANCE}=="0:0", ENV{ID_DRIVE_FLASH_CF}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="048d", ATTRS{idProduct}=="1345", ENV{ID_INSTANCE}=="0:1", ENV{ID_DRIVE_FLASH_SM}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="048d", ATTRS{idProduct}=="1345", ENV{ID_INSTANCE}=="0:2", ENV{ID_DRIVE_FLASH_SD}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="048d", ATTRS{idProduct}=="1345", ENV{ID_INSTANCE}=="0:3", ENV{ID_DRIVE_FLASH_MS}="1"

You’ll notice the idVendor and idProduct coming from the lsusb line above, also the ID_INSTANCE needs to have matching LUNs with the dmesg lines above. Once you’re done, doublecheck and save the file, and then you can reload the udev rules:

$ sudo udevadm control --reload-rules

Any newly mounted memory cards should get a proper icon now.

Not all cardreaders may be as easy as illustrated as above, for example I have a wonderful cardreader that provides no useful information at all:

$ lsusb
Bus 002 Device 009: ID 05e3:0716 Genesys Logic, Inc. USB 2.0 Multislot Card Reader/Writer
$ dmesg | grep -i Direct-Access
scsi 6:0:0:0: Direct-Access     Generic  STORAGE DEVICE   9744 PQ: 0 ANSI: 0
scsi 6:0:0:1: Direct-Access     Generic  STORAGE DEVICE   9744 PQ: 0 ANSI: 0
scsi 6:0:0:2: Direct-Access     Generic  STORAGE DEVICE   9744 PQ: 0 ANSI: 0
scsi 6:0:0:3: Direct-Access     Generic  STORAGE DEVICE   9744 PQ: 0 ANSI: 0
scsi 6:0:0:4: Direct-Access     Generic  STORAGE DEVICE   9744 PQ: 0 ANSI: 0

In such a particular case, you’ll need to experiment by actually inserting various types of memory cards, and checking what device got mounted, and what LUN is it, in the following example I inserted an SD card, which got mounted as sdk, which turns out to be LUN 0:2, which we need for the ID_INSTANCE entries:

$ mount | grep media
/dev/sdk1 on /media/FC30-3DA9 type vfat (rw,nosuid,nodev,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,showexec,flush,uhelper=udisks)
$ dmesg | grep sdk
sd 12:0:0:2: [sdk] Attached SCSI removable disk
sd 12:0:0:2: [sdk] 248320 512-byte logical blocks: (127 MB/121 MiB)
sd 12:0:0:2: [sdk] No Caching mode page present
sd 12:0:0:2: [sdk] Assuming drive cache: write through
sd 12:0:0:2: [sdk] No Caching mode page present
sd 12:0:0:2: [sdk] Assuming drive cache: write through
 sdk: sdk1

Another peculiarity (or feature) of this drive is that it has 5 LUNs instead of 4, this is because it actually has two SD card slots, one for full size SD cards and one for microSD cards. In the end, after some fiddling, I ended up with:

# Genesys Logic, Conrad SuperReader Ultimate
SUBSYSTEMS=="usb", ATTRS{idVendor}=="05e3", ATTRS{idProduct}=="0716", ENV{ID_INSTANCE}=="0:0", ENV{ID_DRIVE_FLASH_CF}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="05e3", ATTRS{idProduct}=="0716", ENV{ID_INSTANCE}=="0:1", ENV{ID_DRIVE_FLASH_SM}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="05e3", ATTRS{idProduct}=="0716", ENV{ID_INSTANCE}=="0:2", ENV{ID_DRIVE_FLASH_SD}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="05e3", ATTRS{idProduct}=="0716", ENV{ID_INSTANCE}=="0:3", ENV{ID_DRIVE_FLASH_MS}="1"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="05e3", ATTRS{idProduct}=="0716", ENV{ID_INSTANCE}=="0:4", ENV{ID_DRIVE_FLASH_SD}="1"
2012
06.23

Arduino vi mode pedal

In the past I was a vim user, mostly because a full vim install had nice syntax highlighting, since recent versions of nano also do syntax highlighting and nano has become available in the default installs of many distributions I’ve switched to nano as editor of choice.

However some colleagues came across the vim clutch the other day, which is a cool hardware hack, to switch between vi’s command and text entry modes using your foot. Coincidentally I had ordered a StealthDuino 32U4 which I received last week. The StealthDuino is basically a miniaturized Arduino Leonardo (in USB stick format), which easily does USB HID (Keyboard & Mouse). The other cool and convenient thing about the StealthDuino is that it has a QRE1113 infra-red proximity sensor on-board, so it’s possible to sense if anything is closely overhead. Which means it can effectively be used as a pedal.

My Arduino sketch can be downloaded here. When you load this sketch into a StealthDuino you can control it directly with your foot (no pedal casing required), however I’d recommend only operating it with your socks on, controlling it with your boots on will quite likely significantly decrease the units lifespan.

 

2012
06.19

Ransomware Unlocker

Some days ago, I came across a Windows machine where a lot of files were renamed (locked-filename.[four random lowercase alphabetic characters]) and no longer readable by the respective applications.  In one of the home directories there was a randomly named executable which many anti virus agents didn’t see as dangerous (including the anti virus agent installed on the machine in question, namely AVG). We checked using virustotal, back then only 15 (or so) anti virus products would detect the file in question, now (as I’m writing this), the detection rate has gone up to 29 anti virus products.

But, regardless of the origin, we were still stuck with lots of unreadable “locked” files. Now, we had access to religious backups, so we could have just reverted to the day before, but I opted to have some fun first.

I transferred a few sample files (two Word documents and a WAVE audio file, both locked and originals from backup) to a Linux machine. Running the file(1) utility on these locked files, they were all identified as data, while the originals were clearly identified as Word documents and WAVE audio. So I was pretty sure something changed in the contents. Next up I ran strings(1) on the locked and original version of one the the Word documents, and strings(1) returned plain text in both cases. So I knew the files weren’t entirely scrambled, but since file(1)‘s main mechanism to identify data formats is looking at the first few bytes in a files, the obvious theory would be that only the first part of these files were scrambled.

After searching a bit for a nice binary diff utility I found vbindiff(1) because xdelta(1) wasn’t cutting the mustard. Running vbindiff(1) on one of the Word documents (diffing the original against the locked version) it became immediately apparent that the first 4096 bytes were scrambled. Same story for the other Word document, but it was less obvious for the WAVE audio file. The difference is that classic Word documents (.doc, not .docx) have a headers with lots of 0×00 and 0xFF bytes in there. Now within the same locked file multiple 0×00 bytes weren’t scrambled into the same byte value, so some form of crypto (with a key) was being applied. When looking at two different original Word documents I noticed that a large part of the header was nearly identical between to the different Word documents. So I took a look at the two respective locked Word documents, and those were largely identical too. From this we can infer that the private key used to encrypt the first 4096 bytes is most likely static between locked files (at least on this system).

Considering the fact that a simple static private key seems to have been used and the fact that locked files weren’t even entirely encrypted, it was my guess the algorithm probably wouldn’t be very sophisticated too. Would it be a simple XOR operation? To find out, I wrote some quick and dirty FreePascal code to read the first 4096 bytes from an original file and the first 4096 bytes from a locked file, and XORed them against each other, effectively outputting the private key (at least, that was the theory). Now after I ran said utility against my three samples files, the resulting private key was identical in all three cases (even for the WAVE audio file). So I was right. It was a simple XOR operation using a static private key.

The next challenge was writing another small utility (again in FreePascal) which reads the first 4096 bytes from a locked file, and XORs it with the data from my generated private key file, and writes it back to an unlocked file, but after processing the first 4096 bytes the rest of the file would be copied verbatim. After running this new utility on all of my samples the resulting unlocked files were identical to the original files. So it really worked. It was this simple.

I built both above utilities in FreePascal, because the Pascal language is what I fall back to whenever I have to code up something quickly. A nice side effect is that the FreePascal code should be fairly portable. You can download the sources here.

Now on a even more amusing note, if you place a 4096 byte file with only zero bytes in it on your filesystem before the ransomware is activated it will most likely by accident generate it’s own private key, as 0×00 XOR key = key.

2012
06.15

As most of you probably know already, there is a cool (and affordable) little colorimeter available now called the ColorHug, and it’s open-source too (including all companion software).

As the ColorHug’s firmware is still being improved, some people have noticed a profile created with the ColorHug makes their display shift excessively to the red, possibly due to a slight measurement inaccuracy.

A display profile generally consist of two main parts, first there is the vcgt (sometimes also called VideoLUT), which is loaded and applied by X11 itself. This is usually a correction for a displays white point (which is where it goes wrong). And the second part is gamma+matrix (which is gamma/hue/saturation correction). So to avoid the red shift we have to skip the first part of profile creation.

To prepare I recommend you (try to) do the following (for this particular procedure):

  1. Note down your display old settings (if you care to go back to these).
  2. Reset your display’s settings to factory defaults.
  3. Adjust the display’s brightness to a comfortable level (you really often don’t need maximum brightness).
  4. Generally it’s a good idea to leave contrast at the manufacturers default.
  5. Change the displays color temperature to 6500K if possible (You might notice your display shift a bit to the yellow).

Then execute the following commands in a Terminal

# targen -v -d 3 -G -f 64 make_model
# ENABLE_COLORHUG=1 dispread -v -y l make_model
# colprof -v -A "Make" -M "Model" -D "Make Model" \
          -C "Copyright (c) 2012 John Doe. Some rights reserved." \
          -q l -a G make_model

The above commands skip vcgt creation with dispcal and do a fairly simple set of measurements and create a fairly simple ICC profile. This simplicity gets us increased robustness in the profile’s creation at the expense of potential accuracy. To be honest I wouldn’t be surprised if commercial vendors use a similar strategy in their entry-level colorimetry products for the consumer market.

You’ll need to either manually import the resulting profile into GNOME Color Manager (to setup the profile at login), or directly configure it in programs like the GIMP. You can load an image like this in GIMP to check if the resulting profile makes sense. Please do mind GIMP has color management disabled by default, so you need to set it up in the Preferences.

Even with the above method, the resulting profile may still be a bit off in the reds (though it will only be visible in color managed applications). If this is still an issue for you, you could try the Community Average CCMX, or possibly my Dell U2212HM CCMX, for which I’ve gotten decent results on non-Dell displays too.

2012
06.09

I’ve been screencasting a while now, mostly about Darktable. And from time to time people ask me how I do it, and what software and hardware I use. So here goes nothing…

Since my main topic is Darktable (a free software photography application), it means my target audience is primarily photographers who use free software. Which means my video’s should be easy to view on a random Linux/BSD desktops. Considering the only video/audio codec that is available on most newly installed Linux desktop are Theora and Vorbis respectively, these were going to be my primary publishing formats. The fact that Theora and Vorbis have been the longest supported format for HTML5 Video is a big plus too (since Firefox 3.6 if I recall correctly). And I surely didn’t want to motivate/require anybody to install Flash to view my video’s.

Another point of concern was audio quality. In general when watching other peoples screencasts, the often poor audio quality was for me the biggest annoyance. Especially for longer video’s where I don’t want to listen to 20 minutes of someone talking through static noise. So I went a bit overboard with this and bought an M-Audio FastTrack MkII (which is plain USB Audio, no special driver required) and a Rode NT1a Kit which I later on mounted to a Rode PSA1 Studio Arm.

Which brings me to my choice of recording application. I can’t say I tried them all, but recording with ffmpeg seemed to slow down my machine too much. So I settled on recordmydesktop and more particularly the gtk-recordmydesktop frontend. After some experimenting I found recording just a part of my screen (1920×1200) to be a nuisance. So I settled on doing all screencasts on my laptop recording fullscreen (1280×800). The recordmydesktop application defaults to recording 15 frames per second, which seems to be fine for my purposes. It defaults to recording audio at a 22050 Hz sampling rate, with me being a sucker for audio quality I changed to that 48000 Hz, which is commonly used on DVDs and other professional audio applications. One of recordmydesktop’s potential disadvantages is that it only encodes it’s capture to Ogg/Theora/Vorbis (.ogv) which luckily for me really isn’t an issue at all. I do max out the encoding quality to 100% for both audio and video.

When publishing my screencasts on the web I just use the HTML5 video support of modern browsers. I use the .ogv file produced by recordmydesktop directly, I don’t re-encode to reduce the bitrate or anything, as the bitrate is already acceptable to begin with, and I don’t want to degrade the quality any more than I have to. While in the past I only provided the .ogv, I recently also caved in providing an .mp4 (H264/AAC) fallback video to be able to support the ever increasing ubiquity of tablets, and secondarily to be able to support browsers like Safari which don’t support free media formats like Ogg/Theora/Vorbis out of the box. So now I’m using ffmpeg to transcode my video, however there are a couple of concerns here. My original recordings were recorded at a resolution of 1280×800, while most tablets (and most importantly The Original iPad), only supports video at a resolution of 1280×720 (H264 Level 3.1), so it would likely choke on it. That said, in many cases it’s not very useful to have 1280×800 on most tablets anyway as 1024×768 is a common resolution for 10″ tablets. So I settled on resizing my screencasts to 1024×640 (which also reduced the bitrate a bit, in the process making it more suitable for mobile viewing). Initially I tried to do the audio using the MP3 audio codec, however iPads seem to dislike that, while Android tablets handled it just fine. So I had to go with AAC and while Ubuntu’s ffmpeg isn’t built with FAAC support, it does however have ffmpeg’s builtin AAC encoder called libvo_aacenc, which isn’t as good as FAAC, but it had to do. So in the end my conversion commandline for ffmpeg is this:

avconv -i input.ogv -sws_flags bicubic -s 1024x640 -vcodec libx264 -coder 1 \
-flags +loop -cmp +chroma -partitions +parti8x8+parti4x4+partp8x8+partb8x8 \
-me_method umh -subq 8 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 \
-i_qfactor 0.71 -b_strategy 2 -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -bf 3 \
-refs 5 -directpred 3 -trellis 1 -flags2 +bpyramid+mixed_refs+wpred+dct8x8+fastpskip \
-wpredp 2 -rc_lookahead 50 -coder 0 -bf 0 -refs 1 -flags2 -wpred-dct8x8 \
-level 30 -maxrate 10000000 -bufsize 10000000 -wpredp 0 -b 1200k \
-acodec libvo_aacenc -ac 1 -ar 48000 -ab 128k output.mp4
2012
06.02

Since I did my last darktable 0.9 screencast library, some things have changed. So at the very least this warranted an update screencast.

Darktable 1.0 Update (download)

Darktable Archiving & Backup (download)

These are the first screencasts that should be viewable on most tablet devices too, albeit with slightly degraded quality.

2012
01.29

Color Management (On Linux)

There seems to be a lot of confusion about what color management is, what it is supposed to do, and most particularly how to use it on Linux. While most information below is generically applicable, in cases where I have to be specific I’ll focus on Ubuntu/GNOME/Unity.

The first thing to get out of the way is the simply question what color manament is supposed to do for you. Color management is used to get consistent and reliable results from device to device to device. So if I take an image with my color managed camera, and display it on my color managed display and print it with my color managed printer it should look nearly the same everywhere. This means it doesn’t per-se make your image look better (whatever that may mean for you). Also what color management can’t do for you is make crappy equipment better than it is. Any color management solution always has to work within the bounds of the equipment it is managing. Of course any color management solution tries to compensate for a device’s limits as best it can, but there are inherent limits. When these limits are hit, colors aren’t accurately reproduced anymore.

Now we need to get some terminology straight. Calibration is modifying a device’s characteristics to match a specification (for example changing brightness of a display). Characterization is recording the devices behavior for correction in software. These terms are often incorrectly used interchangeably (even by me, excuse me when I do). The end result of characterization is a (standard) ICC color profile.

While pretty much any device can be color managed , I’ll focus on displays for the rest of this article.

Now to color manage a display you need a device that can “read” (characterize) your displays characteristics. To characterize your display there are two types of devices you can use: colorimeters and spectrophotometers. Colorimeters are the most common device to characterize displays, as they are fairly affordable (100-200 EUR range). Colorimeters do have their limits, they are in essence just a very special purpose digital camera with only a handful of pixels. While I personally never had any issues, I’ve read about older colorimeters having trouble with new kinds of display technology like LED backlit displays, and some entry level colorimeters may not work as well with professional wide gamut displays (more on that later). The other option is a spectrophotometer, these devices are rather pricy, entry level spectrophotometers like the ColorMunki Photo, are priced slightly below 400EUR (if you see any device priced significantly lower, it’s likely that the device is not a true spectrophotometer). Spectrophotometers actually read a full spectrum of the light they receive, which means it produces a lot more detailed information. This means spectrophotometers are unlikely to be fooled by new technologies. Most spectrophotometers also include a reference light source, which means they can illuminate (for example) paper, so they can be used to profile printers (combined with ink & paper) as well.

So now we’ll need to explain some more concepts. So first I’ll tell you how silly talking about (for example) RGB 245/0/0 really is. Imagine owning a car, and you’re stuck with an empty tank. So using your last few drop of petrol you go to a petrol station. Say you live in Europe, and you say to the attendant, I’ll have 40, so the attendant fills up your car with 40 Liters of petrol. If someone living in the US says the same thing to an attendant at a US petrol station, they’ll get 40 Gallons of petrol. So, you’d say, well I did tell you properly, it’s RGB… But RGB really doesn’t mean anything at all. Since RGB just tells you, you are defining colors in three components: red, blue and green. It doesn’t say anything about what the reddest red is, the greenest green is and let not forgot about the bluest blue. To define this, the concept of colorspaces was created, a colorspace defines the range of colors a device can reproduce, this is also called a device’s gamut. RGB colorspaces are defined in CIE XYZ colorspace. They do this, because XYZ colorspace encompasses all colors the average human eye can see. All RGB colorspaces are a defined as a subset of XYZ colorspace. More importantly in the late nineties two of the most important colorspaces were defined: sRGB (by Microsoft and HP), and AdobeRGB (by, erhm… well… Adobe). sRGB was more or less defined as the average common denominator of most affordable displays. These days anything not explicitly defined a in different colorspace is assumed to be in sRGB. On the other hand, AdobeRGB was defined to encompass much more colors, with it’s main goal of covering most colors professional printing solutions could cover.

Next to defining RGB to be in a colorspace, there is still the issue that the human eye does not experience light in a linear fashion, so we need gamma encoding to make images not look like a murky mess. These days gamma 2.2 is universally accepted as a standard for displays. There are some caveats though. I own a cheap netbook, and it’s display for example seems to have a native display gamma of about 1.8, which means it lacks contrast.

And then there is the issue of whitepoints, since there is no such thing a “just” white. For most purposes, a white point of 6500K (this is at least true for both sRGB and AdobeRGB) is good as a standard neutral white. Higher temperatures in Kelvin make a display look blue (common with laptop displays), and lower temperatures in Kelvin more a display look more yellow.

And last there is the question of luminance, which is a snazzy term for brightness. If your work isn’t color critical, just put your display to a comfortable level (usually not too bright), if your work is color critical, it’s common to calibrate your display to 120cd/m2.

That said, there are some common issues to address. As I said the result of characterization is an ICC profile. ICC profiles usually have the file extension of .icc or on Windows .icm. Depending on the software which generated the profile, profiles can either be version 2 or version 4. And at least on Linux (but also true for older proprietary software), many programs may not properly apply version 4 profiles, so it’s best to stick with version 2 profiles for the time being. Luckily ArgyllCMS, the premier open profiling suite generates version 2 profiles by default.

Also, you need to be aware that most web browers aren’t color management aware (Safari & Firefox are the exception when properly configured). The W3C specified that “the web” should be in sRGB. This basically means that you should only upload sRGB images to websites, if you upload images that are not sRGB, they may not look as intended to your potential viewers (depending on which browser they use). The common problem is that people upload AdobeRGB images to the web, and they get complaints that the images look desaturated (since the web browser is assuming them to be sRGB, even though they are not).

Now back to display profiling, there are several ways to accomplish this on Linux. I’ve talked in the past about manually doing this with ArgyllCMS, which is a suite of commandline tools. There are however some front-ends available. The two most important ones are dispcalGUI and GNOME Color Manager. Both tools have their own target audience, dispcalGUI caters to advanced users who really know color management inside out. While GNOME Color Manager caters to entry level users, and tries to make everything as easy as possible. To be blunt, if everything in this article isn’t really obvious to you, your best bet is probably GNOME Color Manager. GNOME Color Manager generally provides sane defaults, and guides you through the process using a Wizard *cough* Druid (or whats-it-called?)…

Next some information on the general anatomy of display profiles. Display profiles in particular have three important components. There is the VCGT, the TRCs and the XYZ matrix. The first bit, the VCGT, also often called the VideoLUT, is a lookup table which is designed to correct your display’s whitepoint and potential aberrations between the R, G and B channels. The VCGT is loaded into your X11 driver, and only works if your driver is in 24 bit mode. When the VCGT is being loaded into X11 (usually in the login manager or just after logging in) you should see the colors of the display shift a bit. The VCGT is the only bit of the profile which is beneficial to all applications (as it’s applied by X11), the other two parts have to be actively applied by the application (if properly configured, more on that later). So next we have the TRCs which basically models your display’s gamma curve. And last the XYZ matrix determines what maximum red, blue and green are for your particular display. It’s possible to have an XYZLUT instead to get more detailed correction, however I’d never recommend this, as not all applications properly apply an XYZLUT.

Since the last two parts (TRC+XYZ) need to be applied by your color management aware applications, they need to be properly configured. To make this easier there is something called the XICC specification, which allows an “active” profile to loaded into X11 as well, this is very rudimentary though, as it just means the file is being loaded into the _ICC_PROFILE atom (which is basically like an environmental variable), so it can be easily picked up by color management aware applications. Applications most commonly use the LittleCMS library on Linux to apply the TRCs and the XYZ matrix.

GNOME Color Manager (via GNOME Settings Daemon) makes sure that a profile’s VCGT gets loaded into the X11 display driver, as well as setting the _ICC_PROFILE atom. You can check if the _ICC_PROFILE atom has been properly set using xprop:

# xprop -display :0.0 -len 14 -root _ICC_PROFILE

It’s known that proprietary (nVidia/ATi) drivers can cause problems, also dual head setups can complicate things.

Now, some applications do color management by default (assuming the _ICC_PROFILE atom has been properly set), this for example includes Eye of GNOME and Darktable. Other applications seem to ignore the _ICC_PROFILE atom by default, like Firefox and GIMP.

To check if a profile is being applied, you need to good test image to evaluate, I can highly recommend SmugMug’s Calibration Print for this. In GIMP’s particular case, load this image into GIMP, and go to Edit and Preferences. Go to the Color Management section. Then check the “Try to use the system monitor profile” box while looking at the image,  in most cases you should see a change (if not use xprop to check the _ICC_PROFILE atom), and more importantly you should be able to distinguish the top gray patches from each other.

Last there is the issue of images that were adjusted on uncalibrated displays, which is true for probably 99% of all images on the web. If the author had a low contrast unmanaged display, it’s likely he might have increased contrast in a particular image. When you take a look at that image on your color managed display (with proper contrast), it may look too contrasty. And on the flipside, if the author had a high contrast unmanaged display, it’s likely he might have decreased contrast in a particular image. When you take a look at that image on your color managed display (with proper contrast), it may look devoid of contrast. So it’s not weird to have discrepancies between managed and unmanaged setups.

With the above text I hope to have shed some light on color management in general and some of the particular issues regarding it’s use on Linux.

2012
01.21

I’ve been posting a fair amount of photography, imaging and color management lately. While colorimetry can be a good solution to display issues, but a lot of people don’t want to take it that far.

So say you’ve just gotten a new notebook, and like many notebooks the display looks a tad blueish, and you don’t want to invest in a full blown color management solution. There is a fairly simple way to address this issue at least to an extent, and it’s called xgamma (please note that xgamma might not work if your X11 setup is in 16bit mode, which is very unlikely on a modern system).

Now before making any changes it’s a good idea to get a good image to evaluate any changes with. I can highly recommend the Smugmug Calibration Print. So open the calibration print in your favorite image viewer, and do:

# xgamma -rgamma 1.0 -ggamma 1.0 -bgamma 0.9

You should see your display shift in color. Lots of notebook display also tend to lack contrast, so in theory you can use xgamma to compensate for that too:

# xgamma -rgamma 0.9 -ggamma 0.9 -bgamma 0.8

Again check the calibration print again, make sure you can clearly distinguish all the grey patches at the top of the image.

Now when you reboot your machine these settings will be lost. The best way I’ve found to automatically apply these settings seem to be via what’s called XDG Autostart, it’s basically a set of .desktop files that are run during session startup. Most big desktop environments (GNOME/XFCE/KDE) support these.

So, put the following into /etc/xdg/autostart/xgamma.desktop

[Desktop Entry]
Encoding=UTF-8
Name=Set display gamma corrections
GenericName=Set display gamma corrections
Comment=Applies display gamma corrections at session startup
Exec=xgamma -rgamma 0.9 -ggamma 0.9 -bgamma 0.8
Terminal=false
Type=Application
Categories=

Now reboot, and see your gamma settings being applied at during each new X11 login.

Please beware that the above correction are ballpark corrections, for real accuracy you really need to do proper color management.