Downloading photos

Acquiring photos
and camera control

Exif tags

Desktop interaction

Image viewers

Raw conversion

Compared to ...

Gimp 2.2

High ISO noise

Portable Storage

Help needed

Download links

Recent changes

Contact details

OpenRAW

linux reg

October 21, 2006


Nikon D70 under Linux


This page lists Linux specific information, links and tools for Nikon D70 digital SLR owners.

While focusing on the D70, some of the information might be useful for Linux based digital SLR owners, especially for RAW capable models.


Please join in the effort to get Nikon and other camera makers either open up their RAW format or adopt a standard. You may start by reading raw-flaw and sending the petition email to Nikon. This is doubly important since Nikon made a wrong turn and started encrypting the WB of the D2X and the new "entry level" D50.


Downloading photos:

Mass storage mode:

From  Kubuntu breezy onwards The d70 is automatically recognized and mounted. KDE 3.5 automatically adds a desktop icon which should make windows users happy.

PTP mode:
Kubuntu breezy pops a "Camera" desktop icon upon connection.
gphoto2 version 2.1.5 will download all photos to the current directory with 'gphoto2 -P'.

Acquiring photos and camera control:

gphoto2 2.1.5 has some support for the D70,

$gphoto2 --auto-detect
Model                          Port
----------------------------------------------------------
Nikon DSC D70 (PTP mode)       usb:
Nikon DSC D70 (PTP mode)       usb:001,006

You can capture an image with,

$gphoto2  --capture-image
New file is in location / on the camera

The intriguing summary option provides camera status, with a few known fields and many unknowns. I wish (readwrite) would mean can be set by gphoto2, but this will probably never happen.

$gphoto2 --summary
Camera summary:
Model: D70
  device version: V1.01
  serial number:  (null)
Vendor extension ID: 0x0000000a
Vendor extension description: Nikon PTP Extensions
Battery Level(0x5001):(read only) (type=0x2) Range [0 - 100, step 1] value: 100
Image Size(0x5003):(readwrite) (type=0xffff) Enumeration [
        '3008x2000',
        '2240x1488',
        '1504x1000'
        ] value: '3008x2000'
Compression Setting(0x5004):(readwrite) (type=0x2) Enumeration [0,1,2,4,5] value: 2
....
Property 0xd017:(readwrite) (type=0x1) Range [-3 - 3, step 1] value: 0

Steve Harris sent me this patch for libgphoto 2.1.5 (cd libgphoto2-2.1.5/camlibs/ptp2; patch < libgphoto-2.1.5-nikon.patch), which  "adds name support for most of the D70 extension parameters, and renders some of them in a human readable way".
He also led me to ptpcam, which in fact can change camera settings. However it is a command line only affair, and not a particularly nice one to be honest. It seems like all the pieces are in place, but I fail to see how it is usable without a good GUI. Is there such a beast? Can you write one?

gtkam version 0.1.12 compiled and produced an executable and a gimp plug-in. The plug-in adds "Capture from camera" and "Download from camera" entries to the gimp Acquire submenu. Both are useless. Download presenting a garbled 120x160 image. Capture activates the camera and exists with an obscure error message. Manually detecting the camera each activation is no fun either.

EXIF tags:
  • Reading
The best program to read D70 NEF and JPG metadata is Phil Harvey's ExifTool. All the data discussed in the D70 manual (and then some) is recognized (I know that for a fact because I sent the initial data to Phil ...). The full set is quite large so it is useful to create a .ExifTool_config file to view just a subset (Look for "Shortcut Tags" in Phil's page). The latest version can write and transfer tags as well. Minor drawback - it is written in Perl.
  • Restoring
Raw conversion via dcraw discards EXIF tags.  neftags2jpg transfers them back to the jpg.  neftags2jpg uses exiv2 (get version 0.7 or newer).

After compiling and installing, simply call
$ neftags2jpg dsc_XXXX.nef myfile.jpg
  • exifdate is a small shell script which sets the file date to the time a photo was taken. Requires a working ExifTool.

Desktop Interaction:
KDE Konqueror shows NEF thumbnails automatically, since they are in fact TIFF files, declaring the small 120x160 thumbnail as the "main image".  Remember to increase file size in the "Previews and Meta data" Konqueror configuration to at least 6MB.

Image viewers:
gqview is a capable and fairly speedy viewer. To view NEF thumbnails, click Edit/Options.../Filtering, double click on the tiff entry and add ;.nef to the list.
gqview, which uses the GTK library to load images, finds the first (smaller 120x160) thumbnail but knows nothing of the second, larger one. Here is an appalling patch for version 1.5.X to remedy the situation, which requires a working nefextract in you path first. You want jpegtran in your path as well to enable auto rotation (there is a good chance it is already installed for your system).  Right click and choose (Edit -> In the Gimp...) to complete a reasonable single photo workflow.

Photo Management
Goggle Picasa has a working Linux version which (slowly) handles raw images, but may be better than other free solutions at this point.

Raw conversion:
If you are interested in improving your photographic skills (and in some cases the images), it is best to shoot in RAW mode.  However this shifts more of the responsibility in developing prints into your hands, a decision not be taken lightly.
    • Command line
For some time Dave Coffin's dcraw has been the only option. Dave has made an amazing job reverse engineering every major (and some non major) raw formats on the planet. In addition to adding new cameras at a phenomenal rate, Dave is improving the usability of dcraw by adding automatic rotation, color management, more advanced interpolation methods and so on. Here is my modified V8.40 (October 21 2006) version which contains some minor changes used by the gimp plugin (see below) which add reading tone curves and thumbnails.

 UFRaw (0.9.1) has a standalone, 16 bit version for batch processing.
    • Commercial  Software
Bibble 4.x Has a Linux version with D70 support. You can download a time limited trial version and check it out.

I bought Bibble Pro in April 2006 and has been using it since. I have mixed feelings about Bibble. The workflow is well thought of and it is full featured while keeping reasonably fast. However, the main reason I got bibble was to be able to print but this is broken in several ways and even though it is a professional, paid for software in Linux space no one will hear your screams. I am not always happy with the resulting image but I may be expecting too much or fretting about non issues. The Linux version is also prone to crashes, hangs, and getting stuck while happily using 100% of the  CPU.
    • Free for non-commercial use
Mark from dpreview gave the link to through-the-lens RawViewer, a Java/dcraw based viewer and conversion tool.

Personal note: I don't really care for Java. In a true java spirit the user interface has some oddities - The directories are sorted in a semi random order, hidden directories are shown by default, "camera curve" has to be selected for any new file even though it appears as the default menu choice, the jpeg in portrait orientation came out as landscape, if you don't select an output directory the program goes through the motions, prints a large friendly DONE at the end, but does nothing. I did not find any "Auto exposure" button, but I might have missed it.
  • Free for Linux
Anton Kast has sent me a link  to the download page of LightZone, a java based commercial browser and editor which is free for Linux users. LightZone approach is original and based on two key ideas - manipulating the tonal zones visually and applying changes using a stack model. LightZone is using the dcraw conversion matrices and which leads to "washed out" D70 images.
    • Gimp 2.2 plugin
At the start of 2004 there were practically no good options for RAW processing on Linux, free or not. The situation has improved to a point where you can choose between one commercial product and several free options, three of them related gimp plugins. The first is the original PTJ's RawPhoto. The second is my fork (RougePhoto 1.39e, October 21 2006), which is substantially different by now. The third is Udi Fuchs UFRaw, another fork by a D70 enthusiastic. I guess D70 attracts more geeks than other models ... UFRaw, now in version 0.9.1, is undoubtedly the leader in open source raw conversion tools.
 
Please use RougePhoto with at least gimp 2.2 and gcc 3.3. Don't send me emails about problems with earlier versions.

Note: You must use my modified dcraw (above) with RougePhoto. In the future I might integrate it in, but preliminary tests show no speed improvement, so it is very low on my priority list.

Note: If you are getting images with strange colors, try running with  'env LANG=en_US gimp' (Assuming you run under a different setting). I hope to eliminate this workaround when I figure out what is going on.
The locale problem is hopefully fixed in the latest version. Please report if it persists for you.

Note: The plugin will load any file supported by dcraw, which are many.
However, note that for your colors to be closer to reality you need to find the appropriate input ICC profile for your camera. See below.

I tested RougePhoto with the following formats
  • CRW (Canon 300D and 10D)
  • MRW (Minolta)
  • NEF  (D100 and D70)
  • ORF (Olympus E-300)
I would appreciate example files from other formats if you have them.

Note: EXIF tags support is available for
D70 when neftags2jpg is installed. However, gimp uses libexif, and version 0.6.9 (the latest available) handling of NEF makernote is broken. For now I advice removing the test for Nikon in libexif/exif-data.c. It is far from obvious how to do this in Debian - see here for details.

RougePhoto now contains some color management capability via Little CMS. Read Udi's excellent Color management page. To setup RougePhoto, download Nkx_D70_4_0739_02_308.icm from Udi's site and place it in ~/.gimp-2.2/profiles/input. Copy (Or soft link) any output profiles to ~/.gimp-2.2/profiles. See the README file for the gimprc entry needed to complete the setup. I have made only very limited testing using images from other cameras, but duplicating the above with the appropriate files should work. Some 10D owners reported they had no success with the 10D profile in Udi's site. I suggest using either Fred Miranda's or Bibble profiles for the 10D (Can someone find out if there is an official Canon profile?).

The recent version of dcraw (7.30 and up) contains "conversion matrices" from camera RGB to sRGB prepared by Thomas Knoll. If you can't set an input profile, those provide much better output. Great addition, Dave!  As of version 1.36e, RougePhoto will use those automatically if it can't find an input profile. An override toggle is provided in the Setup tab.
Below is a comparison for a Canon 10D image.

cat thumb
Canon thumbnail
cat none
No Color Management
cat bibble
Bibble 10d profile
cat dcraw
dcraw sRGB

Unfortunately the D70 conversion is not as good results in a "washed out" image. Using the Nikon profile is the best bet at this stage for D70 nef's.

Plugin Features:
Controls
  • Spot WB, White and Black Point.
  • Gamma, Exposure and Black level.
  • Auto Exposure/Black point (with separate exposure and Color Clipping threshold).
  • Color Auto Level.
  • Saturation & Shadows adjustments
  • B&W mode.
Display
  • Visual display of overexposure,underexposure, and color clipping.
  • Zooming, Panning and Resizing of preview (thanks to Dov Grobgeld GtkImageViewer)
  • Intensity and Color channel histograms.
  • Manual & Automatic orientation (when supported by camera).
  • Plugin title shows image name and camera make/model.
Setup
  • User control of White/Black/Color Clip cutoff%.
  • Choice of saturation method. Either linear or the one from Udi Fuchs  UFRaw. I don't use saturation in raw conversion, but Udi says it is an improvement.
  • Gamma and Log tone curves
  • Camera Tone Curve. dcraw loading code taken from Udi Fuchs  UFRaw.  (Again only for D70. I will add more if anyone sends me a dcraw patch to read tone curves from other raw formats)
  • Load-able Tone Curves. Unpack curves (Dec 07) into your gimp curves directory (typically ~/.gimp-2.2/curves) and they should appear in the tone curve selection menu.
  • Automatic selection of input color profile. Menu of Destination color profiles. Download Nkx_D70_4_0739_02_308.icm from Udi's site and place it in ~/.gimp-2.2/profiles/input. Place (Or soft link) any output profiles to ~/.gimp-2.2/profiles. See the README file for the gimprc entry needed to complete the setup.
General
  • Image comment contains RougePhoto conversion parameters for later reference.
  • Fast startup.  (2-3 seconds on my 1800XP Athlon using the default setup).
  • Conversion takes approximately 12-13 seconds on same machine.
  • EXIF support for D70 via neftags2jpg.
  • Some customization options via gimprc.
  • Gimp 2.2 ready.
  • Support for Gimp 2.2 thumbnail API. D70 now uses small 120x160 thumbnails for speed.

Raw Plugin Screen Shot
(KDE 3.3.1 - Keramik, GTK 2.4.4 - Aquativo-1.5)

scr shot


Most of the controls and numbers are hopefully self explanatory. A little experimentation should do the trick for others. Here are some explanations and comments on less than obvious features.

  • Left click on the image performs spot white balance, which is based on colors around the spot. It is possible to spot a white/black point (i.e. brightest/darkest) as well by changing Neutral Point to Bright Point/Black Point.
  • The preview will re-size when the size of main window changes. The right mouse button zooms in, Ctrl-Right zooms out, and Middle button  moves around. Alternatively you can use the '=' and '-' keys for zooming, and 'z'/'1' to fit the image back (Similar to gimp Ctrl-E and Ctrl-Shift-E).
Press '9' (As in 9 O'clock) for CCW orientation, '3' for CW orientation, and '0' for the default.
  • Note that a spot white balance changes both temperature and green tint (available only under the WB tab). Temperature determines the ratio between red and blue, while green tint determines the ratio between green and red/blue. So, if you are unhappy with a spot WB press "Camera WB" to return to camera supplied values, since just resetting the temperature will generally give a different result. You can change the red/blue/green ratios directly in the WB tab, but in my opinion this is hardly an effective way to get the correct balance.
  • Note the various options under the SETUP tab. You should experiment and determine what works best for you, then please let me know. Hopefully we can reach an agreement on the best setup, and can either reduce the number of options or at least have a better default setup.
  • For the curves I introduced my own file format. I am not very happy about this, but unaware of any standard format.  I went for text format and utmost simplicity.  There are no load/save buttons - all curves in the gimp curves directory are automatically loaded on startup.
Note that there is no automatic Auto Bright/Black after switching to a new curve.

I re-created  Fotogenic Provia v3.3 and others from the points in D70 knowledge base, but am not sure if I did this correctly or not. Conversion used a gamma value of 0.45 (1/2.2), but you can make a Gamma adjustment for the loaded curve if you wish. I would appreciate some feedback about this if possible.
  • I made a first effort to improve the Histogram. While more cluttered, I think it is more informative to see all colors + intensity together. It helps if you have a large screen and willing to give it more screen real estate by selecting a larger height (via the gimprc option).

Compared to ...:

What can you expect from RougePhoto (version 1.3Xe)? Some (photographically boring) examples follow.

Long Disclaimer:
I am totally unqualified to make such a comparison. The number of factors is great. The choice of Color Profiles can make a huge difference in how the end result looks. I have very little experience using Nikon Capture, Bibble Pro or C1PRO. I ignore important issues such as user interface, workflow features, platform availability and more trivial stuff such as price and speed (but yes, NC is soooo sloooow). The number of examples is too small. Everything below is biased personal opinion. You certainly want to conduct your tests rather than assuming I did the right thing.

Comment: The mediocre results I get from C1PRO makes me wonder if I have some setting problem. Perhaps some kind soul can suggest a possible source of the problem?

Sakari Tanhua
compares dcraw and Bibble here.

Currently I have,
  • 30 days trial version of C1PRO version 3.5.2.61 (Expired)
  • A trial version of Nikon Capture 4.1 (Expired)
  • Bibble 4.1 Windows trial (Expired)
  • RawViewer 1.5.3
  • RawShooter essentials 1.1.3
  • Light Zone for Linux
C1PRO sharpening is set to "soft look", and "Convert to Web Destination". NC sharpening is set to default, whatever that is.
RougePhoto output is "as-is" from the plugin (i.e. no sharpening) without any gimp adjustments. RawViewer has no "Auto EV".


Here are the highlights
  • After taking into account exposure & black point issues, RougePhoto colors are very similar to NC colors.
  • The grass is always greener at Bibble, but look to me a trifle too much. I assume this is an effect of a different destination color space.
  • C1PRO default auto exposure seem to be set too high. same with NC.

The first image is a setup taken indoor in natural light.


C1PRO, Auto Exposure

C1PRO, Manual Exposure +0.60

Nikon Capture 4.0, Default

Nikon Capture 4.0, Manual Exposure +0.33

Bibble 4.1, AutoLevel

RawViewer 1.5.3, Manual EV +20
raw shooter
RawShooter essentials 1.1.3, Auto all
light zone
Light Zone (Default)

RougePhoto, Auto EV (gamma curve)

RougePhoto, Manual EV (D70 normal curve)

C1 Auto Exposure was surprisingly aggressive. NC default is underexposed. I am still looking for an "Auto Exposure" button in NC, some people pointed me to  NC "Auto Contrast" button in the LCH tab. It seems like the right button,  however it is too aggressive as well, so I will not bother showing the result. I entered a value of  +0.33EV manually.

The next is outdoor on a New Zealand cloudy day.


C1PRO, Auto EC

C1PRO, Default

Nikon Capture, Default

Bibble 4.1, AutoLevel

RougePhoto, Auto EV (gamma curve)

RawViewer 1.5.3
light zone default
LightZone (default)



Again C1 auto EC is overboard, and default is under.


C1PRO, Auto EC

Bibble 4.1, AutoLevel

Nikon Capture, Default

RawViewer 1.5.3

RougePhoto, Auto EV (gamma curve)
rawShooter dog
RawShooter essentials 1.1.3
 

400% crops comparing the difference in handling of the dreaded
moire.

dcraw vng
dcraw (VNG)
dcraw ahd
dcraw (AHD)
bible
Bibble (4.2.6)
RS
RawShooter 1.1.3
NC
Nikon Capture 4.2
vng
dcraw (VNG)
ahd
dcraw (AHD)
bible
Bibble (4.2.6)



Gimp 2.2:

High ISO noise:

At higher ISO's, CCD noise is quite visible. It seems Nikon Capture (and other RAW converters)  automatically apply noise reduction in higher ISO's, but at the moment Linux only users have to apply noise reduction using Gimp.

Below is a 100% crop from a photo shot at ISO 800. Below are the results from the professional Noise Ninja (Bill's Box Only), Gimp Despeckle plugin, Dcam Noise 2, GREYCstoration, Film Grain RemoverNoise Remover plugin, and the slightly improved noise remover described later.

One problem in evaluation is that it is easy to spot artifacts, but hard to measure the amount of noise reduction. It is impossible to figure out how to apply the "same amount" of reduction when each method offers totally different, and typically cryptic controls.
For the moment I will ignore this issue, unless it is obvious (see Grain remover below).

original
original

noise ninja
Noise Ninja
despeckle
Despeckle (default)
noise remover
Noise remove
Noise remove CC 20
Noise Remove + CC

Film grain remove
dcam2
Dcam2 Noise
greyc
GREYCstoration


Below are the difference images (via Grain Extract layer mode) to the original and a closer crop for each.
Despeckle, beside being horrendously slow, exhibits strong color dependent patterns. Noise-remover shows smearing, halos and large color loss, especially in the blue channel. Grain remover looks great, unfortunately this is because in it's default setting it removes little noise. In higher settings it exhibits considerable smearing and lose of detail.
I recently got an email from Peter Chiocchetti pointing me to Dcam Noise 2. This seem to be an impressive plugin with many (perhaps too many?) controls. However by using the defaults and just adjusting "Filter Threshold" I got a very good result which seem to have less artifacts than Noise Ninja and still reducing the noise nicely.
Niels Kristian Bech Jensen sent me the link to GREYCstoration, another heavy duty plugin. Just like Dcam Noise 2, it has various cryptic controls and no advice on how to use them, so the chance of doing justice to both are low. GREYC does a nice job of removing noise while preserving color and detail. However it is much slower, turning even exploration via the preview quite painful.


Noise Ninja


Despeckle


Noise Remove


Noise Remove + CC


Film Grain Remove
dcam2 dif
Dcam Noise 2
dmam noise 2
greyc diff
GREYCstoration
greyc

The small, non scientific change I made to noise-remover is to simply disallow pixel changes greater than TH (say 20). The result is obviously more noisy, but does not exhibits strong color patterns. The price is high on another front - I was reluctant to write a plugin for what seems a simple task, however iterating over an image is way over script-fu ability, and even python-fu, which I used, needs to be augmented with heavy machinery in the form of numarray. You can get my python-fu script here.
With the Dcam and GREYC in my Gimp menu I am happy to say Linux users are not at a disadvantage anymore - and as usual - the price is just right.

Portable Storage:
While browsing yesterday I saw that the Vosonic X's Super drive 6230 supports D70 RAW files, and after a brief googling session bought one on impulse over the internet. The unit has a built in CF card reader and a 2.5" screen for viewing (+ zooming), can play audio and video and has an upgradeable hard drive (I got the 80G version).

Unfortunately all the online complaints turn out to be true. The unit is an unattractive silver plastic brick, the buttons feel as if they would last only a short time, the sound quality is marginal and the limited video codec support mean I will have to convert any video I wish to view them on the tiny screen. The user interface is clunky, user manual outdated and useless, and the list goes on.

While you can indeed view NEF files, EXIF info is not available. It is there for JPG's but thanks (again) to Nikon failing to fix this bug even in Firmware 2.0, ISO is displayed as 0 .

On the other hand the unit has an upgradeable drive and batteries, USB 2.0 is fast (backward compatible with 1.1 as well), is Linux compatible by choice and not by hacking, and may improve with upcoming firmware updates. I hope I will have some good things to say after using the unit in the field.

(Oct 2006)  The  80 gig on the X drive has filled up nicely. On the whole it was a good buy, as the unit travels well and still fully functional. Vosonic keeps supporting the model by releasing firmware updates and fixing small issues on the way. My music now resides on an ipod video, so the marginal audio quality is not an issue anymore.

Help needed:
As you can see, my English language & graphic design skills leave a lot to be desired. If you are experienced in either and would like to help, I will appreciate it greatly.

Download links:

Local links

neftags2jpg - Transfer EXIF tags from NEF to JPG.
nefextract - Extract full size preview from a NEF file.
modified dcraw (October 21 2006) - (required by plugin)
RougePhoto (1.39e October 21 2006) - Gimp 2.2 plugin.
Tone curves (Dec 07) - Tone curves for RougePhoto plugin.
noise-remover + CC plugin (Dec 02)

External links

ExifTool - Read EXIF data.

UFRaw- gimp raw plugin.

RawPhoto - PTJ gimp raw plugin.

ptpcam - control camera via USB

libgphoto2 patch



gqview

gimp

gPhoto

Dcam Noise 2

Recent changes:

  • Added File Grain Remover to Noise reduction methods.
  • Small fix: ignore request for camera curve when no curve in file. (Dec 06)
  • Bug fix: Incorrect auto Adjust with user curve. (Dec 07)
  • Added Provia 4.0, P&S 4.0 and EV3 4.0 curves. (dpreview link)
  • Added "Compared to" section (Dec 7).
  • Added a section on RawViewer.
  • Allow value of 0 for "auto level" level.
  • gphoto 2.1.5
  • small bug with tone curve selection.
  • Another try at fixing locale problem.
  • Added .ntc crack program.
  • Initial support for color profiles.
  • Fixed color profile for non D70.
  • upgrade neftags2jpg to exiv-0.6.1
  • added contrast and shadows controls.
  • Added Dcam Noise 2 plugin.
  • Added RawShooter example.
  • Added example for dcraw conversion matrices.
  • Added GREYCstoration.
  • Choice between better&slower AHD interpolation and lesser&faster VNG.
  • Updated neftags2jpg for exiv2 0.7.
  • Added a section about VP 6230.
  • Added LightZone.

Contact details:
Please send comments, suggestions and feedback to  jheled@gmail.com