Fri, 01 Apr 2011

Canon DPP Batch Processing

Smashing a square peg into a round hole

[This information was collected using Digital Photo Professional 3.9.3 on Windows. It may not be applicable to other versions.]

Meet Roadblock

In my current attempt at storing my data for posterity (as if anyone cared), I needed to convert all my photos from RAW-format to JPEG. Part of the problem was, that I had more than 13.000 RAW files in over 370 directories. Luckily DPP supports batch conversion, but only for files within a single directory. So did I really want to initiate a batch process for 370 directories manually? You sir, second row, wearing a yellow Bikini, are right. The answer to that question is, of course: “No frelling way!”

Have a Sandwich

What were my options?

  • Find another software that works the way I want Well, the thing is, that RAW conversion is not a well defined process, so each RAW-processor “develops” pictures in a slightly different way. And I happen to like DPP’s way. So that’s a “no go”.
  • Force DPP to work the way I want Sounds intriguing to me, so I had found my winner. Checking for any command-line parameters for DPPViewer.exe resulted in.. well, nothing. But there were other executables and the most promising was DPPBatch.exe. Starting DPPBatch without any parameters even displays a menu where .vbf files can be loaded. Aha! As it turns out DPPViewer communicates with DPPBatch via temporarily created .vbf-files in the output-directory. A rather old-school approach to IPC, but something that can be exploited for my needs. I only needed to figure out how to create my own .vbf-files to use the DPP converter from the command-line.

Recipe for a Solution

However, after analysing the file format, it turned out to be a bit more complicated than expected, because of recipes that are required in .vbf-files. Analysing a recipe’s format and how it interacts with settings stored within a RAW file itself are a completely different beast altogether and nothing that is accomplished within a few minutes.

So I decided to quickly whip up a Perl script that scans for all RAW files in a directory recursively, stores their original location and moves them to a single directory. Then a single batch process could be started from within the UI of DPP to convert all files at once, after which the script can move the RAW files (and newly created JPEG files) back to their respective directories.

Actually it was not that easy because DPP is still a 32-bit application and ran out of memory when it tried to load that number of files, metadata and thumbnails it. So I had to spread them over a few directories with 2.000 files each. Which allowed me to convert all files by starting only 7 batch processes manually. Compared to the original number of 370 this was highly acceptable however.

.vbf File Format

Nonetheless here is what I’ve found out about the format of .vbf-files. Perhaps someone else can fill in the blanks or find something useful to do with that information.

Data Types

  • Numbers: Unsigned 32-bit big endian integers
  • Strings: The null-terminated string is preceeded by a 32-bit big endian integer specifying the byte-length of the string (including the terminating \0). The characters in the string are 1 byte wide and use the current Windows codepage, not UTF-8 encoding. Way to go!
  • Recipe: Just like strings, recipes start with the number of bytes for the following recipe-chunk. For DPP 3.9.3 that are 1026 bytes.
SversionThe string "[version 1.10]"
SoutputDirectory where the converted files will be saved
Sfilename IF filename IS "\0xff\0x00"
  • suffix is appended to the source's filename (e.g. IMG_0001.jpg becomes IMG_0001suffix.jpg)
  • counter_init and counter_digits can be ignored
  • *the field suffix does not exist!
  • The resulting filename is generated by appending a number that is counter_digits long to filename, with the number being initialised by the value of counter_init (e.g. filename042.jpg if counter_digits were 3 and counter_init 42)
Sicc[5]Absolute paths to 5 different colour profiles
Nkind_of_fileOutput file format just like in the combo box:
0 = JPEG
1 = TIFF 8-bit
2 = TIFF 16-bit
3 = TIFF 8-bit + JPEG
4 = TIFF 16-bit + JPEG
NcompressionJPEG compression quality ranging from 10 to 100
Njpeg_optimizealways 1
affects only JPEGs by reducing their size. So my guess is: optimisation of huffman tables. And [JPEGsnoop]( shows different distributions within those tables when using one or the other setting.
Njpeg_no_jfifalways 0
When resize is activated it is actually 4, so maybe this is more than just a boolean flag but a bitfield.
NwidthWidth and height are always in pixels. Any settings in the UI regarding the units of measurement are only used in combination with the DPI-setting to calculate the pixel-size
N?set to the same value as 'external' by the UI. However changing this value to an arbitrary value doesn't change anything, by the look of things.
RrecipeSome kind of global recipe?
Nimg_countNumber of image-chunks following:
SfilenamePath of the source file
RrecipeRecipe for this image. How does it interact with the global one? How with the settings stored within the RAW file itself?

Update 2012-01-05

Thanks to commenter DSops for pointing out Canon’s free Digital Imaging Developer Programme, which includes an SDK for converting RAW files. However the FAQ states:

Q: What are the criteria for joining the programme?
A: To be eligible to join Canon's Digital Imaging Developer Programme,
   you should be operating as a legally registered company.
   Applications from other users will be considered at Canon's discretion.

So it might or might not work out for you.

Enable Javascript to see comments