Wed, 31 Jul 2013

MinGW and Postmortem Debugging

Crash early and often

Usual Introductory Blah

About a year ago I wanted to dump Microsoft Visual C++ in favour of MinGW, since I’d have really liked to use ‘modern’ C++11 features.

However, the promises of out-of-band compiler updates with better C++11 support and the fact that it is easy to create minidumps via dbghelp.dll’s functionality and debug them with windbg were enough to delay this move.

Until now.

With the announcement of MSVC 2013 and its - still - lackluster support for C++11 I gave in. However losing the debugging support for crashdumps really hurts. But every search on the web only turned up other unsuccessful inquiries about a working post-mortem debug facility with MinGW.

The Problem

The only sane way to create crash-dumps under Windows is to use dbghelp.dll’s Minidump functionality. This minidump can then be loaded with windbg (or Visual Studio). However, both applications need a PDB file to show useful information (unless assembly and memory addresses instead of function names is sufficiently informative for you). A file, gcc does not create, since it stores its debug information in DWARF format. A format neither of those programs can handle.

So how about the other side? gdb can of course handle DWARF files. But it can not load minidump files.


The Solution

Luckily(?) the ecosystem of the D programming language faced a similar problem. So that is where the solution can be found: cv2pdb, which can not only convert CodeView files to PDB, but also the DWARFen ones!

Create the Debug Info

  1. Compile/link your project with debug information (gcc -g)
  2. Use cv2pdb to convert the DWARF debug info into PDB format (cv2pdb -C program.exe)
  3. Strip the executable from any debug info (strip -g program.exe)
  4. Distribute the executable and keep the PDB file

The PDB file and the executable cannot be matched automatically based on their checksum, since the strip changed it. So you need an out-of-band way to match the correct files.

The way I do it: on an actual crash I create the minidump and pack it together with the logfile, which contains the application’s version number, into a zip or tar file before sending it off. That way I can match the minidump to the correct PDB file.

Actual Debugging

Since the executable was stripped of any link to the PDB file windbg tries to load a generic image00400000.pdb file (or image00000000`00400000.pdb for 64bit executables). So make sure that is the PDB’s file name. I for one just archive the PDB files named this way and put them in a directory for each version.

  • The directory containing this file needs to be added to windbg’s symbol search path. (-y <path>)
  • Further more, since the PDB file and executable cannot be matched .symopt needs to be changed, to include 0x40 (SYMOPT_LOAD_ANYTHING). Combined with the default symopts this is 0x80030377. (-sflags 0x80030377)
  • And finally the actual dump file needs to be specified with the -z argument.
  • While I’m at it: I also add -g to skip the initial breakpoint.

All in all I end up with a command line like this to start windbg to do post-mortem debugging with a minidump:

windbg -sflags 0x80030377 -g -y <directory_containing_correct_pdb_file> -z <path_to_dumpfile> <path_of_correct_executable>

Enable Javascript to see comments