Everything you ever wanted to ask about DBGHELP.DLL, IMAGEHLP.DLL and SFP

Plus a little Python thrown in.

Introduction

This document will give you some background information how you can retrieve C/C++ debug information in Win32 systems (that is: Windows NT/2000/XP/2003, but not 16-bit systems like 98 and ME). In the course of doing so, I will present to you a minimal debugger, written in python, that has some benefits over using "fullblown" debugging systems - for example, it is faster and you can script it to watch specific function calls. Neat, right?

Note: Python is used here for writing a debugger for C/C++ or, more generally, any binary Windows application. We will not discuss debugging Python programms - we will discuss writting a Debugger in python. That is different.

What are those DLLs?

Which DLL is the most important? There is no answer to this question, because all of these are important for a debugger.

Where is the problem?

These DLLs - or rather, some of these DLLs - have been distributed as part of the operating system since (at least) Windows NT 4.0. Some Microsoft applications (such as Developer Studio) have also shipped these. Of course, every OS and every app has a different version - does the term "DLL hell" ring a bell? Basically, there are two problems:

  1. Some machines do not have all of those DLLs
  2. Many systems have old or mismatching versions of those DLLs, which results, at best, in mayhem.

Which version should I use?

Microsoft brings out new versions of DBGHELP.DLL as part of the freely available Debugging Tools for Windows. If you are a C/C++ developer, you should install these tools on your machine, for the following reasons:

  1. They include the latest DBGHELP.DLL which, trust me, you want.
  2. They include the great Kernel Debugger, which you can use to debug a remote machine. Its fun!
  3. They include the symbol server / remote symbol handler, which I will discuss further down below.

Since releasing "Windows 2000", Microsoft has packaged the DBGHELP.DLL (as well as the other two) with the operating system, and protected them using SFP (System File Protection). This, bascially, sucks. It means that you cannot replace the DLL - even though the version is completely outdated, broken, and missing a lot of features. The version that you really want is the version that is part of the Debugging Tools for Windows.

The problem is not so big regarding IMAGEHLP.DLL and PSAPI.DLL - these versions are fairly OK, but missing on NT4 systems.

How do I install a new DBGHELP.DLL?

These are your options:

  1. Copy your binary to the DBGHELP.DLL directory
  2. Copy the DBGHELP.DLL to your applications' directory
  3. Rename DBGHELP.DLL to something else, but unique, such as TPKHELP.DLL. Next, patch your binary to link to TPKHELP.DLL rather than DBGHELP.DLL.
  4. Boot 2000/XP in Safe Mode. For reasons only Microsoft knows, "safe mode" means disabled "system file protection". Go ahead and figure it out. In safe mode, you can replace the DLL in system32 (and, necessarily, in dllcache, see below).
  5. Find all DBGHELP.DLL files (there are likely to be several, in system32, in dllcache and in ServicePackFiles. Kill em all. You'll get a Windows warning to insert the original CD to restore these. Cancel this. Copy your new version. You'll get a Windows warning to insert the original CD to restore these. Cancel this. Curse SFP.

If you think that all of these options are more or less crap, then - welcome to the club!

Setting up a symbol server

Problem: Windows has no debug symbols. If you installed the ones you can find on your Install CDs (MSDN version), you will very soon be out of luck: because every time you visit WindowsUpdate to fix another "remote arbitrary code execution" bug, you get new versions and they don't match the symbols in your system.

Solution: Well, Microsoft has had a pretty good idea, actually: installing a public symbol server. You can connect to that machine (provided you are on the network) and get the correct dbg information for all components on your system (on demand), cached locally in a directory of your choice. Nice!

To install, create a system wide environment variable:

_NT_SYMBOL_PATH=srv*c:\cache*http://msdl.microsoft.com/download/symbols;

Logoff / Logon, and you should be fine. Note that c:\cache will be the cache directory, you can change that name.

Note: For this to work you need the symsrv.dll, which is part of the Debugging Tools for Windows. I have mentioned this URL now three times, and I'm pretty sure that by now you have them installed on your system, right? Well, add the directory to your path and you're fine.

Using Python to debug stuff

I have written a Python23 extension DLL pydebug.pyd, that you can use to write debuggerlike applications in Python. You can use it to dump symbol tables, find symbols in memory etc. I have also written a mini Debugger that you can use as a starting point, procman.py. Before you can see what procman.py can do for you, you need to install it. Duh.

Installing procman.py

  1. You need Python 2.3 and the Win32 extensions installed.
  2. You need to install the a refreshed DBGHELP.DLL as described above. The easiest way of doing so is downloading the Debugging Tools for Windows and using sfcrestore.py (see code above!).
  3. Download procman.zip. It is freeware and includes the full source.
  4. You might want to setup the MS symbol server as described above.
  5. Copy procman.pyd to your \python23\dlls directory.

To test it: try this:

python procman.py notepad

After you have closed Notepad, open pyman32.log and enjoy.

What can procman.py do for you?

First off, it is (considerably) faster than a normal debugger. So, if you have something that is a timing issue, you can try it with procman.py.

Second, you can use it to set breakpoints at function calls to DLLs. E.g. to Kernel32!LoadLibraryA. Check the source, you'll find these lines:

self.patches = [
    PatchThis( "KERNEL32!LoadLibraryA", 
        args = [("LPSTR","lpszLibrary")],
        result = "WINAPI" )
]

As you can see, this is a list so feel free to roll your own.

Third, its free, and by toying around with the source you can learn stuff about DBGHELP.DLL.

Enjoy!