Reverse engineering closed source binaries is always a challenge. Using tools such as Process Monitor, API Monitor, IDA and others make such tasks possible to achieve, but still requires good knowledge and experience. Whether you want to exploit or defend (depends on the hat you wear) a specific application, you need to find the right interception point(s). In most cases we are dealing with dlls, which is obviously by getting the dll’s export functions addresses and start investigating from there.
Lately I had to find the right place for interception in order to develop an additional security layer to one of Microsoft’s IIS components. No need to mention that the set of the dlls I’ve investigated is completely undocumented and unfortunately does not even have private symbols in Microsoft’s symbols server. So my guideline was –
- Find a few interesting entry points (exported functions) in some dlls
- Debug the target process by attaching the process using Visual Studio
- Load relevant dll exports (once again, we have no symbols)
- Set breakpoints on my suggested entry points
After eliminating some of the loaded in the process, my suspicious was on one dll with more than 6,000 exported functions. Cutting things short, I’ve found a function, let’s call it foo.dll:func() which can point that I’m in a good spot in the critical path. Setting a breakpoint on this function proved me that I was in the right place. Whenever I performed the operation I wanted to intercept, the operation was hung until the breakpoint was released. Done? Nope. Since my goal was to protect and not just audit an operation, I tested what will happen if I’ll skip the execution of this function or just return access denied. Doing so did not provide the results I was looking for. Although I skipped execution, original operation triggered by the user was still completed successfully. This probably means that this was not the function I was looking for, though I’m very close. Why? Because the breakpoint really held the execution of the whole request, so I’m in a good spot, probably on the right thread as well.
Taking one more try with the same breakpoint has showed something interesting. When the execution has paused because of the breakpoint, I was taking a look at the thread’s call stack and noticed that this function was not in the first frame of the call stack. To simplify it (2 frames scope only), it was looking like this:
boo.dll:newFunc()+0×500 → foo.dll:func()
So, we have a new dll in the game, boo.dll. It seems like the real operation was initiated from the newFunc exported function in boo.dll, and along the way, in offset 0×500, it has reached a ‘call’ opcode to foo.dll:func().
Why? Because of three reasons –
- Setting a breakpoint at boo.dll:newFunc() entry point did not break the execution
- The original name of newFunc() was not related at all to the operation I was looking for
- Each time I performed different operations, I’ve noticed boo.dll:newFunc() in the call stack with different offsets for each operation.
So although at first glance it may look like calls originated from boo.dll:newFunc(), they actually weren’t. Then what is going on here? COM objects were in the game.
Since this dll does not have symbols, Visual Studio only chance to show a more informative call stack, is to load dll exports, so it will look like a minimal symbols version of the dll. However, there are more functions in this dll, so whenever there is a function offset that Visual Studio does not recognize for this dll, it basically look up for the first known symbol and present it like execution originated from this symbol plus the offset for the real address. Meaning, it looks like execution was originated from function X (which we do have symbol for) while execution was originated from function Y (which we don’t have symbol for).
So, apparently boo.dll is not just exporting functions, it also has a couple of dozens COM interfaces. So how do we proceed from here?
We have 3 main challenges –
- Discovering all dll’s COM interfaces, methods and addresses (remember, GetProcAddress() will not do the trick here).
- Making your debugger (VisualStudio in my case), to show the correct call stack, including COM function calls and also to have the ability to set breakpoints at COM functions easily. A good call stack will be bool.dll:ComFunc+offset → foo.dll:func() (since boo.dll:newFunc exported function was just misleading, we don’t want to see it in the call stack).
- Understanding an application API calls flow. If you reverse engineer a set of dlls that include more than a few dozens of APIs, it becomes a challenge to find the right place for interception. We will need logging of all API calls for a specific operation. This is a bit similar to API Monitor and Process Monitor but what both are missing is fully automated logging of COM calls without the need of manually preconfigured definitions file (and also integrating it for Visual Studio).
How can we achieve this? Stay tuned for the DbgGenerator tool I’ve created which allows you to load symbols for such dlls and easily reverse engineer them with no time. I promise good stuff.