Our Office plugin development team was researching some problems with Office and they found this problem.
Our SpyStudio tool intercepts application system calls and retrieves the called functions by inspecting the call-stack.
We were unable to access stack information in the last few 64-bit releases of Microsoft Office products. When we use SpyStudio to intercept an Office installation or most Office applications like Word or Excel, they start normally, but eventually exit silently.
Our initial research showed the culprit to be the Office Software Protection Platform, which would make sense, as it is supposed to hide an application’s safety mechanisms.
However, the Office Software Protection Platform did not keep us from intercepting the 32-bit version of Microsoft Office, so we decided to investigate further.
The .pdata Section
How does the 64-bit operating system do a stack walk when an exception occurs, and how can a debugger know who called a function?
Microsoft added new metadata information which is stored on a special section named “.pdata” in the PE file format specification. When an application is compiled, the compiler stores information related to the prolog of each function. If a function handles exceptions, the compiler also stores data about actions that must happen when the unwind operation is executed.
The operating system and the debugger use a series of RUNTIME_FUNCTION structures to retrieve a variety of information about each function, like how much stack space is reserved for each function usage, which callbacks must be called in an unwind operation, and where assembly registers are stored.
If you want to do your own stack walking, you can start from the current program counter (the RIP register in x64) and look for the RUNTIME_FUNCTION that belongs to it. Then process the UNWIND_INFO items to determine stack usage, and lastly, retrieve the location of the return address of the parent function.
Fortunately, there are some new APIs which makes the job easier. They are RtlLookupFunctionEntry and RtlVirtualUnwind. A stack walking code sample:
VOID StackTrace64(VOID)
{
CONTEXT Context;
KNONVOLATILE_CONTEXT_POINTERS NvContext;
UNWIND_HISTORY_TABLE UnwindHistoryTable;
PRUNTIME_FUNCTION RuntimeFunction;
PVOID HandlerData;
ULONG64 EstablisherFrame, ImageBase;
DbgPrint("StackTrace64: Executing stack trace...\n");
// First, we'll get the caller's context.
RtlCaptureContext(&Context);
// Initialize the (optional) unwind history table.
RtlZeroMemory(&UnwindHistoryTable, sizeof(UNWIND_HISTORY_TABLE));
UnwindHistoryTable.Unwind = TRUE;
// This unwind loop intentionally skips the first call frame, as it shall
// correspond to the call to StackTrace64, which we aren't interested in.
for (ULONG Frame = 0; ; Frame++)
{
// Try to look up unwind metadata for the current function.
RuntimeFunction = RtlLookupFunctionEntry(Context.Rip, &ImageBase,
&UnwindHistoryTable);
RtlZeroMemory(&NvContext, sizeof(KNONVOLATILE_CONTEXT_POINTERS));
if (!RuntimeFunction)
{
// If we don't have a RUNTIME_FUNCTION, then we've encountered
// a leaf function. Adjust the stack approprately.
Context.Rip = (ULONG64)(*(PULONG64)Context.Rsp);
Context.Rsp += 8;
}
else
{
// Otherwise, we call upon RtlVirtualUnwind to execute the unwind
// for us.
RtlVirtualUnwind(UNW_FLAG_NHANDLER, ImageBase, Context.Rip,
RuntimeFunction, &Context, &HandlerData,
&EstablisherFrame, &NvContext);
}
// If we reach an RIP of zero, this means that we've walked off the
// end of the call stack and are done.
if (!Context.Rip)
break;
// Display the context
DbgPrint("FRAME %02x: Rip=%p Rsp=%p Rbp=%p\n", Frame, Context.Rip,
Context.Rsp, Context.Rsp);
}
return;
}
Source: Programming against the x64 exception handling support, part 7
Because the .pdata section is created when the application is compiled, if the program generates dynamic code, like the .NET JIT profiler does, it should also create the corresponding RUNTIME_FUNCTION metadata and inform the operating system of the new dynamic code.
The RtlInstallFunctionTableCallback API adds an entry in an internal processes table maintained by NtDll.dll that helps RtlLookupFunctionEntry and RtlVirtualUnwind find information on how to walk the stack when your dynamically generated code is in the middle of the function calls chain.
Microsoft Office Installer and Applications
So what explains the silent exit of an Office application like Word?
At first we thought that some kind of intentional data corruption was happening in the internal table. It seemed like some kind of anti-debugging technique to keep reverse engineers from seeing the product activation mechanism.
After some trial and error we noticed that RtlInstallFunctionTableCallback was installing a callback to a suspicious routine. Surprisingly, that routine calls TerminateProcess API! When our stack walker function wanted to know the chain of calls, RtlLookupFunctionEntry indirectly called that routine and the program terminated silently.
Could the guys at Microsoft have decided to use this strange method to protect their code? The annoying callback function was added and removed frequently during many operations.
The solution was to add a simple hook to the RtlInstallFunctionTableCallback API to keep it from being added. Now the issue is resolved and the fixed versions of Deviare “Hooking for the Masses” and SpyStudio will be available soon.
Acknowledgements
The support of our Outlook plugin development team, custom development software programmers and data loss prevention development was essential for this article. Thank you!