■ Prevent unwanted DLLs invading your process with a simple detour

xplorer² is a shell integrated file manager, meaning that it loads all sorts of 3rd party shell extensions when it runs. Mostly this is to everybody's advantage, but there are some problematic extensions that don't work well with it — causing xplorer² to crash or hang. Some are badly written, or tested with windows explorer only. Whatever the reason, some programs are best kept at a distance. When faced with such dilemmas, people have to choose which program to keep, xplorer² or the misbehaving shell extension?

The other day it occurred to me that there's a win-win alternative. If I could stop a particular shell extension from loading in xplorer², users would have stability and get to use the other program independently. Such a mechanism could block out unwanted global hooks and other invaders too. But is it possible?

All external code that loads in windows processes comes in a Dynamic Link Library (DLL) and there's a single API used to attach these called LoadLibrary. If we could somehow intercept calls to LoadLibrary and check against a blacklist of DLL names, and only allow the good ones through, that would be swell. In fact COM object (shell extension) creator CoCreateInstance uses LoadLibraryEx instead of the plain variety, but the idea is the same: detour just one API and solve your problem — we like things minimal down here.

The most popular API hooking library is probably MS Detours. Having never used it I can't tell how easy it is, but I found this other one called MinHook that is certainly small and easy and works for both 32/64 bits. It comes precompiled in a tiny 10kb DLL which you can dynamically load without source code integration. Hooking is very straightforward:

#include "MinHook.h"
// Pointer for calling original function (trampoline)
HMODULE (WINAPI* g_fpLoadLibraryEx)(LPCTSTR, HANDLE, DWORD) = 0;

// our intercepted API
HMODULE WINAPI Detour_LoadLibrary_EX(LPCTSTR lpFileName, HANDLE reserved, DWORD dwFlags)
{
	// check if DLL name is allowed
	if(IsDllBlacklisted(lpFileName) )
		return 0; // block it
	
	// else allow normal loading passing it to system API
	return g_fpLoadLibraryEx(lpFileName, reserved, dwFlags);
}

// code snippet to install hook 
MH_STATUS hr = MH_Initialize();

// replace system LoadLibraryExW with our version
hr = MH_CreateHookApi(L"Kernel32", "LoadLibraryExW", Detour_LoadLibrary_EX, (void**)&g_fpLoadLibraryEx);

hr = MH_EnableHook(MH_ALL_HOOKS);					, 
ATLASSERT(MH_OK==hr); // add your own error checking :)

MH_CreateHookApi is the essense, it uses its black magic to reroute all calls to LoadLibraryExW into our own Detour_LoadLibrary_EX, which checks the DLL name against our blacklist (easy to implement as a string array) and either blocks it or lets it load normally — using the original system function pointer. That's it! with a few lines of code any shell extension we don't like can get the boot, whether context menu, text filter or what have you. I tested on windows 10, 7 and XP.

With the main objective in the bag, I tried intercepting a global hook as well (those installed by SetWindowsHookEx like Spy++ SPYXXHK.DLL). The above detouring was good with global hooks for older windows versions, but on win10 it just wouldn't intercept. WTF? LoadLibraryExW is exported from Kernel32.DLL, the module name being used in MH_CreateHookApi (see above code). Could global hooks use another API or mechanism to attach themselves in foreign processes? After a few hours wasted googling, I had the inspiration to write a simple hook DLL and test it with the debugger, placing a breakpoint in DllMain, and the mystery was solved.

If you see the stack trace to the right, a different LoadLibraryExW is used to map global hooks, one that resides in KernelBase.DLL. Why are there 2 different functions for the same task? I don't know, they just are (and they are trully different, it's not a simple redirect). However they do seem to have the same input arguments, so you can easly detour the one in KernelBase by adding a second hook to LoadLibraryExW. And for completeness I added a hook for LoadLibraryW, all checking against the blacklist, 3 APIs detoured altogether to catch everything that may try to load in your process.

I plan to add this DLL blocking functionality in xplorer² v6.0, coming to you this spring sometime. More on this in a forthcoming blog post.