[xplorer˛] — Advanced shell context menu
home » blog » 8 July 2007 [programming]


"Elementary, Watson!" — Sherlock Holmes

Every now and then programming computers can be really exciting. You have to solve a lot of mysteries, a bit like playing Tomb Raider. Microsoft supposedly wants developers to write applications to enrich its platform, but the supplied documentation leaves much to be admired. And unlike your Lara Croft-related problems, there's no online walkthrough to get you unstuck. Ok there's google and Krugle, but for the really hard problems there's no help. The only pages you find on the internet are cries for help by other poor developers without a MSDN subscription. You are on your own. It is a challenge.

If you have been following the beta builds of xplorer˛ you'll know that I figured out how to do the shell context menu for item selections that live in multiple folders, like those in standard search results explorer windows. This has been a long-standing shortcoming of xplorer˛ scrap windows. In this article I retrace the steps that lead to the uncovering of this not-so-elementary mystery.

The multi-folder shell context menu riddle

Getting the plain shell context menu is easy and well documented. All you need is a folder, you pack the pidls representing the selected items and off you go. But if you try the same trick for items that reside in different folders, it won't work. The "reasonable" approach would be to use the root namespace (desktop) folder and pass it a list of fully qualified pidls (like full paths so to speak), but it won't get you anywhere. You will get a menu, but most of the important commands are missing, e.g. Add to zip.

A good first hint was offered by Timo Kunze, a regular platformsdk.shell group contributor (and German translator of xplorer˛ GUI). He pointed out a discussion on CDefFolderMenu_Create2, an obscure API published as part of the microsoft 2001 settlement with the US law. It is very complex and unintuitive and the documentation is a big joke.

Some of its arguments are reasonable, but there are some weird things too, like a list of registry keys (what for? I passed you the selection, filename extensions and all, didn't I?) and a callback function with tons of notifications — all of them with wrongly documented return codes. I spent a day or so googling for CDefFolderMenu_Create2 and I think I've read all the discussions about it, and made some progress, but in the end at best I could just reproduce the half-complete context menu that the desktop folder would supply at a fraction of the hassle through GetUIObjectOf. Bummer dude!

It is clear that this API (and its equivalent SHCreateDefaultContextMenu for windows Vista) was meant for shell namespace extensions (NSEs), i.e. a single folder, so desktop was still stumbling on the fully qualified pidls I was passing. All the people that claimed to have figured it out were talking from the NSE perspective, which wasn't what I was after. Adding to my frustration, the menu I was getting would work properly some of the time, when the selected items happened to lie in various folders under the filesystem equivalent of the desktop folder (e.g. E:\Users\nikos\Desktop). But throw in something from another partition and all the interesting menu commands were gone, WTF?!

Back to the usenet discussions. People were talking about the intricate relationship between the context menu and the dataobject that represented the selection. Could that be the hint? But surely the folder implementation I use written by microsoft shell team's capable hands (through SHGetDesktopFolder) would not fudge its own dataobject? Or would it?

Enter the final piece of the detective work. I downloaded a simple context menu extension sample for text files, built the DLL and used xplorer˛ as its executable for debugging. I added plenty trace statements to monitor which methods were being called and peppered it with breakpoints. For plain context menus everything would be fine, the extra command would show nicely whenever a text file was right-clicked.

So why didn't it show up for multifolder selections? Because its initialization method IShellExtInit::Initialize was checking the supplied dataobject for CF_HDROP in order to read the filenames, and there was none to be seen. Bingo. Thank you desktop folder!

That was the eureka! moment, I had the diagnosis of the problem. How about the cure? Surely I wouldn't start writing a replacement IShellFolder implementation from scratch, reinventing the wheel just for a little errant dataobject? Of course I wouldn't, this is xplorer˛ and we do things in the simplest possible way!

Subclassing COM interfaces

Every programmer knows about window subclassing, where you extend the basic functionality of a standard windows component in a few small areas, leaving the bulk implementation in place. But can you apply the same technique on an IShellFolder? Sure, why not?

The basic idea is to write a class defining a fake COM object of the interface type you are "subclassing". Most of the method calls will be routed to the real object, except for those you want to override. A bit like DefWndProc. The only tedius bit is that you have to provide skeleton implementation for all the interface methods. Observe this listing:

class CIShellFolderSpy : public IShellFolder
{
public:
   CIShellFolderSpy(LPSHELLFOLDER sf) {
      sf->AddRef();
      m_iSF = sf;
   }

   ~CIShellFolderSpy() { m_iSF->Release(); }

// IUnknown methods --------
   STDMETHOD_(ULONG, AddRef)() {
      return m_iSF->AddRef(); // pass it to "base"
   }

   // ... likewise for all other methods

// IShellFolder methods ----
   virtual HRESULT STDMETHODCALLTYPE GetUIObjectOf( // this method we "override"
      /* [in] */ HWND hwndOwner,
      /* [in] */ UINT cidl,
      /* [size_is][in] */ LPCITEMIDLIST *apidl,
      /* [in] */ REFIID riid,
      /* [unique][out][in] */ UINT *rgfReserved,
      /* [iid_is][out] */ void **ppv)
   {
      if(InlineIsEqualGUID(riid, IID_IDataObject))
         // i ignore the pidl array supplied and return the good data object
         return m_pMDObj->QueryInterface(riid, ppv);
      else {
         // i've seen IDropTarget requests, let base handle them
         return m_iSF->GetUIObjectOf(hwndOwner, cidl, apidl, riid, rgfReserved, ppv);
      }
   }

protected:
   LPSHELLFOLDER m_iSF; // the real underwriter
   CMultiDataObject* m_pMDObj; // that's mine with proper HDROP
};

Once you have a valid folder object e.g. via SHGetDesktopFolder, you instantiate the fake object and pass the real COM pointer in the constructor. If you want to get really fancy, you can have the class dynamically allocated and synchronize its lifetime with the COM object, i.e. when the reference counter reaches zero, then delete this; However for this example it suffices to have a single global object.

CDefFolderMenu_Create2 walkthrough

I'll cut to the chase and present all the necessary steps. I leave the exact code required as an exercise but you should have no trouble reproducing it, since I am mentioning all the possible pitfalls. The starting point is an array of global PIDLs (desktop-based), that you wanted to pass to GetUIObjectOf, representing the items whose context menu is to be shown.

  1. Create a data object on the selection. Don't use shell's GetUIObjectOf since it's not going to have the essential HDROP. Spin your own IDataObject instead. This is easier said than done but if you have an explorer type of application you most probably have such a class already.
     
  2. Load CDefFolderMenu_Create2. It's exported from shell32.dll, ordinal #701. In later windows it is exported by name too.
     
  3. Open registry keys for the file types in the selection. This HKEY array is the argument ahkeyClsKeys and is required; without it CDefFolderMenu_Create2 won't do much. All file types come from HKCR hive. For example if your selection is full of text files, you'll need the registry keys:
    HKCR\*
    HKCR\.txt
    HKCR\txtfile
    Note there's a bug in this API so don't pass more than 16 registry keys!
     
  4. Supply a valid callback. Don't pay attention to the documentation, it's rubbish. Here are the messages you have to respond to (you don't have to do anything except for returning the correct code, unless you want to add your own menu items):
    DFM_MERGECONTEXTMENU: S_OK
    DFM_INVOKECOMMAND: S_FALSE
    DFM_(everything else): E_NOTIMPL
     
  5. Initialize IShellFolder wrapper. This is the CIShellFolderSpy class mentioned above. It hosts the desktop IShellFolder and also knows about the dataobject we created in step 1. The only method you need to override is GetUIObjectOf to return the "good" dataobject.
     
  6. Pass all arguments to CDefFolderMenu_Create2. Most importantly you have to supply the "wrapper" folder for psf. You should get back an IContextMenu object that you can use in the usual fashion.

We are not quite done yet. The ultimate goal is to execute the menu command the user has selected. The context menu object we received can handle most commands but some commands you have to deal with yourself, like "properties". One way to do this is inside the callback (DFM_INVOKECOMMAND handler). You can try returning S_FALSE and see if you can get an implementation for free, otherwise you'll have to do it yourself. But as a proud programmer of a file manager I'm sure you have already code to show properties etc.

I have tested this code with all windows back to w98 and it works fine. I am not sure if it will work for NT4/95 but you can try!

I can say without doubt that solving this riddle gave me the most pleasure in all my programming career. A mixture of detective work, guesswork and some hacking. Who needs playstation when you have to reverse engineer windows API? :)

Post a comment on this topic

AddThis Social Bookmark Button

 

 

What would you like to do next?

Reclaim control of your files!
  • browse
  • preview
  • manage
  • locate
  • organize
Download xplorer2 free trial
"This powerhouse file manager beats the pants off Microsoft's built-in utility..."

download.com
© 2002—2007 Nikos Bozinis, all rights reserved