I meet the issue on both Vista and XP. When I open several windows (maybe 10), the system begins to show problem. The context menu cannot open or any new GUI program cannot run etc.
I did a search today and found many people met this issue. A recommended fix is below:
To make this change, navigate regedit to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems. The Windows value contains a big honkin string, and one part of it is SharedSection=xxxx,yyyy,zzzz. The second number (yyyy) is the one that you want to increase. Standard registry editing disclaimers apply, YMMV.
It seems to work after I made the change.
More details, please see
http://support.microsoft.com/kb/184802
http://weblogs.asp.net/kdente/archive/2004/06/04/148145.aspx