Experiencing problems w/ 16-bit executables

[Aaron Krebs]

I am running the WinVNC server on a Windows 95 machine.  There is a
software suite I wish to be able to control, but depending on the order in
which I run the VNC viewer (from a Solaris workstation - though it occurs
no matter what platforms I view from), I get a GPF occurring in
MSVCRT.DLL.  The 16-bit application in question is not built with
Microsoft libraries (it was built with older Borland 16-bit libraries),
and should not (directly) be calling MSVCRT.DLL, which is 32-bit anyway!

If the 16-bit application is running before I try to view, I get the GPF.
If I try to run the 16-bit application AFTER starting the viewer, I
get fine behavior.

Even when I get the GPF, both the application and WinVNC appear to run
correctly.

I have run the WinVNC executable as a service or application, and have
used DebugLevel=10 in the registry to notice that (as far as WinVNC goes)
the GPF dialog pauses WinVNC execution at a different point every time.
The log files indicate no errors (so I did not bother to attach them).

I have run the MSVC debugger on WinVNC and VNCHooks and established that
the SetWindowsHookEx call in SetHook (VNCHooks/VNCHooks.cpp) triggers the
GPF, even though it returns with no error codes (using GetLastError) and
the handle it returns is fine.  WinVNC continues to run (apparently) 
successfully.

A fellow developer and I have built WinVNC with MSVC 5.0 and 6.0, using
the provided MSVCRT libraries or the latest ones available from Microsoft,
with no change in behavior.  We believe that if the 16-bit application is
running in advance, it is hooking into the message queue and causing
WinVNC's subsequent attempt to set up a hook to trigger the GPF -- even
though the call itself 'succeeds'.  If that's the case, we believe this is
just a problem internal to Win95.  A windows 98 box running the same setup
does NOT exhibit this behavior - it is just not at this time feasible for
us to deploy Windows 98.

So, can someone out there suggest a course of action that might get us
around the problem?  Or does anyone have any experience with a similar
situation, who could either confirm or contradict our conclusion?  We
don't feel we've proven the problem exists outside VNC (or the 16-bit
application, for that matter), though we suspect it.  It's hard to see a
program work fine until VNC is run and not initially suspect VNC could be
tweaked/fixed.

Thank you for any advice you might offer me!

Aaron


Aaron H. Krebs
huskers "at" laser.net


---------------------------------------------------------------------
The VNC mailing list - see http://www.uk.research.att.com/vnc/intouch.html
---------------------------------------------------------------------