The version of xdm that ships with Solaris 8 and 9 is incompatible with VNC 4. This incompatibility is the reason the VNC window disappears after connecting and logging in. I was able to get VNC 4.1.1 to work by downloading the XFree 4.3.0 source, building it and using the xdm that comes with it.