VNC and OpenSSL (long)

[Joe Ammann]

[ On Tuesday, January 4, 2000 at 09:59:58 (-0600), Habermann, David (DA) wrote: ]
> Subject: RE: VNC and OpenSSL (long)
>
 > My WinVNC.log
 > file does indicate that an SSL connection is being made (although I have not
 > independently confirmed this). 

You should be able to see the currently selected encryption method on
the vncviewer's connection info dialog.

 > I was amazed at how easy you made this for
 > us.  Just FYI: I did see one new additional link warning that I don't
 > believe I saw before:
 > 
 > Linking...
 > LINK : warning LNK4098: defaultlib "LIBCMT" conflicts with use of other
 > libs; use /NODEFAULTLIB:library

Yes, I know. This has something to do with the OpenSSL DLL being
linked with some libraries, that conflict with the default libraries
for VNC. I don't know (yet) how to circumvent this cleanly. So if some
Windows compiler specialist has some ideas, I'd be happy to learn.

 > I like your ideas regarding the PAM-infrastructure authentication system.
 > I've been wanting to authenticate against NT for quite some time, but my
 > options have been limited.

Yeah, I'd really like to discuss this further. Of course, this would
mean an extension to the RFB protocol. Anyway, I'm not happy with the
current implementation because client and server have to be configured
correctly to work. If a client with SSL enabled connects to a server
without SSL, the client hangs indefinitely. I'd rather have them
negotiate the desired protocol.

This should actually be encoded in the ProtocolVersion
handshake. E.g. if we introduce a new protocol version "RFB
003.004\n", we could then add another round of negotiation just for
"connection options". Those could be SSL, zlib, other types of
compression, etc.

A similar scheme could then be applied for authentication options. In
the Authentication negotiation, we could also add new types of
authentication. E.g. for NT domain authentication, we could just
transfer a username/password pair in cleartext (over a previously
secured connection) etc.etc.

But these types of changes have to be sanctioned by the people from
AT&T, otherwise it will be a nightmare for me to keep those
current. (comments?)

 > I'm not sure why the server-initiated connection would require a client-side
 > certificate (probably just my ignorance) since I believe all the server does
 > is open the IP connection.  The client actually still sends the first text
 > in the negotiation process, so it would still be responsible for asking for
 > the SSL connection and negotiating ciphers, etc.

Stupid me! Of course this should work! It should be easy to do just
the socket connect the other way round (server connects to client) but
keep the SSL handshake in the same order (server does an SSL_accept,
client does the connect). I will try this tomorrow.

Regards, Joe

---------------------------------------------------------------------
The VNC mailing list - see http://www.uk.research.att.com/vnc/intouch.html
---------------------------------------------------------------------