VNC strong authentication, fixing the registry permissions

Andrew van der Stock ajv "at"
Thu, 23 Nov 2000 23:40:03 +0000

Hi there,

Forgive me if this is a rehash of earlier conversations, but here goes:

In response to the VNC weaknesses documented in Hacking Exposed (not a bad
book, certainly better than most BFBs), and the latest round of bugtraq
announcements, I decided to spend a little time getting to know the VNC code
base. On Tuesday, I downloaded the winvnc 3.3.3r7 source and started
hacking. On Wednesday, I read the various FAQs and went through the
contributions. I found the tridia VNC develop sites (this is a version that
I was previously unaware of).

I come from a security background (I'm a security architect by trade), and
we use VNC all over the place despite the weaknesses. We currently use ssh
tunnels, but it is not a total panacea, particularly on non-Unix platforms.
Unless you can find a stable native sshd port for Win32 (there are various
non-native cygwin-derived sshd ports, but these are all flawed as they are
not true NT services), sshd is port forwarding the VNC traffic over
potentially insecure network segments (typically true in a colocation

I've searched the archives, and I know of the NTLM and SSLeay and ssh
contribs/methods. But these are irrelevant as they are not in the current
distribution that freshmeat et al point to. The problem with security is
that it is a continual learning and improvement process. Therefore, although
these lessons have been learnt, and the improvements contributed, it amounts
to nothing because it's not in the current public distribution.

I'm too new to *VNC development processes to understand the how and the why
of why certain things are in the CVS mainline, and why somethings that I
would love to see mainlined are in contrib. I'd be happy for someone to
explain that to me privately, or give me a week or so whilst I go back
through the archives.

Here's a list of things I'd like to work on, either with people with similar
interests or if you give me enough time, by myself if no one else has my
particular itch.

Registry permission fixes

Addressing the recent bugtraq articles against WinVNC, I have finished some
changes which I am testing at home which silently check the registry key
permissions at service installation and server initialization time, and
attempt to silently set them correctly if incorrect. This is unnecessary on
Win2K (which I use), but is necessary on NT 4.0. It's irrelevant on Win9x.
If someone can supply me with the InstallShield files, I can help you make
changes there to make the installer do it properly as well. I have
InstallShield from Visual Studio 6 installed. I'd prefer to move to MSI's as
these are the installation method du jour.

Encrypting traffic

It's relatively easy to extend the current 3des helper class into a full
blown 3des-cbc encryptor/decryptor. This would satisfy security freaks like
me. Obviously in countries with silly crypto laws, this would have to be an
option that can be compiled out. But it should be possible for a server to
only listen for 3des-cbc connections. Until IPsec becomes more common (it's
on Win2K, and the *bsd's have quite a reasonable implementation
(, Linux really needs to catch up (Free S/wan has in the past
been just plain dodgy, unsure of current situation), other platforms...)
individual protocols should take at least basic precautions against snooping
and replay.

New authentication schemes

1. Using something like wide mouth frog mutual symmetric key authentication
(see Applied Crypto for more details, or here for a Java implementation:

This contains the text from AC pretty much ripped directly. The
vulnerabilities mentioned can be fixed. I'm considering making this
authentication protocol #3 in my code.

2. Kerberizing VNC

I can't find any references to this (yet). I'd be interested in working with
anyone who has already started on this or

Kerberos is a relatively secure mutual authentication scheme, with the major
benefit of being a public protocol (therefore it can work on MacOS X as
well). It's currently available on both Unix and Win2K platforms, and
therefore a relatively good target for a new authentication scheme. It also
fixes a WinVNC limitation: the single user account. Having the ability to
audit who uses the VNC service is a major boon from my perspective.

This might be best done in two stages:

* Intranet: kerberos enabling done to ensure that the Kerberos protocol is
correctly implemented and it works. This requires a second port (445) to be
open as well. I'm considering making this authentication protocol #4

* Firewall-friendly: proxying the authentication traffic between the TGS and
the VNC client via the VNC server over the open VNC port. It is assumed in
this model that the VNC server has full visibility of the TGS. I'm
considering making this authentication protocol #5. To allow a single binary
to run on most platforms, it should be weakly linked, importing the
libraries as necessary using programmatic means rather than late binding. On
platforms without shared libraries, or for the truly paranoid, statically
linking is a possibility to be looked at as well.

If you're unfamiliar with Kerberos, here's a suitable web site:

3. Fixing authentication method #2 weaknesses

I'd like to revisit the current #2 authentication, which uses a fixed string
to crypt and decrypt a 8 character string supplied by the user.

Mathematically, there's no reason to make the password run back to being
clear; it's possible to make the password run through a digest of some form
(md5 or sha1) and then store that. This will prevent one class of attack
that retrieves the password from the vnc password file or registry key and
then can trivially decrypt it. The potential change will allow an attacker
to run a brute force dictionary attack against the resulting clear text
digest, and this is about as hard as crypt() is today (ie still breakable,
but no longer sub-second unless they have a fscking big pre-calculated

The problem with this is that it will require changes in all VNC viewers if
the server cannot make the same c/r result from the stored digest. If this
cannot be made to be compatible, then I wouldn't waste any time on it and
suggest moving to either one of the above two approaches.

Group Policy awareness

One of my favorite Win2K features is group policy. It allows an enterprise
to have a view on a product and set their policy once in Active Directory.
Machines that are joined to an AD domain automatically pick up assigned
group policy. It's extraordinarily nifty. Programmatically, GPO's are easy
to hook into (about the same level of complexity as reading/writing to the
registry) and will make enterprise administrators really happy (no more
visits to individual machines, no registry hacking).

However, this may be a waste of time; Win2K Professional is the gap I'm
closing here as a free RDP client comes with every license of Win2K server
and above to allow remote administration. I can't talk about Whistler (I'm a
beta tester), but let's just say that Win2K Professional is the sole product
that Group Policy might help address from an enterprise point of view.

What do people think?


Andrew van der Stock, MCSE
Information Security Consultant
Information Security Group

(W) 61 02 9342 8488
(M) 61 0412 532 963
(F) 61 02 9342 8546
(E-mail)  - andrew.van.der.Stock "at"

For internal Information Security Advice:
CWO Information Security Group profile:
Personal profile:

TridiaVNC -
To unsubscribe, send a message with the line: unsubscribe vnc-list
to majordomo "at"
See also: