bug report

Dave DeBarr debarr "at" mitre.org
Thu, 24 Dec 1998 16:28:59 +0000


Xvnc 3.3.2r3 crashes when used with Solaris 7 CDE 1.3 and vncviewer
3.3.2r3 (using raw encoding).  One of the CDE 1.3 clients issues the
following X11 protocol request:

    hex: 3E 01 00 07 03 80 00 3A 03 80 00 3A 03 80 \
         00 3E 00 03 00 02 00 02 00 02 00 00 00 26
                                       ^^ ^^
    ... a CopyArea request with width equal to zero.

    NOTE: CopyArea corresponds to the Xlib XCopyArea() function.

Xvnc reads the request, then calls ProcCopyArea() [dispatch.c:300] which
calls rfbSpriteCopyArea() [dispatch.c:1591] which calls rfbCopyArea()
[sprite.c:1061] which calls rfbSendFramebufferUpdate() [draw.c:723] which
calls rfbSendRectEncodingRaw() [rfbserver.c:839] which blows up at
rfbserver.c:1004 (division by zero):
    nlines = (UPDATE_BUF_SIZE - ublen) / bytesPerLine;

bytesPerLine is assigned the value zero in line 982 because w (width) is
zero:
    int bytesPerLine = w * (cl->format.bitsPerPixel / 8);


suggestions:

The CDE client should be modified.  Issuing CopyArea requests with
width zero is a waste of bandwidth.  During a _short_ session, the CDE
client sent 245 CopyArea requests with width zero.

The Xvnc server should be modified to prevent division by zero.

NOTE: This problem does not seem to occur if hextile encoding is used.

--
Dave DeBarr
The MITRE Corporation, MS W548
11493 Sunset Hills Rd; Reston, VA 20190
Voice 703-883-6544; Fax 703-883-3308



---------------------------------------------------------------------
The VNC mailing list     -   see http://www.orl.co.uk/vnc/intouch.html
---------------------------------------------------------------------