Xvnc Crash on Solaris

jlnance@intrex.net jlnance "at" intrex.net
Fri Jun 28 14:12:01 2002


Hello All,
    I have a reproducable crash that occurs in Xvnc 3.3.3r2 when run on
Solaris.  It does not happen if I run Xvnc under Linux.  Its fairly easy
to reproduce:

1) Start up vnc with the Xvnc process running under Solaris
2) run mozilla using the Xvnc process as its display
3) go to www.redhat.com

I did a little investigating, and the crash is occuring in free() which
usually means that something else corrupted memory.  I built Xvnc from
source and purified it.  There were a large number of purify warnings,
but I think the ones responsible for this problem are some array overwrites
that occur rigth before the crash.  I have included the relivant portions
of the purify log below.

Thanks,

Jim

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893 at Thu Jun 27 14:20:27 2002)
  * Purify 2002.05.00 Solaris 2 (32-bit) Copyright (C) 1992-2001 Rational Software Corp.  All rights reserved.  
  * For contact information type: "purify -help"
  * For TTY output, use the option "-windows=no"
  * Command-line: /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc :2 \
    -desktop X -httpd /home/jnance/arch/Sun/vnc/classes -auth \
    /home/jnance/.Xauthority -geometry 1152x864 -depth 24 -rfbwait 120000 \
    -rfbauth /home/jnance/.vnc/passwd -rfbport 5902 -ac -nolisten local 
  * Options settings: -chain-length=32 -purify -cache-dir=/tmp/jcache \
    -always-use-cache-dir \
    -purify-home=/tools/rational/releases/purify.sol.2002.05.00 
  * License successfully checked out.
  * Command-line: /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc :2 \
    -desktop X -httpd /home/jnance/arch/Sun/vnc/classes -auth \
    /home/jnance/.Xauthority -geometry 1152x864 -depth 24 -rfbwait 120000 \
    -rfbauth /home/jnance/.vnc/passwd -rfbport 5902 -ac -nolisten local 

[snip]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9091 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff308 in the heap.
  * Address 0x9ff308 is 1 byte past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	malloc         [rtlib.o]
	Xalloc         [libos.a]
	AllocatePixmap [libdix.a]
	cfb32CreatePixmap [libcfb.a]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9091 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff30c in the heap.
  * Address 0x9ff30c is 5 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	malloc         [rtlib.o]
	Xalloc         [libos.a]
	AllocatePixmap [libdix.a]
	cfb32CreatePixmap [libcfb.a]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9091 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff310 in the heap.
  * Address 0x9ff310 is 9 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	Xalloc         [libos.a]
	AllocatePixmap [libdix.a]
	cfb32CreatePixmap [libcfb.a]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9091 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff314 in the heap.
  * Address 0x9ff314 is 13 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	AllocatePixmap [libdix.a]
	cfb32CreatePixmap [libcfb.a]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9090 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff318 in the heap.
  * Address 0x9ff318 is 17 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	cfb32CreatePixmap [libcfb.a]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9090 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff31c in the heap.
  * Address 0x9ff31c is 21 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9090 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff320 in the heap.
  * Address 0x9ff320 is 25 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (9090 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0x9ff324 in the heap.
  * Address 0x9ff324 is 29 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	main           [libdix.a]
	_start         [crt1.o]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (44 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0xa006dc in the heap.
  * Address 0xa006dc is 5077 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (44 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0xa006e0 in the heap.
  * Address 0xa006e0 is 5081 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (44 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0xa006e4 in the heap.
  * Address 0xa006e4 is 5085 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (44 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0xa006e8 in the heap.
  * Address 0xa006e8 is 5089 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (44 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0xa006ec in the heap.
  * Address 0xa006ec is 5093 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
ABW: Array bounds write (44 times):
  * This is occurring while in:
	cfb32FillRectTile32Copy [libcfb.a]
	cfb32PolyFillRect [libcfb.a]
	ProcPolyFillRectangle [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Writing 4 bytes to 0xa006f0 in the heap.
  * Address 0xa006f0 is 5097 bytes past end of a malloc'd block at 0x9e6ef8 of 99344 bytes.
  * This block was allocated from:
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]
	*unknown func* [pc=0xcc]

****  Purify instrumented /home/jnance/src/vnc_unixsrc/Xvnc/programs/Xserver/Xvnc (pid 17893)  ****
COR: Fatal core dump:
  * This is occurring while in:
	t_splay        [malloc.c]
	t_delete       [malloc.c]
	realfree       [malloc.c]
	cleanfree      [malloc.c]
	_malloc_unlocked [malloc.c]
	MaLlOc         [libc.so.1]
	malloc         [rtlib.o]
	Xalloc         [libos.a]
	AddResource    [libdix.a]
	ProcCreatePixmap [libdix.a]
	Dispatch       [libdix.a]
	main           [libdix.a]
	_start         [crt1.o]
  * Received signal 11 (SIGSEGV - Segmentation Fault)
  * Faulting address = 0xdc
  * Signal mask: (SIGSEGV) 
  * Pending signals: