Corrupted screen with VNC on Linux for S/390

[Patterson, Ross]

Ross Patterson <Ross.Patterson "at" CA.Com> writes:
> Rob van der Heij <rvdheij "at" iae.nl> writes:
> >I am trying to get VNC working on Linux for S/390, and after 
> >browsing the mailing list archives I still need some help. 
> 
>                            The echoed characters are 
> gibberish, but commands typed into xterm do what they should 
> (e.g. "write" sends intelligble messages to other users, 
> "exit" closes the window, etc.).

I know replying to one's own mail is gauche (and probably disgusting in some Wired JargonWatch list), but I've found what seems to be the major cause of trouble.  IBM System/390 boxes are big-endian, and of course Intel x86'es are little-endian.  There are at least three cases in the VNC server source where the attempt to be variously-endian fails.  

In vnc_unixsrc/Xvnc/programs/Xserver/hw/vnc, rfb.h defines macros for mixed-endian byteswapping (Swap16IfLE() and Swap32IfLE()) but hard-codes the flag that determines the server's endianness to "1" (meaning little-endian).  I think there should be some conditional definition, or at least it should be called out in the README etc.

In the same directory, init.c defines the PF_EQ() macro assuming that byteswapping isn't necessary for 8-bit pixels, but in fact seems to be (although this doesn't make any sense to me).  Running vncserver with "-depth 32" produces a non-garbled display (xterm contains readable characters, etc.).  Either it's an endian-ness issue or there's some side effect of the translation functions in translate.c that this configuration requires and that rfbTranslateNone() (which is used with -depth 8) doesn't supply.

Lastly, in the same directory, translate.c defines the BGR233 pixel format with a constant 0 for rfbPixelFormat.bigEndian (meaning little-endian).  If the evidence above is correct that there is actually some endian effect for 8-bit pixels, this needs to be conditional as well.

VNC isn't working perfectly on System/390 yet, but now it's basically usable.  There are still some pixel artifacts to clean up, and that may be the last of it.

Once all is well, would portability patches be welcome?

Ross Patterson
Computer Associates Int'l, Inc.
---------------------------------------------------------------------
To unsubscribe, send a message with the line: unsubscribe vnc-list
to majordomo "at" uk.research.att.com
See also: http://www.uk.research.att.com/vnc/intouch.html
---------------------------------------------------------------------