I think I've found the bug. The attached patch (already sent upstream
at http://lists.nongnu.org/archive/html/qemu-devel/2011-03/msg00270.html
and http://patchwork.ozlabs.org/patch/85359/) should fix the problem
(which occurs after 2GB of network traffic on a tight-compressed VNC
connection)
>From e51c7b1ab05d4a6fe4d153b2769290d37e077479 Mon Sep 17 00:00:00 2001
From: Roland Dreier <rol...@purestorage.com>
Date: Thu, 03 Mar 2011 16:39:35 -0800
Subject: [PATCH] vnc: tight: Fix crash after 2GB of output
If one leaves a VNC session with tight compression running for long
enough, Qemu crashes. This is because of the computation
bytes = zstream->total_out - previous_out;
in tight_compress_data, where zstream->total_out is a uLong but
previous_out is an int. As soon as zstream->total_out gets past
INT_MAX (ie 2GB), previous_out becomes negative and therefore the
result of the subtraction, bytes, becomes a huge positive number that
causes havoc for obvious reasons when passed as a length to
vnc_write().
The fix for this is simple: keep previous_out as a uLong too, which
avoids any problems with sign conversion or truncation.
Signed-off-by: Roland Dreier <rol...@purestorage.com>
---
ui/vnc-enc-tight.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
index af45edd..59ec0e3 100644
--- a/ui/vnc-enc-tight.c
+++ b/ui/vnc-enc-tight.c
@@ -829,7 +829,7 @@ static int tight_compress_data(VncState *vs, int stream_id, size_t bytes,
int level, int strategy)
{
z_streamp zstream = &vs->tight.stream[stream_id];
- int previous_out;
+ uLong previous_out;
if (bytes < VNC_TIGHT_MIN_TO_COMPRESS) {
vnc_write(vs, vs->tight.tight.buffer, vs->tight.tight.offset);
