Fixing a seven-year-old QEMU VNC bug that corrupted virtio-gpu consoles

Every so often we'd get a report of corrupted video in a LabConsole class. LabConsole runs live, instructor-led IT training in the browser: each student works on real lab machines through a web console, and the instructor watches every student's screen at once from one grid. Now and then one of those consoles showed thin slivers of old terminal text after a fast scroll, left behind on a screen that should have been clean. Nobody could make it happen on purpose. Closing and reopening the console always cleared it, which made our own VNC proxy the obvious suspect.

It wasn't the proxy. The slivers come from a race in QEMU's VNC server that has shipped in every QEMU release since 4.0, in April 2019. I sent a fix to the qemu-devel list on October 28, 2026, with results from a reproducer that triggers the bug on demand: unpatched QEMU master lost an update in 35 of 150 test cycles with four clients connected, and the patched build lost none.

The bug affects VNC clients of any QEMU guest whose display device replaces its surface often. virtio-gpu guests running a double-buffering compositor, such as GNOME on Wayland, replace it on every frame.

Where the pixels go

QEMU's VNC server tracks what each client still needs with a dirty bitmap, one bit for each 16-pixel run of a scanline. When a client asks for an incremental update, vnc_update_client() gathers the dirty rectangles into a job for the VNC worker thread, clears those bits, and resets the client's pending-dirty count. The worker thread then encodes the job and sends it.

The VNC server encodes from its own copy of the screen, the server surface. vnc_refresh_server_surface() runs on a timer, copies any chunk where the guest's surface differs from that copy, and marks the chunk dirty for every client. A chunk where the two already match is left alone.

A page flip breaks that bookkeeping. With virtio-gpu, each flip is a SET_SCANOUT to a different resource, which replaces the display surface and calls vnc_dpy_switch(). That calls vnc_abort_display_jobs(), which throws away any job the worker has not started:

%%{init: {'theme':'base','fontFamily':'Open Sans,sans-serif','themeVariables':{'fontFamily':'Open Sans,sans-serif','fontSize':'16px','lineColor':'#00609A','edgeLabelBackground':'#FFFFFF'},'flowchart':{'htmlLabels':true,'useMaxWidth':true,'nodeSpacing':60,'rankSpacing':50,'curve':'basis'}}}%%
flowchart TD
    A["<b>1.</b> vnc_update_client() queues a job<br>and clears the client's dirty bits"] --> B["<b>2.</b> The guest flips: SET_SCANOUT<br>replaces the display surface"]
    B --> C["<b>3.</b> vnc_abort_display_jobs()<br>discards the queued job"]
    C --> D["<b>4.</b> The next refresh finds the guest surface<br>matching the server surface"]
    D --> E["<b>5.</b> The client keeps the old pixels"]
    classDef infra fill:#E6F0FA,stroke:#0078C1,stroke-width:2px,color:#003C60
    classDef accent fill:#FBE3B3,stroke:#ED8C0C,stroke-width:2.5px,color:#3A2A08
    classDef deny fill:#FBE2E2,stroke:#C0392B,stroke-width:2px,color:#7A1A1A
    class A,B,D infra
    class C accent
    class E deny
    linkStyle default stroke:#00609A,color:#003C60

The lost update. Step 3, in amber, discards a job whose dirty bits step 1 had already cleared.

After step 3 the dropped rectangles have no dirty bits, and the server surface already holds their new pixels, so step 4 finds nothing to send there. The client keeps the old pixels until those chunks change again. Reopening the console clears them because a new connection asks for a non-incremental update, which sends the whole screen.

The same abort path has a second bug. If the client sent another update request while the job was queued, the dropped job's job_update state is never cleared, and vnc_should_update() refuses every later update for that client, including forced ones. The client's display freezes until it reconnects.

How long it has been there

The worker thread and vnc_abort_display_jobs() arrived with the threaded VNC server in 2010, and the abort was harmless while a surface switch re-dirtied every client. Three later changes set up the two bugs:

• ui: fix VNC client throttling when forced update is requested (December 2017, QEMU 2.12) added the job_update state. An aborted job left it set, and the client stalled.
• vnc: detect and optimize pageflips (January 2019, QEMU 4.0) stopped re-dirtying clients when a switch keeps the same size and format. That made a page flip cheap, and it is where the lost pixels start.
• vnc: fix update stalls (March 2019, QEMU 4.0) cleared job_update after an abort, but only when no newer request was pending, which left the freeze in place for clients that request often.

The lost pixels have been in QEMU for seven and a half years, and the freeze in some form for eight and a half.

The fix

The patch retires a dropped job whether or not a newer request is pending, keeps a pending forced update, and marks the whole screen dirty for that client (the code comment is trimmed here):

     QTAILQ_FOREACH(vs, &vd->clients, next) {
         vnc_lock_output(vs);
-        if (vs->update == VNC_STATE_UPDATE_NONE &&
-            vs->job_update != VNC_STATE_UPDATE_NONE) {
-            /* job aborted before completion */
-            vs->update = vs->job_update;
+        if (vs->job_update != VNC_STATE_UPDATE_NONE) {
+            if (vs->update != VNC_STATE_UPDATE_FORCE) {
+                vs->update = vs->job_update;
+            }
             vs->job_update = VNC_STATE_UPDATE_NONE;
+            vnc_set_area_dirty(vs->dirty, vd, 0, 0,
+                               vnc_width(vd), vnc_height(vd));
+            vs->has_dirty++;
         }
         vs->abort = false;

A job that finished normally has already had job_update cleared by vnc_jobs_join(), so the branch runs only for a job that was discarded. The cost is one full-screen update each time that happens, which is what every surface switch cost every client before 2019. The patch, ui/vnc: recover a job dropped by vnc_abort_display_jobs(), is under review on qemu-devel.

Reproducing it

On a real desktop the corruption is intermittent, so I wrote a reproducer that triggers it on demand. It builds QEMU master twice, with and without the patch, inside a Fedora container, and boots a tiny guest whose init is a static C program. On each cycle the guest draws a screen of text-like noise into a DRM dumb buffer, then switches to black and flips between two black buffers at 60 Hz, the way a compositor does. The picture stays black, but every flip replaces the display surface.

A Python harness connects VNC clients that behave like noVNC: Raw encoding, with one incremental update request always outstanding. When the guest goes quiet, the harness compares each client's framebuffer with QEMU's own screendump, taken over QMP. Here is one client's framebuffer from an unpatched run, captured while screendump showed a solid black screen:

The client missed the switch to black entirely. One non-incremental request repaired it.

Results on QEMU master 73294d0 (x86_64, KVM):

The failures follow queue position. Client 0 lost no updates, and clients 1, 2 and 3 lost 18, 25 and 35. The worker thread starts the first client's job at once while the others wait behind it, and a flip in that gap discards them. One non-incremental request healed every lost update.

The single-client case did not reproduce on this idle test host. Our consoles are a single VNC client per VM, because the LabConsole proxy fans one QEMU connection out to every viewer, and they do show the symptom on busy KubeVirt nodes. With one client, the window is the worker thread's wake-up latency, and host load stretches it.

Covering the gap in LabConsole

A QEMU fix reaches our KubeVirt VMs only when it ships in a virt-launcher image, so the LabConsole VNC proxy works around the bug in the meantime. When a console's screen stops changing for one second after a burst of updates or keyboard input, the proxy asks QEMU for one non-incremental update and passes it to every viewer, at most once every five seconds. A sliver left by fast scrolling now clears about a second after the scrolling stops. The workaround lives in one module of the proxy, and it comes out once the fix reaches our KubeVirt nodes.