Clover’s Toy Box development: Fumbling around with transparency on Wayland.

Wayland is a newer application display/input protocol on Linux and some other operating systems. I recently got an AMD graphics card that works with Wayland and switched from using GNOME with X11 to Wayland.

Clover’s Toy Box and “Toy Box renderer for Spearmint” work on GNOME with Wayland using SDL 2 library.

However my Qt app Clover Resource Utility’s 3D model background was transparent and failed to clear the model between drawn frames which created a hall of mirrors effect.

It took 8 hours to understand and fix the Resource Utility issue yesterday. Though I did take a break at one point and was distracted by watching YouTube the entire time.

Fix it

What do I know:

• I had previously read that OpenGL on Wayland (and also unrelated, Vulkan in general without enabling an extension) display alpha instead of displaying the window as opaque.
• The Clover Resource Utility background has been broken for awhile—displaying as black—unrelated to Wayland.
• The 3D scene is rendered to a framebuffer object and then drawn to the widget framebuffer (either blended or copied directly).

So combining these things, it seemed I broke the background to be transparent black. I got the background to work fairly easily by changing the 3D scene to be “transparent” which caused it to blend with background instead of overwriting the background alpha with 0 alpha of the 3D scene. This fixed the background color as well. This was the older behavior of the application.

Fix it again

The 3D scene was specifically not transparent/blended because of issues with Quake 3 map materials writing 0 alpha and expecting it to be opaque.

I previously added a hack to make non-blended materials write 1.0 alpha (only necessary when a transparent/blended framebuffer object was used). However this detection failed in simple cases like blending a texture over a lightmap. It basically only fixed textures without a Quake 3 shader.

Testing this older behavior however had very strange effect that I couldn’t guess causation and behaved differently on X11 and Wayland. On Wayland the transparent materials were accumulatively added to the window each frame with hall of mirrors effect until the surface was white. However I clear the 3D scene and widget framebuffers each frame. It kind of seemed like premultiplied alpha without clearing the widget framebuffer but that doesn’t make any sense either. Where is it coming from?

I tried to fix it by forcing writing 1.0 alpha when drawing the 3D scene framebuffer to the widget framebuffer. However I changed the color multiplier in the vertex shader, not the actual alpha of the 3D scene framebuffer texture in the fragment shader, and I didn’t realize it for hours.

Fix it again, correctly

This left me with an issue I could not explain the cause of and searching online lead to nothing (despite finding and reading about other issues related to Qt Wayland not displaying OpenGL content, general OpenGL transparency, etc).

Qt problem? GNOME with Wayland problem?

I was comparing using both of the Qt X11 and Wayland backends and testing changing various things, semi-random because it made no sense. I occasionally thought I fixed it but then realizing I need to test maps besides q3dm1 because my “non-blended shader hack” fixes it but I didn’t manged to fix the alpha. There was a lot of truly strange results.

Eventually I tried enabling alpha for the Qt OpenGL context. (I had code for it commented out and I was just semi-randomly testing things.) This fixed the weird “accumulatively added to the window each frame” and made X11 and Wayland behave the same.

After that I was able to understand, I did not correctly fix it to write 1.0 alpha when drawing the 3D scene framebuffer texture to the widget framebuffer. Correcting that to actually work, fixed the issue.

Conclusion

Make sure to write 1.0 alpha to the OpenGL framebuffer on Wayland if OpenGL alpha buffer isn’t enabled.

That’s how I spent 8 hours to fix the transparent HOM background while still correctly ignoring Quake 3 map materials writing weird alpha.

I thought about trying to use RenderDoc to inspect it at one point and I don’t really have a reason for why I didn’t.

Maybe if I took a break for the day, I would of realized that in order for non-1.0 alpha to misbehave when I’m clearing the 3D scene and widget framebuffers each frame, non-1.0 alpha must be making to the final framebuffer.

If having OpenGL alpha issues on Wayland (or maybe in general), enabling alpha in the OpenGL context so it’s displayed (correctly) may be helpful.

I tried to capture the Quake 3 map bug here by recording a video: surface is opaque, screenshot: surface is transparent. I love undefined behavior.

Future

The titlebar and resizing on GNOME with Wayland depends on the application to implement it. SDL supports it using the libdecor library. (libdecor will be added in Flatpak freedesktop 23.08 SDK releasing around the end of the month.) I would want to ship libdecor along with SDL in my application for full self-contain Wayland support.

(Outside of Flatpak) I would prefer not to deal with shipping libdecor and it’s plugins that assume that cairo or GTK 3 are installed. It seems like it would be simpler to add support for handling the titlebar and resizing in my application.

However SDL seems to need improvement for “hit testing” for determining if the cursor is over the titlebar/edge to work correctly. (In my test, resize doesn’t work on Windows, resize cursors don’t appear when hovering over edge on Linux (X11/Wayland), and I haven’t tested macOS.)

SDL 3 has gained support for popup menu and tooltip windows and displaying the default titlebar context menu across Windows, Linux, and macOS. I would like to build on it to make a simple GUI framework to replace using Qt for Clover Resource Utility.