Clover’s Toy Box development: adding an audio mixer, expanding DDS image support, and year’s end recap.
Audio
I added support for playing multiple audio files at once. My initial goal is more-or-less feature parity with Spearmint’s audio mixer. I haven’t completed that yet.
Quake 3 has a pretty basic audio mixer; OpenAL 1.1 and SDL_mixer support most of the features directly. The main complex features would need to be implemented on top of the mixer are ambient sounds (volume affected by multiple locations but only one playing sound with one max volume) and game-specific logic for stopping sounds when starting a sound (like limit per-object).
In Spearmint, I added support for multiple listeners for splitscreen. OpenAL 1.1 and SDL_mixer do not directly support this. It should be possible using OpenAL with multiple AL contexts to have “multiple listeners” by duplicating all audio buffers and sources. It would be possible to manually spatialize sounds for SDL_mixer.
OpenAL?
I originally planned to use OpenAL 1.1 (specifically mojoAL and OpenAL-Soft). The main reason to use OpenAL was to not have to deal with multi-threading myself and (using OpenAL-Soft) up to 7.1 surround sound mixing. (Though I didn’t find OpenAL implementation for GameCube/Wii which I pretend to target.)
Using OpenAL doesn’t seem too difficult for Spearmint features (though I’m not sure ambient sounds can be implemented correctly) but I think additional features will become very complicate or unnecessary memory usage.
Using OpenAL wasn’t as simple as I expected. I still had to write the framework of an audio mixer but without the literal audio mix function. Having to deal with multiple OpenAL devices and contexts with duplicated audio buffers is complicated and unnecessary memory usage. I plan to add basic effects like volume fading … which OpenAL 1.1 does not support. These kind of push me to want to streaming the audio for each source to OpenAL. (I handle the actual full audio buffers and preprocessed effects for the source and OpenAL just does spaticalizing/mixing with short temporary per-source audio buffers.)
I had a basic incomplete OpenAL mixer working and then I ripped it apart to just add my own audio mix function which at it’s core it just adding audio samples scaled by volume.
Toy Box audio mixer for Spearmint
I modified Spearmint to support an external audio mixer library similar to external renderer library. I made a small library to implement Spearmint sound interface over Toy Box mixer. This is how I did most of the testing of the audio mixer.
This hasn’t been merged into the public Spearmint git repo as it’s not really useful outside this test and I didn’t convert the built in mixers to use it.
Toy Box audio mixer
I implemented audio sources with instance IDs so that individual sources can be modified or stopped. (The instance IDs prevent reallocated sources being affected by old out of date source handles.) There is also object IDs to allow updating position of all sounds by object like in Quake 3.
I made a list of about 50 features in the Spearmint audio mixer and reviewed the RTCW/ET features I haven’t implemented in Spearmint yet. I implemented most of them.
I implemented some long wished for features: pausing audio sources and (RTCW/ET) per-source volume and range. It would be possible to pause audio sources when the game is paused now. Though to be more practical I probably need to add some setting for whether it’s a game or menu sound.
Spearmint has four separate audio source systems for playing sounds (sfx, looping sfx / real looping sfx, music / streaming sounds, and raw samples from videos/VoIP). Toy Box has one audio source system. The somewhat strange “stop looping sound if not added this frame” is not directly supported; it’s the applications job, so it’s implemented in “Toy Box audio mixer for Spearmint” library.
Missing features compared to with Spearmint:
- Audio resampling (needed for playing sounds at correct speed/pitch)
- Doppler (dynamic audio resampling)
- Stop sound when starting sound (limit count per-object, free oldest when out of sources).
- Limit total volume of looping sounds by sfx
- Audio recording
- Audio recording volume meter
DDS Images
I added support for uncompressed DDS image formats to have feature parity with image support in Spearmint opengl1 renderer. I added a few others DDS features that are supported by ioquake3/Spearmint opengl2 renderer.
- Add support for uncompressed image formats, in additional to “ABGR8” format.
- Add support for uncompressed mipmaps.
- Add support for cube maps.
- Add support for red and red-green channel-only compressed images (BC4/BC5).
(ioquake3 opengl2 renderer supports some additional compressed formats—BC4S, BC5S, BC6H and BC7—that Spearmint does not and I haven’t added support to Toy Box.)
The DDS image format (DirectDraw Surface) is a container format used by Windows DirectX SDK that supports image data in many different formats. Toy Box had support for DDS images with DirectX texture compression 1-5 (DXT1-5, also known as Block Compression 1-3) and uncompressed 8-bit RGBA.
I added support to Toy Box for loading DDS images with uncompressed RGBA image data with various bit depths and color channels present such as RGB565, L8A8 (luminance-alpha), and RGB10A2. All are converted to RGBA8 internally; known as “ABGR8” in DDS terms as the names use reversed channel order. (Spearmint also converts them to RGBA8.)
Mipmaps are a series of 50% size copies of the image that are used to improve image quality further away from the camera.
I added support to Toy Box for using the uncompressed RGBA mipmaps from the DDS file instead of generating new mipmaps. This can be used for faster loading or effects purposely using different images for the mipmap levels. (Spearmint incorrectly generates new mipmaps but ioquake3 opengl2 renderer uses the mipmaps from uncompressed RGBA DDS images.)
Cube maps are 6 images for the sides of a cube that can be used for reflections and the sky. I added support for loading DDS cube maps and added basic cube map reflection rendering.
There are variants of Block Compression for red and red-green channel-only compression (known as Block Compression 4/5, OpenGL RGTC1/2, DDS ATI1/2, and 3Dc+/3Dc). It uses the same compression format as DXT5/BC3 alpha block but for red and green blocks. Red-green compression is useful for normal-map compression. Red-green compression offers better precision for gradients than RGB compression (DXT1/BC1) and the normal-map light direction is normalized so the third missing value can be calculated in a shader. Red and red-green compression are supported by ioquake3’s opengl2 renderer but not Spearmint (different DDS loaders).
I added support for red and red-green compression formats to Toy Box. I support uploading R/RG compressed images to OpenGL directly or first manually decompress the image if it’s not supported by the OpenGL driver. My image writing/converting code will also decompressed them if writing a image format besides DDS (such as PNG or JPG).
ioquake3’s opengl2 renderer’s DDS loader also supports BC6H and BC7 compression. I’m not planning to add them to Toy Box at this point. Just passing the data to OpenGL isn’t complicated but decompression is complicated. I want to be able to decompress all images to load them on OpenGL 1.1 or write them to a PNG image.
I’m only targeting feature parity with Spearmint opengl1 renderer but BC4/5 red/red-green compression wasn’t very difficult to add to Toy Box. I will be ignoring the other parts for now.
Year’s End
There was 406 code revisions adding 27,000 new lines of code (including comment and blank lines). It’s about 25% of Clover’s Toy Box ~106,000 lines of code (not including third party code).
Changes of the year:
- Improved rendering performance and fix various issues.
- Added support for seven more skeletal model formats; Quake 3 MD4, Doom 3 MD5, and some derivatives of them.
- Added support for OpenGL 1.1 and OpenGL ES 1.1 and to prefer OpenGL ES 2 via ANGLE on Windows over OpenGL 1.1.
- Added an audio mixer and loading six audio file formats.
- Improved game text input and added support for buttons on newer controllers.
- Added browsing archive files and viewing text files to Cover Resource Utility.
- Added basic support for running libretro cores.
- Added support for controlling PC “RGB lighting” using OpenRGB network protocol.
- Initial Android port.
- Fixed running on Windows XP and ReactOS.
Year six of development ends. Year seven begins.