Bernhold's random tech tips: Nine different audio encoders 100-pass recompression test

Have you ever wondered how different codecs are affected by re-encoding / re-compressing? Of course, recompressing audio is a bad idea, but sometimes can't be avoided. Quality loss will inevitably occur, but are some codecs more resilient than others? To clear things up, I did a test with the following encoders:

Encoders used

WMA Professional 10 (wmapro), Expression Encoder 4
WMA 9.2 (wma), Expression Encoder 4
Musepack (mpc), mpcenc
Fraunhofer Mp3 surround encoder (mp3s), mp3sencoder
LAME (mp3), lame
Quicktime AAC (qaac), qaac
Nero AAC (nero), neroaacenc
Vorbis OGG (vorbis), oggenc2
Opus (opus), opusenc

Encoder settings

I tested all encoders with default settings, except for the quality / bitrate setting. For the main test, I used quality-based VBR (variable bit rate) encoding. I set the quantifier for quality so that I approximated two quality levels, low (~96 kbps) and high (~256 kbps). For the detailed test, I added encodings using CBR (constant bit rate) and ABR (average bitrate).

I encoded the original sample with the respective encoder, decoded it back to WAV and encoded it again, for 100 times. For decoding, I tried to use the "official" decoder where available, for example I used neroaacdec for decoding Nero, and opusdec for decoding Opus. Then I listened to the results to determine which encoder produced the best results.

Results

AAC is the clear winner by far. It is virtually unaffected by the number of passes. All other codecs had degraded sound quality increasing with the number of encoding passes, especially at low bitrates.

At low bitrates, AAC was the only codec providing satisfactory results. All other encoders fall way behind and produce audible compression artifacts such as cracking noises, muffled sound and hissing. At high bitrates, LAME and Musepack can compete with AAC, but all other encoders fall behind.

It's interesting to see how much encoders profit from an increased bitrate when recompressing many times. For AAC, as the clear winner, it didn't matter. That being said, Musepack placed 9th with low bitrate settings, but at high bitrate, it was as good as AAC and was tied for 1st place. This is similar to LAME, which produced loud cracking noises at low bitrates and placed 6th, but sounded almost perfect at high bitrates and was tied for 1st place as well.

In contrast to LAME and Musepack, the Opus encoder was nearly unaffected by the bitrate. It produced very similar results at high and low bitrates, it also didn't lose too much quality from 10th to 100th pass. In the aspect of consistency, it's only second to AAC. However, consistency doesn't mean quality here, rather mediocrity from start to finish - it had a rough and distorted sound. To be fair, the Opus codec was designed for speech, not for delivering high fidelity music. The Fraunhofer MP3 encoder behaved similarly.

The different bitrate distribution modes (CBR, ABR and VBR) produced interesting results. VBR should, in theory, always result in the best possible quality. But at low bitrates, WMA and LAME actually sounded better in CBR mode, while adding loud cracking noises in VBR mode. WMA had the weirdest behavior in this aspect. At low bitrates, CBR sounds better than VBR, but at high bitrates, VBR sounds better than CBR. In reality, this didn't matter though, as WMA produced extremely poor results on any setting.

In general, WMA and the Fraunhofer MP3s codec were the most disappointing. WMA produced loud hissing and cracking noises, while the Fraunhofer encoder sounded bland and muffling, discarding brilliance and detail. The only reason Fraunhofer placed decent is that it doesn't produce loud cracking or hissing noises, which to my ears is even worse than just muffled or dull sound. Of course, that's purely subjective.

Some encoders did not only degrade sound quality, but also had some other quirks. For example, the LAME encoder lowers the volume with every encoding pass. The 100th pass was virtually inaudible. I had to normalize the audio to hear anything at all. Other encoders produced erroneous files and garbage. The Fraunhofer encoder added silence to the beginning and end of each file and repeated parts of the sample at the end. After 100 passes, it created a 12 seconds file (the original file was 7 seconds). Winamp and Foobar2000 even reported a length of 1:02 minutes for the Fraunhofer file, however the playback ended after 12 seconds. The Vorbis encoder did a similar thing, which resulted in a reported length of 2 seconds, while the playback ended at 7 seconds. I can't really say if I did something fundamentally wrong or if it's the encoders fault, but in the end, the Fraunhofer and Vorbis encoders produced corrupted files. For the listening test, I tried to fix all errors like added silence or corrupted files as I wanted to judge the sound quality only.

For the test sample, I used the first 7 seconds of Stripper Girl from Steel Panther, as it has both guitar and vocal sounds, nicely isolated from each other. I embedded all the sound files as HTML5 audio so you can hear the results for yourself. The samples are re-encoded as OGG Vorbis at high bitrate because most browsers won't play formats such as Musepack, Opus and WMA. Because the samples are re-encoded, they are not 100% accurate, but this inaccuracy is negligible.

Results by codec (100 passes, from best to worst)

For the sake of brevity, this table only contains VBR results after 100 passes. For results after 10, 25 and 50 passes, as well as CBR and ABR results, scroll to the bottom of this page.

VBR, low quality (~96 kbps)
#	Original (WAV)
The original. For the web page player, I encoded it as OGG Vorbis ~230kbps which should be transparent.
1	AAC (Nero)
The clear winner. Very clear sound and minimal quality degradation after each pass. After 100 passes, it sounds almost the same as after 10 passes.
2	AAC (Quicktime)
Almost as good as Nero, but has some clicks added (listen closely to the beginning of the 2nd guitar chord).
3	Opus
As you will hear, it gets a lot worse from here. Opus delivers a rough sound with audible distortions.
4	MP3 (Fraunhofer Mp3s)
Lost a lot of detail, very muffled, but no hissing or cracking noises.
5	OGG Vorbis
Muffled, hissing noise
6	MP3 (LAME)
Retained some detail and is not as muffled as others, but adds a loud, sharp hissing noise
7	WMA
Very prominent hissing noise
8	WMA Professional
Like WMA, but a little worse
9	Musepack
Musepack comes in last. Sounds like a radio with no antenna.

VBR, high quality (~256 kbps)
#	Original (WAV)
The original. For the web page player, I encoded it as OGG Vorbis ~230kbps which should be transparent.
1 (tie)	AAC (Nero)
Again, very clear sound and minimal quality degradation after each pass.
1 (tie)	AAC (Quicktime)
As good as Nero
1 (tie)	MP3 (LAME)
LAME very much benefits from the added bitrate. There's no more cracking noises. It sounds as good as AAC.
1 (tie)	Musepack
As good as AAC
5	Vorbis
Retained some detail and crispness, but has a garbled, glitchy background noise
6	Opus
Rough sound with audible distortions
7	Fraunhofer MP3
The added bitrate didn't help. Still very muffled and no detail.
8	WMA
Not muffled but loud cracking noises
9	WMA Professional
Like WMA

Detailed results:

7 comments:

AnonymousApril 9, 2013 at 10:53 AM
Interesting.

However, as a real codec torture test, I'd recommend you to do the following before each pass:
* Change speed randomly (within a range of +/- 20% so it'll not cut out audible frequencies at 48 kHz)
* Change volume randomly (+/-20%)
* Add some sound of random duration at both start and end (0.05 to 0.15 seconds each) - use actual sound and no silence as codecs might just cut that out

And after decompressing the compressed data undo the above steps.
AnonymousApril 20, 2016 at 2:57 PM
Hi,
I'm the author of Opus and found your cascading results odd, so I decided to investigate a bit. As far as I can tell, the poor cascading results you're seeing aren't due to the codec itself, but rather to the resampling that opusenc performs for 44.1 kHz input. If I convert your file to 48 kHz and *then* run it 100 times through Opus encoding+decoding, I get much better results. I posted the results I'm getting for 100 cycles at 96 kb/s VBR:
https://jmvalin.ca/misc_stuff/sgirl_opus_cascade100_96k.wav
UnknownApril 25, 2016 at 10:45 AM
Hi again,
It turns out that the "problem" wasn't even the resampling itself but that the way we were doing it was causing a delay by a fractional number of samples, causing the frame alignment to slightly drift for each encoding pass. Just removing that fractional delay made Opus behave at least as good (possibly better) than AAC, even when resampling every time. That being said, the interesting test (and the common was people test) would be to do "asynchronous cascading" for all the codecs. i.e. after each encode+decode cycle, introduce a random delay in the audio. It's well known that if you have a fixed frame size (which is the case of Opus and AAC, but not Vorbis), then you get much better cascading quality if the frames always align than if the alignment is different every time.

Jean-Marc (author of Opus)
AnonymousFebruary 7, 2018 at 1:34 AM
Hey! I´m writing a written exam about audio-compression and the mp3-story, so I had enough input to be attracted by your experiment, wich is by the way very cool and almost similar to my idea, only with a different kind of test!

As I was reading your Blog I was wondering, how lossless Codecs would do...? I know, that there is no sensible rest to hear when compressing for example with FLAC. Altough I´m not shure, what it is like if you resample them a hundred times? Would be another experiment for u!;)
AnonymousMarch 10, 2018 at 2:44 AM
Answer to the person above me: lossless codecs are lossless, which means that no matter how often you re-encode them, nothing will be lost.
AnonymousDecember 13, 2021 at 5:48 AM
All the files are missing !!!

Original (WAV)
cbr, high quality
cbr, low quality
abr, high quality
abr, low quality
abr, med quality
vbr, high quality
vbr, low quality

Original (WAV)
cbr, high quality
cbr, low quality
abr, high quality
abr, low quality
vbr, high quality
vbr, low quality

Original (WAV)
cbr, high quality
cbr, low quality
vbr, high quality
vbr, low quality

Original (WAV)
cbr, high quality
cbr, low quality
vbr, high quality
vbr, low quality
abr, high quality
abr, low quality

Original (WAV)
cbr, high quality
cbr, low quality
vbr, high quality
vbr, low quality
abr, high quality
abr, low quality

Wednesday, March 20, 2013

Nine different audio encoders 100-pass recompression test