CommVerge - Home

This page will serve as a central jumping-off point for the entire project. Here you'll find links to the first article and any future updates, pointers to auxiliary information like tables and graphics, downloadable files that come into play in the project, resources that I've found useful during my quest, and more. This'll be a "living" document; updated regularly as the project proceeds. So check back occasionally, and drop me an email with any feedback. If we get some good discussion going, we'll even post some of it here. Thanks!

* You will need the free Adobe Acrobat Reader to view the PDF files. If you don't already have it, you can download it here.

Other resources

I've come across a number of other audio compression benchmarking studies on the Internet, most of which are too superficial, too unscientific and/or too biased to be useful, IMHO. Regarding bias, one common trick used by some very well known codec manufacturers involves encoding a monophonic version of an audio track using their software, while using a stereo (thereby more challenging) version of the track for their competition. Buyer beware! I won't bother linking to the studies that exist on each codec developer's site, and anyway I encourage you to give greater emphasis on independently-published studies.

With that warning out of the way, I've found the information in the studies below to be of particular value. Keep in mind as you read through them that some are rather old, using encoder and/or decoder products and versions which are now vastly improved. Some are quantitative, attempting to use numerical analysis to compare alternatives in terms of their frequency response, phase preservation, required system memory and performance, etc, while others rely on the more qualitative 'how does it sound' A-vs-B comparison technique.

• Robin Whittle's lossless and lossy compression studies are quite good, include sample sound files and plenty of links, and were one of the primary motivations for my own work.

• Arny Krueger's PCABX site contains detailed before-and-after oscilloscope displays showing the audio alterations caused by various codecs, as well as a number of test tone files.

• By now, I'd hoped that the MP3 decoder-to-decoder differences found in Robin Whittle's 1998 study had resolved themselves. According to David Robinson's voluminous and comprehensive PhD study, however, this unfortunately doesn't yet seem to be the case.

• Some of you might know David Weekly as the guy whose site, at one point in the early days of MP3, was consuming 80 percent of Stanford University's outgoing bandwidth. David did some of the earliest evaluation work on Windows Media Audio, comparing it to both MP3 and RealAudio. Keep in mind when reading his report, as David himself mentions and Microsoft confirms, that later WMA revisions resolved many of the issues he found.

• In their September 1, 1999 issue, PC Magazine conducted a listener test of a number of lossy codecs.

• NSTL also conducted an MP3 vs RealAudio vs WMA listener study in 1999, with some very interesting results (note, this links to an Adobe Acrobat file).

• Mark McHarg benchmarked the Shorten, WaveZip, and Wavpack lossless compression algorithms, and you'll find an archive of his subsequent DAT-heads mailing list posting here (or you can search the DAT-heads archives for this and other interesting digital-audio data).

• Other sites presenting results of lossy compression comparisons include those by David Bradbury, L. Z. Hanson, John Hayward-Warburton, Will Ryu, and Panos Stokas.

Test clips

Copyright and licensing restrictions preclude me (obviously) from posting the uncompressed or compressed versions of the music files that I used in my testing, as well as the files I obtained from the PCABX website. However, below you'll find the test tones that I've thus far generated myself. Warning: These files are quite large—5.04 Mbytes bytes each).

Also, give a listen to inphase.wav (431 KB) and oophase.wav (431 KB). Although both contain 1-kHz tones, the two channels in oophase.wav are 180 degrees out of phase with each other, while both channels in inphase.wav are identical. You should notice quite a bit of "spaciousness" in oophase.wav, compared to inphase.wav. It's this channel separation that may get lost when a lossy encoder eliminates phase differences between two channels at equivalent frequencies.

Here's some other sources of useful audio files, both comprising synthetic test patterns and real music and voice, that you might find useful in your own analyses.

• PC Magazine study

• EBU Sound Quality Assessment Material (SQAM) samples

• Make sure you read through the PCABX license agreement before downloading any of the files.

• The Signal Processing Information Base

Excellent sources of dual-channel test signals include the multi-disc Hollywood Edge Test and Measurement Series, InterStudio's Sound Check 2 disc, and (particularly for low bass) the International Auto Sound Challenge Association max dB disc. For multi-channel test tones, turn to Dolby's DVD Demo and Test Disc and DTS's Music Demonstration and Set-Up Disc. And, if you'd like to better understand the different types of noise in order to create your own test clips, turn to Don Morgan's article as well as information on Robin Whittle's site.