Audio Quality of MP3 versus CD

This work is licensed under a Creative Commons License.

Audio Quality of MP3 versus CD

Author

There has been much debate on the sound quality of MP3's vs the 16-bit linear PCM used in producing audio CD's. Not being able to find much in the way of critical test results, I set out to perform some tests of my own. As a baseline, I chose 192Kbps as the lowest MP3 bitrate, since this seems to be a commonly agreed upon threshold for "near CD quality," and most of the MP3's I've listened to encoded below 192Kbps have sounded too degraded for my tastes.

Phase 1 - Ripping and Encoding

I chose 2 test tracks for my tests:

Evgeney Kissin's rendition of Rachmaninov's Concerto #3. I chose this track because it is an excellent DDD recording of virtuoso piano playing with an orchestra. The track contains a very large dynamic range, which helps to test for DNR compression by the MP3 process.

Mary J. Blige's "Sweet Thing" is a typical pop mix which has undergone a lot of postprocessing in the studio. The dynamic range is so compressed that when viewing the raw samples, most of the track is saturated at almost 100% amplitude level. On the other hand, the vocals are so "live" sounding that you can almost close your eyes and feel that you are in the same room w/ MJB. Also, the highly emphasized percussion with cymbals allow testing for high freq response and distortion.

Ripping/Encoding Process

All of the encoding was done on a 400MHz XL2 Winbook notebook running Windows 98. The ripping and encoding was done using Exact Audio Copy V0.9prebeta 7 via a BladeENC V92.6 DLL which I compiled myself from the source code obtained from the BladeENC site. The DLL was compiled with VC6 w/ full optimization. I used EAC to rip because I am fanatical about ensuring the integrity of the ripped data. Similarly, I used BladeENC because it is touted as being one of the best free encoders on the web in terms of audio quality at higher bitrates. Both test tracks were ripped at 192, 256, and 320Kbps bitrates and burned to a CD-R.

Freq analysis

Freq analysis Pink Noise using Cool Edit 2000 revealed different results from those obtained in a test done at Ars Technica. Perhaps the differences are either due to changes between the 0.91 version of BladeENC used by Ars Technica and the 92.6 version I used, or the fact that Ars Technica encoded the MP3's at only 128Kbps. The Ars Technica report showed a cutoff frequency of 16KHz, but as can be seen in the plots below, my own tests showed fairly similar spectra all the way past 20KHz for the PCM data and all of the tested MP3 bitrates. (BTW, Cool Edit is indeed very cool...a masterpiece of programming)

Frequency Analysis of 16-bit linear PCM 44.1KHz stereo pink noise

Frequency Analysis of 192Kbps MP3 44.1KHz stereo pink noise

Frequency Analysis of 256Kbps MP3 44.1KHz stereo pink noise

Frequency Analysis of 320Kbps MP3 44.1KHz stereo pink noise

Freq analysis of MJB's "Sweet Thing" was more interesting. The spectra were markedly different spectra between the original 16-bit linear PCM and the MP3's as well as between the MP3's themselves. Note that the spectra of the MP3's improved as the bitrate increased. At 192Kbps, the BladeENC cut off around 16KHz. Strangely, it had a small bump near 17KHz on one channel, and large bump near 18.5KHz on the other channel. With 256Kbps encoding, the frequency response extended to 18KHz on one channel, and past 19KHz on the other. Note the raggedness of the spectrum past 18KHz, though. At 320Kbps, both channels extend past 19KHz more smoothly. In all of the graphs MP3's showed characteristic gaps, caused by frequencies being discarded in the psycho-acoustic MP3 encoding process.

Frequency Analysis of original 16-bit linear PCM data ripped from CD

Frequency Analysis of 192Kbps MP3

Frequency Analysis of 256Kbps MP3

Frequency Analysis of 320Kbps MP3

Most adults, especially men, have ears which cut off at or below 16KHz. However, my own listening tests have shown that I can hear all the way to 20KHz (although only at very high SPL's...I fried a very expensive Dynaudio tweeter in the process *groan*), so I was eager to find out if I could hear the loss of frequencies above 16HKz in the 192Kbps tracks. While frequency analysis is of important academic significance, the bottom line is whether or not human ears can hear any difference.

Subjective Listening Tests

First and foremost, it is important to try to eliminate bias by the testers due to preconceived notions. Therefore, in the first phase, I performed blind testing, where the listener was not aware of whether they were listening to a CD or an MP3. I. Computer Environment

Test rig:
Machine: Winbook XL2-400 running Win98SE
Decoder: Winamp 2.02
Output: Labtec USB self-powered speakers - using the USB speakers effectively bypasses the ESS Maestro sound system of the Winbook.

The Labtecs are better than average computer speakers, but they still sound boomy and have weak high end - rather mediocre by home audio standards. Moreover, the cheap onboard amp introduces significant distortion, too. Surprisingly, even with these sub-optimal speakers, two testers in blind tests of both tracks were able to tell that the CD was slightly better sounding than 192Kbps MP3. The differences were very suble, however. There was slightly more distortion of drum cymbals in the Mary J. Blige track, and the highs sounded slightly more emphasized - surprising, since the cutoff of the MP3 was 16KHz vs the CD's 20KHz. The Rachmaninov track sounded surprisingly similar - the piano sounded almost identical to the original, but the strings seemed a little subdued. When comparing CD vs 256Kbps and 320Kbps MP3, however, one listener actually preferred the MP3 - it sounded brighter, while the other found no difference. Neither listener could tell the difference between 256Kbps and 320Kbps MP3.

II. HiFi Test

Test Rig:
Sherwood RD-7106 100x5 WRMS Theater Receiver. Don't laugh. Although Sherwood is considered one of the "armpit" audio brands those of us who know better save a lot of righteous bucks. This machine can be had for a mere $179 at Fry's Electronics, and blows away other equipment costing triple in terms of sound quality. The power amp uses discrete MOSFET's for the output and sounds very clean (the DSP effects except for Center Surround suck, but they don't apply to this review). The amp was set to stereo mode, and all EQ were set flat.

Decoder: Apex Digital AD-600A DVD/MP3 player. I got this machine when the word was out on the Internet that it could play DVD's from any region, as well as disable the Macrovision so you could dub your DVD's to VCR. What's nice is that you can set it to digital PCM output mode, so its cheap electronics are really out of the loop, and the PCM stream is sent directly to the preamp for D/A conversion. The tests were done using this PCM mode, meaning that D/A conversion was performed by the Sherwood. The MP3's were burned to a CD-R to play on the Apex.

CD Player: TEAC PD-D850 18-bit/8 times oversampling 5-disc changer. This is a modestly priced unit - probably not the greatest player, but the only one I own. It only has analog outputs, so the D/A was performed by the CD player, and analog output was fed into the Sherwood's preamp. This of course, introduced another uncertainty into the test, since different D/A's were used between the MP3 and CD playback (2007-04-05: In retrospect, I should have used the Apex DVD player to play the audio CD's. At the time, I didn't realize that it had this capability. This way, the raw PCM audio stream from the audio CD's would have gone through the exact same D/A path as the MP3's.)

Output: Lincomatic speakers, a 2-way ported home-grown system using Dynaudio Esotec 15W75-LQ-4 mid/woofs and Esotec D-260 tweeters which I designed and built myself. Although still in the tweaking stage (the crossovers aren't even tuned yet), they sound tight and clean.

Unfortunately, I was not able to find a helper, so I performed the HiFi tests myself. Admittedly, they were not blind tests, but I feel that I can be reasonably objective anyway. Given the fact that the speakers were of excellent quality, with about 50-20KHz bandwidth, it was surprising that the tests results were very similar to that using the computer with the Labtecs. When testing the Mary J. Blige track the 192Kbps MP3 again sounded slightly brighter than the CD. I was rather shocked that I could not tell at all that the MP3 cut off at 16KHz - 4KHz earlier than the CD. The percussion was slightly more distorted on the MP3 as with the Labtecs, and again, the highs sounded slightly brittle. The imaging of the MP3 was surprisingly similar to the CD, but this time the vocals sounded slightly distant on the MP3, and not quite as "live." The piano in the Rachmaninov track sounded identical between the 192Kbps MP3 and CD, but the strings were noticeably subdued on the 192Kbps, and sounded slightly less "live" - they sounded more subdued and did not have the same sparkle as on the CD. The 256Kbps and 320Kbps MP3's, however, were indistinguishable from the CD, as well as from each other. I also listened to Grover Washington, Jr's "Lock It in the Pocket" (from his Live at the Bijou album), comparing 192Kbps MP3 with the CD, and interestingly, the main difference I noticed was that Grover's tenor sax had more presence, and seemed to jump out to the front of the stage in the MP3, indicating a midrange boost in the MP3. I did not detect the muddying of the stereo imaging in the MP3 reported by some other tests that I've read; the soundstage seemed identical to me on all of the MP3's and the CD.

Conclusions

All in all, I was impressed by the quality of MP3's vs CD. Unlike the preconceived notions I had when I set out to do these tests, the 192Kbps MP3 was indeed "near CD quality." The degradation was indeed subtle, but noticeable. It is my belief, however, that on the HiFi setup the 192Kbps MP3 is more fatiguing to the ears during extended listening. Surprisingly, 256Kbps and 320Kbps MP3 were virtually indistinguishable by my ears from the CD, except for a light boost in the upper midrange and treble.

So what does this all mean? My feeling is that 192Kbps MP3 is more than adequate for listening with a computer or in a car. The distortion and lousy frequency response of the systems themselves (as well as the high ambient noise in a car) make the 192Kbps MP3 sound so similar to the original CD that the advantage of the smaller storage requirements of the MP3 far outweigh the sonic benefits of the CD. It's nice to have 5-6 audio CD's worth of music on a single MP3 CD - almost like having a CD changer in your hands. On the other hand, for critical listening on a HiFi, I feel that 256Kbps is the best bitrate to use, because it sounds significantly better than 192Kbps. Since I couldn't hear any improvement at all when going to up to 320Kbps, I don't recommend going this high for the simple reason that the file sizes are significantly larger.

MP3's ripped with BladeENC sound far better than I expected, and I believe that most listeners (except neurotic audiophiles those with extremely expensive equipment who claim that $800/meter speaker cables sound significantly better than my cheap but high quality 12 gauge stranded copper cables) would find that the difference in audio quality between the original CD's and 256Kbps MP3 is insignificant.

Update

Putting my controlled testing aside, for practical purposes, maybe it doesn't really matter if lower bitrate MP3's sound inferior to CD's. For casual listening (e.g. on PA systems, trashy built-in laptop speakers, or in noisy environments, such as in a convertible with the top down) 128Kbps is probably more than adequate. I have done some further informal tests with non-critical listeners, and they really don't notice (nor do they care about) the difference in audio quality between CD's and 128Kbps MP3's.

Sometimes, it is beneficial to put aside scientific/engineering mumbo jumbo, and instead analyze the cost/benefit ratio. Given a fixed amount of storage, one can store considerably twice as much music at 128Kbps than at 256Kbps. Before deciding what bitrate to use, analyze the target audience and listening environment, and then make a educated decision as to what bitrate is appropriate. 128Kbps has been chosen by many people because it's of sufficient quality for the vast majority of listeners. If you like to listen on a portable device with small storage, such as a 256MB flash-based MP3 player, you can fit more songs on the device if you use a lower bitrate. Anything lower than 128Kbps is unacceptable, IMHO. Personally, I will stick with 192Kbps for all but my most treasured pieces, which I will encode @ 256Kbps, but each to their own.

Update 2006-05-19

It has been a number of years since I wrote this article. In the interim, the LAME MP3 encoder has undergone many improvements, while BladeEnc has been widely discredited. Currently, I favor VBR (variable bit rate) encoding, which gives you better sound quality vs size, because the encoder dynamically adjusts the bitrate. I'm following the recommendations from HydrogenAudio (http://www.hydrogenaudio.org/forums/index.php?showtopic=28124, using LAME 3.97b2, and am very happy the results. I've found the High Quality settings (-V 2 --vbr-new) - ~190kbps - to provide sufficent quality that I find CD's burned from the output to be sonically indistinguishable from the originals. Previously, I tried an alpha version of LAME, found serious distortion during playback; stay away from alpha versions. The HydrogenAudio link above also links to RareWares (http://www.rarewares.org/mp3.html), which is a great place to obtain quality compiled binaries of LAME.

For batch transcoding, try MediaCoder. It's basically a Swiss Army Knife for transcoding audio or video from one format to another. It's great for converting among all the formats (e.g. RAM/WMA/MP3), or just for changing the bitrates in MP3's. Though I haven't used it yet for video, it can transcode video for iPod's and PSP's.

Here are some great tools for fixing tags and renaming files:

MP3 Tag Tools
MP3Tag - universal tag editor

Another indispensible tool is Mp3Gain. If you find yourself constantly adjusting your volume control when playing different MP3 tracks, you'll love this tool. It does "volume leveling" so that your songs sound about the same volume. What's great is, it's non-destructive, so you can easily undo its results if you don't like them. Unlike some other tools which normalize the actual samples, MP3Gain only adjusts the gain factor in each MP3 frame, so you don't have to worry about it introducing distortion to your files.

Additional Reading

http://www.geocities.com/altbinariessoundsmusicclassical/mp3test.html Original article in Deutsch http://www.heise.de/ct/00/06/092/
http://arstechnica.com/wankerdesk/1q00/mp3/mp3-1.html
http://www.minidisc.org/hifichoice_april2000.html

Top