In this list of arguments against mp3s, I argued that searching for, buying, and listening to mp3s degrades the quality of the listening experience because it causes you to focus on individual songs isolated from one another, rather than on albums that cause you to focus on the larger hanging-together of the songs.
During any episode of listening to music, you might cover up to two hours, which is enough time for about 40 songs. If you are drawing those from the tens of thousands in your iPod, how can you be sure that those 40 were the best complements to each other, and how can you be sure that the order you listened to them in made this gestalt effect as great as could be? Obviously you can't. In fact, most people are just putting a huge number of songs on shuffle, or are choosing the 40 on-the-fly one after the other with no thought.
The collection of songs and the order they appear in on an album, by contrast, are the result of painstaking labor on the part of the musicians themselves and recording industry professionals. They sink or swim based on these decisions, while you don't at all, so their crafted album is going to destroy your slapdash playlist.
That's all obvious enough, but is there empirical evidence that it's true? Sure: if throwing together a bunch of good songs were better in quality than making a focused album, then compilation CDs would sell better than albums. The medium is the same, the search time is the same (in the same store), and the price is often lower for compilations! When the only real difference is the quality of the listening experience, almost without exception people prefer albums to compilations.
An iPod owner's 15-song playlist will be even worse than a compilation CD, though, because the issuers of the latter at least have some incentive to think about which songs to include and in what order. Generally they won't work well with each other because they weren't composed with the others in mind, drawn as they are from so many different musicians and teams of recording professionals and zeitgeists. But still, they've thought about it more than you have. If compilation CDs don't sell well, your playlist burned onto CD would sell even worse. The objection that pre-made compilation CDs don't sell well because they can't be customized by the individual as a playlist can is also bogus. Recall that no one actually customizes their playlist, in the sense that people used to spend hours thinking about which songs to include on a mix tape and in what order. Generating a playlist, by contrast, is a thoughtless process, not one of fine-tuning the list to your unique tastes.
I think everyone realizes this because no iPod owner, aside from a handful of indie dorks, is arrogant enough to see themselves as a playlist magician -- like, "Hey, I'm so good at this, I should turn it into a hobby or make some money off it!"
Greatest hits CDs by a single artist do a lot better, of course, for the simple reason that the songs come from a common source and will naturally mesh together better than if they came from all over the place. I'm not making the hardcore fanboy argument that greatest hits albums are for housewives and little girls -- which they are -- but simply that the original albums sound better. Although the greatest hits come from a common body, they don't necessarily come from the same mind. Artists change over time, and including songs from too-different periods can clash -- even worse if the artist gets worse over time, and in listening to the greatest hits you feel more depressed from watching their energy fade away. Madonna's multi-platinum greatest hits album, The Immaculate Collection, is like that -- it all goes to hell starting with "Express Yourself," and it's a real downer having to halt a great stream of music in the middle of the CD.
So for mp3 slaves who insist on a lower-quality listening experience, a good rule-of-thumb would seem to be restricting the songs you're drawing from to come from a single artist. That's still not the same as a greatest hits CD -- what songs to include and in what order are still left up to someone who has little incentive to get it right -- but it's a lot better than putting the entire archive on shuffle or cobbling it together as you go.