Update: 3 Feb. 2008
As of iTunes 7.6, it appears that Smart Playlists now treat "greater than" and "less than" correctly (e.g. "Rating is greater than 2" now does include songs rated 2.5). So a playlist defined by rating "greater than 2" and "less than 3" now (in most cases) selects just 2.5 rated songs, as it should. That is what I hoped for all along, so unless your ratings are more fine-grained than half-stars, this built-in approach will be more reliable than my Smart Playlists below. (If you use an automated song rating script or otherwise use more precise ratings, my half-star range lists below may still be useful, assuming that Apple's implementation really now is mathematically correct.)
After you install the iTunes 7.6 update, the first time you run iTunes it will correct all of your existing Smart Playlist definitions to have the right behavior. Ironically, that means that if you already had my Smart Playlists installed, they will be broken by the update (just as if you had viewed their properties and clicked "Ok", I think). In any case, it's hard not to suspect that my investigation of this issue contributed to Apple fixing it in the next update, so I'm happy.
Executive summary
When using iTunes's hidden ability to show half-star ratings, normal Smart Playlists invariably round down (e.g., the rule "Rating is greater than 2" fails to match a song rated 2.5). Using an external editor, I have created definitions for Smart Playlists that select only those items with fractional star ratings. The XML files linked below under Smart Playlists for Half-Stars can be imported into iTunes to give you that capacity. I have also come up with an explanation for how the "Play higher rated songs more often" feature of Party Shuffle probably works (and why it ignores fractional star ratings); I discuss it under Weighting Ratings in Party Shuffle.
Open question: is there any way to convince Party Shuffle not to select any tracks that are currently shown in the Party Shuffle window? Please let me know! (It would actually be sufficient to just avoid picking the currently playing song, perhaps by removing it from the source playlist, but I don't know how to do that either.)
Introductory remarks
When I'm in the moood for music, I often listen to a random selection of songs using Party Shuffle mode in iTunes. That means that my listening choices revolve around playlists rather than albums, usually Smart Playlists based in part on each song's rating. By cleverly constructing my Smart Playlist criteria, I've can get iTunes to do pretty much what I want it to.
But at some point (version 6.02?), I learned that iTunes had quietly started to display half-star rating increments (which I assign on my Mac using the Add-Subtract A Half Star AppleScript; I've heard that there is also a way to assign half-stars on Windows). I started to use them for the handful of songs that really were right on the borderline, but my Smart Playlists didn't behave the way they should have.
Although the iTunes user interface will happily display half-stars, its rating features almost invariably ignore the half increments completely (the fractions are always rounded down). I couldn't create a Smart Playlist that would distinguish between a 3 and a 3.5: a rule like "Rating is greater than 3" has mathematically incorrect behavior, since it does not include songs rated 3.5. The "Play higher rated songs more often" in Party Shuffle also doesn't behave quite the way you'd expect, though that's harder to notice (this probably goes for the "Choose higher rated items more often" option when filling my iPod Shuffle, too). So I've put this page together to share a bit of what I've figured out with the world at large.
Internally, iTunes has apparently always stored more than just the number of stars (from 0-5). Instead, the actual rating of a song is recorded as a score from 0 to 100. One, two, three, four, or five stars correspond to scores of 20, 40, 60, 80, or 100. From what I've read, anything with a score from (say) 60 to 79 will show up as three stars; the one new exception is that a score of exactly 70 shows up as three and a half.
Smart playlists for half-stars
Happily, it is possible to create a Smart Playlist that correctly selects half-star ratings. I'll explain how in a moment (it's not easy, and you can't do it from within iTunes itself), but the punchline is the following set of playlist files. If you import them into iTunes (using the "File>Import..." menu item), they will automatically select only those items whose ratings fall strictly in the ranges specified. (Note that as of iTunes 7.6, the lists in the first column are no longer necessary.)
Be careful! If you ever view info on these playlists within iTunes and click "Ok" (rather than "Cancel"), they will revert to the default behavior for "equals 0", "equals 1", and so forth: you will have to re-import the playlist (or revert to a backup of your iTunes Library). That's especially painful when you've used these lists as building blocks for other Smart Playlists (which is the whole point).
So how does it work? I'm glad you asked. I originally thought that iTunes simply wasn't capable of distinguishing half stars in Smart Playlists (nobody online seemed to have done it). But I noticed that two of my early Smart Playlists with the rule "rating is greater than two stars" actually did include songs rated 2.5. However, a new Smart Playlist with completely identical rules did leave them out as usual. That proved that the limitation wasn't built in, but rather that it was just being enforced by the current user interface (and that something in that interface must have changed since those two special lists were created). So I exported the original playlist and its new copy to an XML file and opened them in a plain text editor to compare them head to head.
Most of a Smart Playlist's exported XML file just lists the tracks that it contained at the time; I deleted those bits. Apart from some identifiers and version information, what remained were two block of encoded data, the "Smart Info" and the "Smart Criteria". The encoded data shows up as line after line of numbers and letters, most of them just capital "A"s. This is actually a base64 encoding of internal hexadecimal data: A=0, B=1, C=2, D=3, ..., Z=25, a=26, b=27, ..., z=51, 0=52, 1=53, ..., 9=61, +=62, and /=63. (There seem to be other special codes for any trailing "incomplete" bytes that don't end cleanly on a base64 boundary.) It might be interesting to puzzle out the whole system, but I just worked out enough to make sense of the ratings rules. In particular, I quickly saw that the "Smart Info" data wasn't relevant here (it contains things like the "Match only checked songs" setting, but apparently not the detailed rules).
So "Smart Criteria" is the key. The data section here gets longer the more rules the playlist has. In a bare bones playlist with just one "Rating is..." rule and nothing else, this means just under five lines in the XML file. (You can follow along at home using the files linked above; some browsers even have a nice mode for displaying XML.) Midway through the third line is a "Bk", followed (usually) by "AAA" and then a code for what the actual rule is. "BkAAAAQ" means "greater than", "BkAAABA" means "less than", and "BkAAAEA" means "in the range" (which iTunes displays as "is equal to" if both ends of the range round down to the same number of stars). For the record, "BkCAAEA" apparently codes for "not in this range"; that's how iTunes encodes the "rating is not" rule. Note that in base64, AQ=16, BA=64, and EA=256 (I'm guessing that the base64 encoding is misaligned from a byte boundary here: in native code these are probably 1, 4, and 16).
The endpoints of the range are encoded later; in the XML files linked above, the lower endpoint appears late in the fourth line and the upper endpoint appears early in the fifth. Essentially, all you need to do is figure out the lower and upper ends of the range you want on the 0-100 score scale and translate it to base64. The only issue is that for these particular files, the encoding again doesn't line up with a byte boundary: the encoded numbers in the files above are actually four times the desired values. (So in the "2<x<3" file, Ck=164=4*41 is the code used for "just above 2 stars" and Ds=236=4*59 is the code used for "just below 3 stars".) Some of this is easier to see in the Apple "Property List Editor" program (which displays data in hexadecimal blocks rather than base64), but it's much harder to compare two data strings in its limited user interface. Copying the hex data from there into a text editor might be the best of both worlds.
If the mode isn't "in the range" but rather "greater" or "less", the current iTunes version just sets both endpoints to the same value. Internally, both modes really are strict inequalities (e.g. ">" rather than ">="), so "less than 3 stars" corresponds to the code for a score of 60. But "greater than three stars" uses a score of 79: it explicitly excludes ratings between 61 and 79, not just 60 (or perhaps 69) as one might expect. I was intrigued to see that the old playlists that actually did already include 2.5 star songs in "rating is greater than 2" used the code for 40 as the lower bound (the upper bound is apparently ignored in "greater than" mode, as it was set to 100 without excluding five star songs). My guess is that when iTunes was modified to be able to display half stars, they also changed how "greater than" ranges were encoded; my two "special" playlists must have been created before that change.
One could in principle create a whole range of Smart Playlists this way with any desired set of range criteria (I was able to track this part of the data down even in a playlist with an extra rule or two), but it's a pain and it's fragile (in that you can't ever edit these playlists within iTunes, and it's easy to accidentally break them just by viewing the criteria even if you don't think you made a change). So I've opted to just create the five "fractions only" playlists above and never modify them from within iTunes: I use them in the rules for my other playlists. (I have a "Strictly Rated 4" playlist with the rules "Rating is 4 stars" and "Playlist is not 'Strict 4<x<5'", for example. Other examples are more interesting but more complicated.) Unless you want to dig into the internals, you can just use these same five lists as well: I've already done the hard part.
Weighting Ratings in Party Shuffle
A number of people online have done experiments to figure out just what's going on when Party Shuffle is told to "Play higher weighted songs more often". One of the best is the article Extended iTunes Rating by Matthew Schinckel on the OmniNerd site: that article gives graphs of play count vs. rating for the full 0-100 range of scores and a table of the numerical play counts for all of the possible half-star and full star ratings.
I won't try to summarize all of that data here; it's a nice, clear article if you're interested in the details. But I will add a little insight into what I think is actually going on. (Hmm... letting someone else do all the experiments and then swooping in to explain the results? I really am a theoretical physicist.) As far as I can tell, iTunes is picking tracks from a sort of virtual playlist that contains:
- 1 copy of each song with rating R=0 stars,
- 2 copies of each song with rating 0<R<1,
- 3 copies of each song with rating 1<=R<2,
- 4 copies of each song with rating 2<=R<3,
- 5 copies of each song with rating 3<=R<4,
- 6 copies of each song with rating 4<=R<5,
- 7 copies of each song with rating R=5.
This is surprisingly easy to verify from the data in the final graph (or table) in that OmniNerd article: just round off each play count to the nearest thousand and you'll see that each data point is 2000 times the number of copies mentioned above. Maintaining this simple pattern may be why Apple hasn't incorporated fractional ratings into the system (though I might actually prefer the "even more often" effect of a 1-11 system that included all of the half-star increments instead of the current 1-7 system). It's probably no coincidence that these are essentially the same ranges used in Smart Playlists (perhaps Apple doesn't want to change one without changing the other).
Up to Steuard's personal page.
Up to The World of Steuard Jensen.
Any questions or comments? Write to me: steuard@slimy.com
Copyright © 2008 by Steuard Jensen.