Why don't reviews and star-ratings sync-up?

  • 8
  • Question
  • Updated 4 years ago
  • Answered
My only problem with AllMusic has never been one of functionality (it has always been, IMO, very user-friendly), but rather with consistency between reviews and star-ratings. It's all too common to see a review with a phrase like "not only the band's greatest album, but one of the greatest albums of the era," accompanied by a 3-star rating. Subsequently, the following album will often be described as "an extreme let-down" and "signs that the band was beginning to lose focus and run out of steam," accompanied by a four-star rating. What gives here? All I can think is that the reviews are written by individuals, while the star-ratings are assigned by committee. But regardless of the reason(s), it makes for some pretty glaring inconsistencies. Any way you guys could fix that?
Photo of Mark Weisinger

Mark Weisinger

  • 31 Posts
  • 7 Reply Likes
  • frustrated

Posted 8 years ago

  • 8
Photo of Sean Alexander Corbin

Sean Alexander Corbin

  • 0 Posts
  • 0 Reply Likes
Thanks for the responses. Unlike some on this thread, I think you guys are remarkably consistent considering the scope of work Allmusic does compared to the vast majority of music reviewers. Thus why I initially found the OCMS review odd. Keep up the good work Allmusic!!
Photo of Marty Barrows

Marty Barrows

  • 1 Post
  • 0 Reply Likes
This reply was created from a merged topic originally titled
Grading/review discrepancies.


I have used your site for years and do find it (usually) helpful though there are missing artists and albums occasionally, but I have one big question for you....why is it I see some artists/albums have a 4 out of 5 star rating yet when I read the review, it's ripped to shreds? 4 out of 5 stars would indicate a high rating (and I'm talking about your rating, not User Ratings)...perfect example: I was just looking at Candy Dulfer's Saxuality album and AllMusic says 4 out of 5 stars but the review says (direct quote): "...a forgettable R&B/pop/jazz recording that's smothered by excessive production". That doesn't sound like a 4 star record to me. Why the conflicting grading/review?
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
Hi Marty,

Thanks for pointing this one out -- I'll have Matt Collar, who covers a lot of jazz for us, check it out.

Hard to say how this happened exactly -- it's possible we may have had a different review in place 10-15 years ago.

My guess, though, is the problem we often have with genres like smooth jazz that can be hard to love. If you like smooth jazz, this is a four-star album, but for general connoisseurs and even jazz critics like Alex Henderson, it's a difficult listen.

Anyway, thanks again for bringing it up -- we'll get Matt on it right away.

Thanks,
John Bush
AMG Senior Managing Editor, Pop Music
Photo of Marty Barrows

Marty Barrows

  • 1 Post
  • 0 Reply Likes
I'm going to add a little more fuel to the fire. Right now, I'm in the process of going through my personal collection of 30K+ CD titles in order to downsize. During this process, I'm using Allmusic to expedite listening to various songs (saves time form having to load and unload discs constantly as well as offering me some quick info on any artists I may have forgotten details about). One of the artists I just looked at is Fandango (the Nick Simper band, not the one with Joe Lynn Turner). They released two albums and both have 2 star ratings on Allmusic. Both titles were combined a few years back on a twofer release and Allmusic's rating is 4 stars (and the review speaks more favorably of them than the original single album reviews did). Shouldn't something like that be corrected...say when the new review was done, shouldn't someone have done something about the older reviews/ratings so as to not cause general confusion?
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
Sorry, this is another one I didn't see when it was originally posted. This is a big discrepancy and I'll have an editor take a look at it. (Maybe 2 + 2 = 4? Sorry, bad joke.)

Seriously, though, bring on the fuel! I appreciate the comments that we're remarkably consistent, and I also appreciate the avalanche of constructive feedback on this page that shows the few places where we haven't been consistent.
Photo of Andy DeNardi

Andy DeNardi

  • 231 Posts
  • 20 Reply Likes
How many reviewers do you have on staff? Do you have freelancers to handle specific genres? I can easily imagine how hard it is to find people who enjoy and have a good understanding of certain types of music. I, for one, would like to see a reviewer with excellent credentials in ethnic and non-western music. Most "world" albums don't carry much detail or go unreviewed. I'm not volunteering, but have you tried reaching out to some of the more esoteric blogs?

For what it's worth, Scott Yanow rated Saxuallity at 3 stars in the 2nd edition of the print AMG Jazz Guide, Henderson pumped it up to 5 stars in the 3rd edition and he then dropped it down to 4 stars in the 4th Edition. Throughout that time, it's always been highlighted as her best effort. I sense that there was some expectation that she would flower into something greater and subsequent albums didn't live up to those hopes, which caused reviewers to think less of her previous work. If you enjoy that type of music then it's worthwhile. I certainly have corners of my music library where my taste doesn't agree with others.
Photo of rootsmusic

rootsmusic

  • 570 Posts
  • 3 Reply Likes
Andy, I'd prefer that AMG focus on reviewing than on historically reconsidering the rating of reviewed albums (see https://getsatisfaction.com/allmusic/... ). Ratings aren't very meaningful to me due to this thread's problem. But editor picks help me decide which album to start exploring, especially when the unfamiliar artist's catalog encompasses different styles.
Photo of Andy DeNardi

Andy DeNardi

  • 231 Posts
  • 20 Reply Likes
I must not have stated my case clearly. I want to know how many reviewers that have on staff and how often they go outside to get them done. Ratings are meaningful to me, but I'm already aware that they're very fluid on AMG. I did not, and have not ever, asked for reconsideration of a rating.

I simply want to know that whomever was chosen actually had an affinity for that style of music and wasn't just handed the job because they were "the jazz guy". AMG probably has adequate depth to cover rock from Bieber to Cannibal Corpse, but what about the blues? I assume from your username that you understand how someone who favors Stevie Ray Vaughan might have difficulty warming up to a Butterbeans & Susie record from 1925.

I specifically made the point that although the reviews on this Candy Dulfer album have been all over the place, what was important was that it was always an editor's pick and that should be considered an important data point.

I did ask for someone capable to handle non-western albums because they're not adequately documented, but didn't state that I wanted them to go back and do previous albums.

I've read many of your previous comments and know that you have a long-standing interest in music. I don't always agree with your opinions but you have valid points. I hope that we can continue to hash out our differing viewpoints. It's becoming so difficult on all of these faceless forums.
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
Hi Andy,

We have a total of 23 writers on our staff, including our US and UK office and our classical department. We have a smaller number of freelancers, mostly non-US/UK writers (i.e. not writing in English), which fluctuates according to who’s active and who’s inactive. And you’re right, it is difficult to find people who enjoy and have a good understanding of certain types of music. We’ll never lack for people interested in indie rock, but in some very popular genres (like gospel and Latin music and commercial pop), it’s very difficult to find people who can give music the full critical treatment.

That's a very interesting perspective on the different ratings in old editions of the jazz guide – during the ‘90s, we had Ron Wynn and Scott Yanow editing those books (Alex Henderson has been a freelancer, but not an onsite editor) and you can see each editor’s thoughts about Candy Dulfer specifically and smooth jazz in general.

With the decrease in popularity of jazz over the past 20 years, it’s become less feasible for us to carry a writer with smooth jazz expertise. Currently, you will see anyone from Thom Jurek to Al Campbell to Matt Collar to Sean Westergaard and even Andy Kellman covering jazz. I believe their range of interests – from old to new, straight-ahead to free, smooth to raw (?) – gives us what we need for jazz specifically.

Expertise in non-western music is not only hard to find, but also to justify in helping us make a living. Here too, although we don’t have writers whose only focus is non-western music, we have 6-7 writers who are consistently reviewing music under that umbrella. (Incidentally, the new redesign makes it very easy to find recent material in each genre – check out this link for our international section: http://www.allmusic.com/genre/interna.... You might find some odd choices there, but we consider Anglo and American folk, and even styles like Cajun, to be international music.)

Hope this all helps – Matt has already spent some time on Candy Dulfer yesterday, and he should have some answers for us later today or tomorrow.
Photo of Chrysta Cherrie

Chrysta Cherrie

  • 731 Posts
  • 9 Reply Likes
Hi Mark,
John may well correct me, but my assumption is that GRRR! is rated 4 1/2 stars for now because we don't give out 5 stars right off the bat. One of the criterion for a 5 star album is passing the test of time; even though this is a compilation it may be subject to the same scrutiny as a proper album.

Thanks.
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
Compilations can indeed go in with an immediate 5-star rating, since the music is not new. For us, not giving 5 stars to new music is mostly about not wanting to get carried away with how much we love an album right off the bat, which seems to happen a lot... ;>

Anyway, my guess for GRRR! not getting the full 5-star treatment is how long it was -- since we like to make our pick album an economical way to be introduced to an artist -- but I'll see if Tom (i.e. Stephen Thomas Erlewine) can pass along his thoughts.
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
And here's what Matt Collar had to say about Candy Dulfer:

"Based on the user feedback and after reviewing the album in question, I decided to replace the existing review for Saxuality to better reflect its quality and importance in Dulfer’s discography."

He also re-reviewed Sax-A-Go-Go. Since the new reviews won’t appear on the site until early next week, here they are below...

Saxuality
Dutch smooth jazz saxophonist Candy Dulfer's debut album, 1990's Saxuality, made a splash both critically and commercially upon its release and helped propel her to global stardom. The daughter of saxophonist Hans Dulfer, Candy Dulfer had performed since she was an adolescent and by her early twenties was opening for Madonna and Prince. Saxuality builds upon these experiences with productions from multi-instrumentalist Ulco Bed that are equal parts Prince, David Sanborn, and '80s Miles Davis. Although Dulfer's slick approach here fits squarely in the pop-jazz vein, she was inspired early on in her career by such players as Sonny Rollins and Maceo Parker. Consequently, these influences help make Saxuality a more funky and engaging listen than many similar albums of the time. While not exactly innovative, Bed's mix of programmed beats and synths next to actual instruments was fairly inspired and the album works as a bridge between the club-oriented acid jazz coming out of Europe in the late '80s and the radio-friendly smooth jazz of American artists like George Benson and Spyro Gyra. Not only did Saxuality perform well for Dulfer, selling well over a million copies worldwide, but it also garnered a Grammy Award nomination for Best Contemporary Jazz Album. ~ Matt Collar

Sax-A-Go-Go
Saxophonist Candy Dulfer's sophomore album, 1993's Sax-A-Go-Go, built upon the smooth jazz of her debut while also playing up more of her hip-hop and dance music influences. Once again working with producer/multi-instrumentalist Ulco Bed, Dulfer delved even deeper into the club-ready funk and acid jazz that was in its heyday during the early '90s. These are synthesizer and drum machine-heavy productions showcasing Dulfer's high-energy saxophone lines. In that sense, tracks like the title cut (featuring rapper Easy Mo Bee) and the swinging funk number "Bob's Jazz" sound like instrumental takes on the hip-hop and R&B sound of groups like TLC and Bell Biv DeVoe. A slick studio production for sure, but Dulfer's longstanding love of artists like Maceo Parker, Miles Davis, and Prince came through. In keeping with this more organic, swaggering sound, Dulfer covered '70s jazz-funk pioneer Les McCann's classic "Compared to What" and delivered a convincing take on Average White Band's "Pick Up the Pieces." The result is an album that successfully conveyed Dulfer's own jazz and funk-based style, just as it celebrated her standing as the queen of smooth jazz party music. ~ Matt Collar
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
And one more thing: I just talked to Tom, here's what he said about GRRR!:

"This was a tough call, as it is an excellent compilation in all its incarnations but the fact that there are so many versions -- a 2-CD, a 3-CD, a huge limited edition box -- is the reason why it's at 4 1⁄2, not 5: the Forty Licks comp benefits from its simplicity, so that's why it remains at 5."
Photo of thunderraver

thunderraver

  • 0 Posts
  • 0 Reply Likes
After surfing this thread a few comments as a former music reviewer/critic for other publications:

To the readers commenting on the reviewers' "inconsistencies": Keep in mind that all of the "reviews" and "ratings" are merely the opinion of the "reviewer" or "rater." And, also keep in mind that reviewers/critics are -- unintentionally and even in the best of circumstances -- trying to be clever/insightful/profound/knowledgeable at different times and sometimes all at the same time. That, combined with the fact that they (generally) probably listen to waaaay too much music, all lead to reviews that say things like "so-and-sos most uneven album" accompanied by a 5-star rating. Somewhere in the nooks and crannies of that reviewer's Big Brain that may make sense. To you and me, it may not. Take it for what it is: one person's opinion about one album/artist. That's difficult to do, considering that the opinion is published to a wide audience and not just expressed over beers at a local pub. But, the fact is, that's exactly what it is: one person's opinion at one point in time.

I personally know one case of a review of another publication who slammed a particular hip hop album in print. Many years later, he would come to regard the album as a classic. Months after writing the review he regretted it, realizing that his harsh review resulted from the fact that he'd simply listened to the album way too much before writing about it. It was one of his favorite groups and initially he loved the album. Then, he kept listening to it over and over and over again. In the process, he began to nitpick. And the full set of nitpicks wound up being the focus of his entire review. It happens. Fortunately for him, he was able to write about that whole experience and share it for a broader audience.

The point here is that: reviewers are human with their own biases and foibles and shortcomings.

On a personal note: I'm an avid reggae fan and, arguably "know" more about it than the average music fan (and the average reggae fan). And, some of AllMusic's reviews and ratings of reggae stuff are IMHO not accurate/off-base/just plain wrong. And after the initial "WTF!!!?!?!?" feeling, I go back to my point above. I do, however, lament the fact that some Great Music may not be purchased/listened to because some people will base their buying/listening decisions completely on the reviews they read. But, that's the nature of the beast. (Reviewers are reviewers not publicists.)

Indeed, one could also argue that part of a music reviewers/critics role is to incite discussion and even debate over an album or artist. So, the fact that there are objections to reviews and/or ratings here could be considered part of the process/discussion that AllMusic is unwittingly intending to promote.

And, to the extent that this is the case, Rage on!
Photo of Andy DeNardi

Andy DeNardi

  • 231 Posts
  • 20 Reply Likes
Thunderraver -

I understand your point that a review is a snapshot of a moment in time for the writer and that they may rate a record somewhat differently at another time. I don't have a huge problem with star ratings going up or down. If anything, it's helpful because it gives me an idea of the critical community's commitment to a specific opinion.

But to give an album four and a half stars, yet say "Most of the 73 performances are difficult to sit through"...that's just bad writing. If it deserves a high rating despite the faults, those qualifications need to be spelled out within the review.

It's like giving a film an Oscar, then saying that it was too tedious to watch. It needs to be explained that acting was exceptional, the special effects were unprecedented and the score was perfectly fitted to the action. And for those reasons, an otherwise unwatchable storyline must be tolerated to in order discover new standards in the art.

In a recent example, the review for the Rolling Stones GRR! states "best overall Stones comp to date" but gives it fewer stars than Forty Licks. Above, John Bush cites the many permutations as a partial reason why it wasn't rated more highly. That paragraph belongs in the review. "best overall Stones comp to date but...the Forty Licks comp benefits from its simplicity, so that's why it remains at 5".

It's a bad example of course. A few years from now, we'll only have one basic version of that set and either it or Forty Licks will be deleted in favor of another compilation. It's damn hard to make comparisons between collections because they have a short shelf life that renders the opinion moot.

AMG is brave to assign star ratings at all. Many don't because of the reasons that you've cited. It's unfortunate that we don't have another omnibus music site that can be used for a second opinion. Perhaps the recent changes will compel someone to create one. I have hopes that the addition of user ratings on this site will provide that function over a long time frame.
Photo of Glen Bourgeois

Glen Bourgeois

  • 0 Posts
  • 0 Reply Likes
Personally, I don't know if I need another site offering _experts'_ opinions on albums. I still enjoy reading AllMusic reviews for some of the albums I now know by heart, and I still remember some of the "Similar albums" proposed to me through the years which fast became favorites (amongst them, Love's "Four Sail", The Strokes' "Car on Fire", Supergrass' "Road to Rouen" and "A Better Land" by Brian Auger's Oblivion Express). Perhaps we need less of "These are THE EIGHTEEN Miles albums to start with" and more of "These are the first eighteen JAZZ albums you need to hear". Hard hop for beginners? Jazz Samba for beginners? Progressive Rock for beginners? Southern California Soft Rock of the Late '70s for beginners? Appalachian Field Recordings for beginners? The "first 20 you need to hear" possibilities are endless, and may _theoretically_ be easier to compile than to whittle down the list of essential Miles albums (and are you going to give In a Silent Way, or even Panthalassa, first pick before Four & More or Kind of Blue? Of course, those are my favorites but possibly far from yours). The debates, however, will probably equally be endless.

I remember snickering at some "You might like"-styled suggestions AMG made through the years, then getting my hands on a listening copy and being blown away. I also remember being in complete disagreement with an AMG critic before over certain albums. (I even have an old AMG database CD-ROM which seems to predate the Internet... Talk about brief reviews! Gentle Giant's "Octopus" gets four sentences, period.) Everybody loves to feel like they know better than the expert from time to time. But for me, I find what best compliments AMG are user-based, genre-specific review sites (such as Proggnosis and Progarchives for progressive rock, for example). After a while, you get a feel for which users seem to best share your tastes, and their raves soon become the first albums for which you reach. Combine that with AMG's expert-driven, "start here"-focused approach, and you're getting somewhere. Just don't tell me a 3-star album is an "artist pick" when there are two five-star albums elsewhere in the discography, and a 2.5-star album alongside them is given an ecstatic review. Of course, I'm sledge-hammering all the examples into one theoretical discography... but that discography may actually exist.

(And for the record, all respect to Francois Couture and other reviewers, but from what I've read (or seen undeveloped) of Quebec progressive rock groups on AllMusic, it seems you could use someone like me... if I only had the time.)
Photo of Cameren Lee

Cameren Lee

  • 0 Posts
  • 0 Reply Likes
Peter Hammill's earlier albums are mostly given 3 stars, but the actual reviews seem more...positive than that. The best case is with Chameleon In The Shadow Of The Night:

Chameleon In The Shadow Of The Night is said to boast “much stronger compositions than the preceding Fools Mate” and apparently “is regarded as one of Peter Hammill's best albums”. Both of these statements are made null and void by the 3-star rating, because:

1. It still has the same rating as Fools Mate.
2. There are 17 Peter Hammill albums with higher ratings.

Am I just perceiving a problematic discrepancy, or is this review not as faintly praising of the album as the rating would indicate?
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
Hi Cameren,

Thanks for the feedback -- we have a lot of Peter Hammill fans here (including yours truly), so I was surprised to hear about any ratings discrepancies. I think you're right, though -- those first few albums all having three-star ratings look very suspicious, and definitely don't reflect the variations I'd expect from Greg's reviews.

I'll do some consulting with our other Peter Hammill fans here and let you know what we decide. (In the meantime, feel free to suggest replacement ratings.)

Best,
John Bush
AMG Senior Managing Editor, Pop Music
Photo of Andy DeNardi

Andy DeNardi

  • 231 Posts
  • 20 Reply Likes
http://www.allmusic.com/album/circle-...

This group tends to play bluegrass-style music for children. "Enthusiastic," "well-meaning, " and "child-appropriate" apply; however, something is missing. The albums don't quite click -- not any of them. For ages three to five.


Rather a weak review for a five star album, album pick, and best of ten, don't you think?

As I go through your five star picks via Advanced Search, I get the feeling that you were motivated to pick at least one album each year from a host of genres, like children's, soundtrack, gay, jazz, classical, etc. I notice a lot of five star albums that were only four and a half stars six months ago. Albums by artists that are well enough established so as to not need a reevaluation of what they released thirty or forty years ago.
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
Yes, that's an incredibly weak review for a five-star album, thanks. Children's music is definitely a genre that doesn't get much attention, so I'm not surprised to find odd discrepancies there.

Could you let me know a couple examples of albums that were bumped up to five? The only project we've had recently is around R&B, after Andy Kellman realized that R&B from the '80s and '90s was severely under-represented at the five-star level.

Thanks again,
John
Photo of Andy DeNardi

Andy DeNardi

  • 231 Posts
  • 20 Reply Likes
In this review for Freedom Fire: The Indestructible Beat of Soweto, Vol. 3, you gave the album 5 stars.

In this review for the very same album, it only rated 4 stars.

And this one, for again, the same album, has no star rating at all. Who could blame you, it's all too confusing!

The first, five star listing appears to be ancient. If it were me, I'd go with the 4 star one.

And thank god for this forum because if I submitted this through your faceless correction submission form, the problem would still exist five years on. It's easy to ignore problems when you don't have to answer for it.
Photo of rootsmusic

rootsmusic

  • 570 Posts
  • 3 Reply Likes
Good catch, Andy! I suggest that you post this as a new problem though, because the album shouldn't have been inputed twice into AMG's database. (The problem isn't regarding "syncing" between each review and its respective rating.)
Photo of Zac Johnson

Zac Johnson, Official Rep

  • 3335 Posts
  • 189 Reply Likes
Yep. Those three releases from 1990/1991/1995 should be merged. Nice catch.
Photo of johnbush360

johnbush360

  • 136 Posts
  • 20 Reply Likes
And apologies for your difficulty with the submissions form. Just so you're aware, we get at least 600-700 pieces of feedback every week, so we have to prioritize them accordingly.
Photo of Chrysta Cherrie

Chrysta Cherrie

  • 731 Posts
  • 9 Reply Likes
Andy, sounds like this has been sorted out on Rovi's end. It'll update to AllMusic within a couple days.
Photo of Gar Seeya'

Gar Seeya'

  • 20 Posts
  • 0 Reply Likes
Very positive review for Joe Walsh's "You Can't Argue With A Sick Mind" and yet it only has 2 stars.