Ratings

  College Football

  College Basketball

  NFL Football

Article Archive

Predictive and Earned Rank Systems

All else being equal, what’s better: a team winning 2 games by 1 point apiece, or a team winning one game by 30 points, but also losing another by 1 point?

  Team A Team B
Game 1 Win by 1 point Win by 30 points
Game 2 Win by 1 point Lose by 1 point

This isn’t an easy question to answer. The answer is that it depends on what you’re trying to measure. Are you trying to predict who is more likely to win in a future head-to-head matchup? Then the answer is that Team B is better. Team A barely won both games. They may have gotten lucky. Team B had bad luck in that they lost by a point, but also had a very dominant win. If you know nothing else about the two teams, Team B is clearly the favorite going forward.

But again, that’s only if your objective is prediction.

What if you are trying to determine, bottom line, which team is having a better season? Then the answer is that it’s Team A. They’re undefeated. Team B had a close loss, but in sports, close doesn’t cut it. Wins and losses are how you decide what a team has earned. Margin of victory is just style points, and style points don’t matter when you want to measure what a team has earned.

So when measuring teams, the first question to answer is: what are you measuring, past performance or future potential? Algorithms that measure past performance are called “Earned Rank” systems (also called “Retrodictive”). Algorithms that attempt to measure future performance are called “Predictive” systems. Neither type is inherently better than the other. They’re both useful, but for different things.

There are many Predictive systems out there. Some of the most well known include Ken Pomeroy, Massey ratings, Sagarin ratings, ESPN’s Power Index and BPI, Inpredictable, SRS, and others. The advantage of these systems is that you can incorporate a lot of data into the system, including margin of victory, location, injuries, and more. The process of accounting for those extra variables is inherently subjective, and the formulas need to be tweaked occasionally as time goes on, but that isn’t necessarily a bad thing. Many of these algorithms are very good at predicting future performance, and are rooted in solid mathematical theory. But even if one of those systems was truly perfect, it would only be a perfect Predictive system. It would not necessarily be a good Earned Rank system.

An Earned Rank system doesn’t try to make the best predictions. Instead, it tries to measure how a team has done—past tense. It measures its body of work over the season. A properly constructed Earned Rank system can only account for wins, losses, and strength of schedule. Even though additional data would allow for better future predictions, it would distort the measure of what a team has earned. A “lucky” win is still a win. That win was earned, and must count just as much as if it were a blowout. And a “competitive” loss is still a loss. In sports, close doesn’t count.

RPI is the most well-known and controversial Earned Rank system. Others include the Elo rating system and the Colley Matrix. There are problems with those systems, but the criticisms of those other Earned Rank systems don’t apply to Predictive systems. Predictive systems have their place.

Predictive systems are the best choice if you’re playing a pick ’em game, or trying to fill out your March Madness bracket, or anything else where you’re trying to predict the outcome of a future game.

Earned Rank systems are the best choice if you’re trying to decide who has earned a spot in the postseason and what their seed should be, regardless of the sport.

In the NCAA basketball tournament, there are times when the oddsmakers favor a 10 seed over a 7 seed. Is this a problem? Or a mistake by the selection committee? Not at all. It may just be a case where the 7 seed overachieved during the regular season. They still earned their 7 seed, even if they earned it with a few close wins. Just because the oddsmakers don’t favor the 7 seed doesn’t mean that they were seeded wrong. It just demonstrates a real-life example of how Predictive systems and Earned Rank systems measure different things.

One Predictive system, Inpredictable, ranks teams based on actual betting lines. Among rating systems that allow an element of human opinion, Inpredictable is probably the most reliable. Inpredictable shows that there are times in college football where oddsmakers consider a team to be in the top four, but have lost too many games to be considered for the playoffs. Despite their obvious talent, they haven’t earned a place in the playoffs. In fact, the most talented team of all can miss the playoffs if they don’t earn it during the regular season. Again, this shows the difference between measuring how good a team is likely to be going forward, and what a team has earned.

There’s nothing wrong with Predictive systems. They’re (obviously) the best systems for prediction. But when you’re trying to measure what a team has earned in terms of wins and losses, they’re a poor substitute for an Earned Rank system.