Adjusted Plus-Minus: Hustling Outside the Box Score?

The Sloan Sports Analytic Conference took place this weekend and one of the popular topics there has always been player evaluation metrics. One of the player evaluation metrics that seems to garner a lot of discussion there is Adjusted Plus-Minus (APM).


The conference featured two research posters on APM in basketball this year and was featured in the research paper of one of its finalists last year as well as a panel discussion titled, "What Geeks Don't Get: The Limits of Moneyball." Dallas Mavericks owner Mark Cuban has stated on more than one occasion that he's a fan of using APM to build strong lineups.

The value of APM is supposed to be that it accounts for interactions on a basketball court that impact the outcome but don't get recorded in the boxscore (e.g. setting screens, man-to-man defense, etc). This value was supposed to be justified by its ability to explain wins in the NBA, but it was questionable due to the inconsistency and statistical insignificance of its results. The problems with APM were documented (and explained) by Professor David Berri at the Wages of Wins Journal.

Another blogger from the Wages of Wins Network, Arturo Galletti, deconstructed the APM model used at basketballvalue.com with  hopes of improving its consistency and found it has another big problem - APM doesn't have the explanatory power previously reported because the model was created using "bad math" that would get pharmaceutical companies shut down.

I participated in a podcast on Friday with Arturo, Prof. Berri and Andres Alvarez where the issues with APM were discussed and I think the issues are pretty big.

The Issues

• The creators of the APM model Arturo deconstructed had to know they were using bad math. One of the creators, Dan Rosenbaum, consulted with the Cleveland Cavaliers. So the question becomes, "Did he use that bad math to hustle the Cavs out of consulting fees?" And why would any NBA team buy "bad math" in the first place? All NBA teams have statisticians, so shouldn't they be able to distinguish "bad math" from "good math" so they don't get hustled? And if they can't, isn't it their own fault?
• Yes, in the real world it's caveat emptor (buyer beware). The problem, of course, is that the buyer gets screwed when the information asymmetry between the buyer and seller is high. There's no Carfax for advanced NBA stats (far as I know). Yes, I equate the sellers of "bad math" to used car salesmen.
• The Sloan Conference was organized by Daryl Morey, the General Manager and Managing Director of Basketball Operations for the Houston Rockets. Morey has a bachelor's degree in Computer Science from Northwestern University "with an emphasis on statistics" and MBA from MIT. He was lauded in the New York Times by Michael Lewis for bringing "Moneyball" to the NBA with the Rockets' proprietary version of APM. Does the Rockets' version of APM use the same "bad math"? 
• The owner of the Rockets is Les Alexander, a Wall Street investor. If the Rockets are using similar "bad math" to what Arturo discovered, then what's the difference between MBAs hustling billionaires in the NBA with "advanced" quantitative analysis vs. MBAs hustling billionaires on Wall Street with the same? And what are the odds that one guy (Michael Lewis) would make both hustles popular (Lewis popularized the quants hustle on Wall Street in Liar's Poker and the quants hustle in sports in Moneyball)?
• Many have said the financial crisis could have been avoided (or its impact lessened) if better regulations were in place. Well, at last year's Sloan Conference, Cuban made the point that the NBA should create a central repository of advanced stats that are available to all the teams (I think he was specifically talking about defense). Should the league office do the same for analytics? Should the stat geeks behind StatsCube (the NBA's official repository for advanced stats) provide consulting services to teams to act as a Carfax for bad quants? Is StatsCube itself tainted with bad data or does it only provide analysis using peer-reviewed models? How does the NBA fix this problem? How much money was lost to this hustle? Doesn't it have to be millions in player contracts and  consulting fees?
• Lastly, how did APM impact the fans? How many wins were lost to bad analysis? Avery Johnson said a version of APM cost the Mavs a playoff series against Golden State in 2007 when they had the best record in the NBA. Is it possible that titles were lost to APM?

Those issues are discussed in part during the podcast but there probably needs to be a lot more discussion about those topics and more by people that do more for the NBA than blog about it on their spare time.

The good news for Heat fans is that their statistical consultants, Shmuel Einstein and Bob Chaikin, don't appear to use adjusted plus-minus and the Miami Heat Index doesn't either.