Thank God nobody takes me seriously.
Okay, I'll admit it. I was happy to forget statistics after studying it as part of Algebra II back in high school. Seeing John Hollinger's name come up as a subject of derision more than once over at Dallasbasketball.com inclined me not to trust his methods, and the latest rankings (the Mavericks have actually slid down to 13) just seemed silly.
Then I read Rob Mahoney's post saying very clearly what brand of salt Hollinger's opinion should be taken with. Not evil, just overly objective (athletes are not just and only stat machines). So I deferred to Mr. Mahoney's opinion, because he knows more about the sport and the advanced stats used in defining it. I posted an apology and all was well.
Then this evening when I got home, this hit me in the face like a rolled-up sweat sock.
So how seriously am I supposed to take a numbers guy who comes up with imaginary matchups, cites factors used in making his decisions that he specifically ignores in his calculations (the Bucks trade for Salmons was better than the Mavericks trade for Butler/Haywood/Stevenson?), and glosses it over with things like, "[T]he only reason the Magic lost is because, 'Orlando also shot 4-for-25 on 3s in that game -- an event unlikely to repeat itself in future meetings.'"
Uh-huh. We're back in "bad science" territory.
And then over on the boards . . .
I'm including the full post because it bears reading. It's also gotten the Good King Of Numbers stamp of approval:
"But, here's the thing...I'm not even sure Hollinger knows what true 'statistics' are. I understand that baseball 'stats' are relatively simple 'hits/at bats = BA', but Hollinger goes by this 'Advanced Level Stat' creed, so he should be held to higher standards. I'm speaking specifically of his power rankings system...I haven't looked the intricacies to his PER calc (maybe they're good). His formula 1)doesn't use ANY statistical method to evaluate the numbers, and 2) (speaking as a financial analyst who scrutinizes formulas all the time) is completely arbitrary.
"Here's the formula RATING = (((SOS-0.5)/0.037)*0.67) + (((SOSL10-0.5)/0.037)*0.33) + 100 + (0.67*(MARG+(((ROAD-HOME)*3.5)/(GAMES))) + (0.33*(MARGL10+(((ROAD10-HOME10)*3.5)/(10)))))
the first part is a strength of schedule calc, and the second bracket is the strength of schedule in the last 10 games. You'll notice he's multiplying the first one by 2/3 and the last 10 games by 1/3, which just means that's he 'weighting' the significance of these numbers, and then adding them together. Well....how did he come up with 2/3 and 1/3? That's totally arbitrary...it can be adjusted at his whim. If he had solid STATISTICAL evidence proving that weight, then that would mean something. But the fact that he's using such simple fractions is a clear indicator that he pulled them out of his ass. Same thing with the Margin of victory portion of the calculation. A REAL statistical model (and, mind you, I'm not expert...I will have my MBA in a few months and I've been exposed to lots of stats methods, but I don't claim to be any good at them) would look for SIGNIFICANCE in the numbers that he is coming up with. That is what statistics is all about...what are the MEANINGS of these numbers. Are they reliable? How volatile are these variables? Well, the fact that the Mavericks are being popped out as #13 in his rankings should send a red flag to John that his 'model' is off. It's not reliable. But, instead of looking for ways to improve it (which would be welcomed by anyone I would think) he tries to defend it (with shocking arrogance, I might add) and paints himself into an even smaller corner than he was before.
"He's not a 'statistics' guy in a real sense. He comes up with sports 'statistics', but tries to pass them off as something more than they really are. A person better at stats than I should take all of his final regular season power rankings from the past few years, and run regression analyses against actual finishes as well as head-to-head matches during the season. The answer to 'how reliable are these rankings' is in there...it's just math. But, as I sit here and look at this formula more, I want to laugh at how simplistically stupid it really is. Yes, it will rank teams semi-appropriately, but he acts like it can quantify those 'hidden' variables in teams. It's a joke."
-duxthaman
http://dallasbasketballdotcom.yuku.com/topic/40338
Is there a peer review process for sports statisticians? And if there isn't, could one be created? It's one thing for a guy like Hollinger -- who's really for entertainment purposes only -- to cover the weaknesses in his system by massaging numbers and flashing number-scaled red herrings. It's quite another if the stats guys in front offices do so. If you believe him, Avery Johnson swears he went SmallBall against Golden State in 2007 because he followed the advice of the Mavericks' statistician who told him it would be advantageous to do so.
-BJ
Friday, March 12, 2010
I'm Sorry, I'm Confused
Subscribe to:
Post Comments (Atom)
2 comments:
There's not really a peer review process per se; other statisticians will certainly let their opinions be heard on the metrics of a variety of analysts (Hollinger and David Berri being the most notable), but even that criticism won't be enough to totally discredit anything in the eyes of most.
Hollinger's work is far from flawless. The calculations do have problems, particularly the power rankings (I find PER to be much more palatable).
But, I mean, they're power rankings. His are the same as anyone else's: laced with subjectivity. But whereas Marc Stein might look at different criteria (W-L record, overall feel for how hot a team is, etc.), Hollinger does plug-and-play with his formula. The weightings are going to be arbitrary because the very nature of power rankings is to be arbitrary.
Now, the explanations are a bit odd. I'd completely agree. The logic isn't consistent, there are definite holes there, and John isn't acknowledging the limitations of his thought process or the system (which is never good).
Should you buy what Hollinger's selling? Eh, up to you. I don't see the harm in treating him the same way you'd treat any other columnist/analyst, even if he has more of a statistical bend than most.
The thing that bothers me about Hollinger -- beyond a homer hearing what she doesn't want to hear -- is he's using numbers (and the average schmuck's lack of understanding of them) to give his opinions an authority they don't deserve. He might be on a similar level to a Marc Stein or an Art Garcia, but he's taken more seriously because he sounds so much smarter.
And as you pointed out, his work has a lot of influence on people's opinion of advanced statistics. If his formula is discredited, it leads to questions about the validity of advanced stats as a analytical tool. Might be a good thing; might not be.
Thank you for the reply. I appreciate it. :-D
Post a Comment