The Shrine

I was just thinking the other day, when you get something like say Prince of Tennis where one of the guys say data predicts the serve will go into the middle 95% of the time, and yet techincally the ball can still go somewhere else besides the middle and the statistics would still be correct.

Hollinger's Playoff Odds has Miami Heat at 100.0% of making the playoffs earlier the season, and at some point it became 99.9% when they started losing a bunch. How can the system be right if you go from 100% to 99.9%? Yes it doesn't factor in injuries but no one on Miami got injuried during that stretch I think. Doesn't the fact that you predict the Miami Heat have a 100.0% chance to make the playoff at an earlier mark means there is no conceiveable way their chance to make the playoff is ever below 100% in the future (again, ignoring stuff like say LeBron and Wade both got injuried tomorrow which is clearly not modeled)? Otherwise what the heck are you simulating in these predictions? Lakers currently are projected to have a 99.9% chance to make the playoffs, but if they don't make it, is the simulation wrong? No, because it has a 0.1% chance of happening! So how can such a model can ever be proven false? You can't just load real life again and replay the same season a thousand times!

I know these systems are founded upon stuff that (usually) makes sense, but then a system that isn't falsifiable is not science. The only way you can prove a statement wrong is if I predicted some team will win 100% of the time or 0% of the time. I can tell you right now that I predict the Eagles have a 75% chance of winning the Super Bowl, and no matter what happens I can't be wrong because this event only happens once. Now obviously you can consider a larger sample size but if you're talking about stuff like predicting winner of Super Bowls it's not something that happens enough to rule out statistical anomalies unless you're predicting Super Bowl winners for the next 25 years, and chances are by the 10th time you turn out to be wrong people would've forgotten about your predictions anyway. There's this geek smackdown event they held in ESPN where various guys use their fancy model to predict who's going to win the NBA Playoffs, and they can only predict the winner about ~70% of the time. That's not really saying much when you consider you've a 50% chance to pick the winner even if you knew absolutely nothing about the teams, and I think you can easily hit 70% if you just pick the team with home court advantage.

It must be nice to have a job where you can never be wrong!

Here's what I think would be actually interesting, verifiable simluations. Say, take 60 games into the NBA season and use your model to project the playoff seedings. You got 3/4 of the season so that better be enough data, but there's still enough unknown left that you can't possibly just look at the current standings and say that's going to be the result.

It's ironic to note that the Celtics were well behind the Magics/Cavaliers in every statistical measure (+/-, win/loss, average point margin, whatever) of a *good* team last year and yet the guy who won the smackdown picked Celtics to win based on his instincts.

Don wrote:Hollinger's Playoff Odds has Miami Heat at 100.0% of making the playoffs earlier the season, and at some point it became 99.9% when they started losing a bunch. How can the system be right if you go from 100% to 99.9%? Yes it doesn't factor in injuries but no one on Miami got injuried during that stretch I think. Doesn't the fact that you predict the Miami Heat have a 100.0% chance to make the playoff at an earlier mark means there is no conceiveable way their chance to make the playoff is ever below 100% in the future (again, ignoring stuff like say LeBron and Wade both got injuried tomorrow which is clearly not modeled)? Otherwise what the heck are you simulating in these predictions? Lakers currently are projected to have a 99.9% chance to make the playoffs, but if they don't make it, is the simulation wrong? No, because it has a 0.1% chance of happening! So how can such a model can ever be proven false? You can't just load real life again and replay the same season a thousand times!

Such a system is already wrong, if it's making 100% anything in its predictions. It's possible that it's a rounding error, but it's definitely not 100%. Somebody could maim or kill every Lakers member, and then they wouldn't make it to the playoffs. Sure, it's beyond the realm of any reasonable possibility, but it's not 0%.

However, these type of systems are not based on single events, like you're saying. They are based on previous experience, of both the team and basketball in general. Because it's not pure mathematics, it's not an exact science. Not like, say, poker odds.

You have a 30.375% chance of making a flush on the turn or river, if you have a flush draw on the flop. No more, no less. The percentage is scientifically and mathematically provable. It is an exact fact. However, when you play the game and ten times out of ten you're not getting that flush, that doesn't mean that the percentage is wrong. It just means that you were unlucky those ten times. Poker has the advantage (and disadvantage) of random chance to obfuscate what would normally be a very predictable system. The fact is that it IS predictable, but the random element makes people think that it ISN'T predictable.

That's why poker players talk about short-term vs. long-term advantage. Technically (and it does occur often), any yahoo could make it to the final table of a WSOP tournament and win. But, to consistently do that requires a much better understanding of the game. Same goes for cash games. Many players can make some short-term gains, but it's the ones that understand the odds of the game (which is the CORE of poker) that will get long-term profit over and over again.

Going back to basketball, it's that random chance that obfuscates just how predictable (or unpredictable) the system is. We don't have the luxury of knowing all of the variables like we do with poker. However, the swing of that random chance may actually be much less than poker. In poker, one hand could be a 2-7, which you win with, and another could be pocket aces, which you lose with. In basketball, you have the same members in the same teams with the same couches for every game. They perform much the same way, and it's a high scoring game, so you don't have those random puck shots like in hockey that could win a 1-0 game. In other words, despite the numerous variables that we can't predict, basketball could actually be more predictable than poker, because of that lack of a serious swing between wins and losses.

Just look at the number of upsets in any given season. When has a 16th seed beaten a 1st seed in college basketball? Even when you are talking about lower/higher seed combinations, upsets are still fairly rare.

Obviously no system can predict things like say a terrorist blew up the plane the Lakers are on, but like I said when the Heat went from 100.0% to make the playoffs to 99.9%, they didn't suffer any major injury (if at all). They merely lost a few games in a row which lowered their *power level* so to speak in the formula which caused the future simluation to no longer give them a 100% chance to make the playoffs. Of course the simulation probably assumes the team's power level never changes in the season, so they might as well hand out DBZ scouters and then you can just read each team's power level and know who's going to win ahead of time.

The thing is with Poker everything is proveable (by math) and also verified by simulation. If you played 1000 hands where you started with pocket aces your win rate should approach whatever math predicts. You would be very surprised if you win only say, 1% of those games, as the chance of that happening should be borderlining impossible.

On the other hand if I say the chance of Lakers winning the next 10 games is a number greater than 0% and less than 100% there is absolutely no way to prove this is wrong or not regardless of the outcome. Now if you make that stretch go for 100 games you could start noting like how come the Lakers lost 10 games where I predicted they've a 99% chance to win? But even if you go that far, I can probably just do something like:

1. Make an educated guess for record after 100 games, say 70-30 is what I think the Lakers will do.
2. Divide 7000 by 100 games. Here you can look at the schedule and mess around with the numbers some, like if you see Lakers on a road game against a tough team you can say chance to win is 50%, and give Lakers an 85% chance to win if the enemy is say, the Clippers.

As long as I don't get too greedy and predict something like a 99% chance to win, and assuming the Lakers finished with a record within 10 games of what I guessed their final record is, then I'm sure if you work out the math it has to be the case that my indivdual game predictions were within some reasonable expectation. Now obviously if Lakers finished that stretch with a 30-70 record then my predictions would be very bad, but it's really not that tough to guess within 10 games of a team's ultimate result assuming you're talking about a stable team (no major injuries, no major acquisitions). In fact if one of that happens you can simply say 'Well I didn't know Dwayne Wade was going to join the Lakers halfway after Kobe Bryant got injured', and I think it'd be hard for anyone to demand you have a model that can account for stuff like that.

But then you end up with a prediction that's not verifiable in any way. Let's say on ESPN the WSOP % for each hand is now replaced by random numbers I made up, so you'll see a guy start with pocket aces and it says '1% to win'. After you see about a hundred of those hands you should, even if you have no idea what the math is involved in Poker, be suspicous how come that '1% to win hand' wins the vast majority of hands. There are clearly enough hands played in the WSOP with pocket aces shown on TV to make you say, "I think there's something wrong with the number 1% to win with pocket aces". But there isn't enough sports games to do this, and certainly there isn't enough Super Bowl winners to make any accountability for a prediction based on percentages. When you see a WSOP guy hitting say a 25% chance to win 3 times in a row you obviously trust the model enough to say that the guy is just pretty lucky (1/64 chance of that happening). But can you trust a model that says the Lakers have a 75% chance to win the championship for the next 3 years when they failed to win any? Statistically, it's exactly the same (25% event happening 3 times), and yet I obviously just made the latter up while the former is clearly something that must have happened at some point in WSOP.