It turns out that if you try to develop a forecast model while being transparent for 11,000 people you take some lumps. This past week was one of those when the model I’ve been transparently developing throughout the Fall season didn’t give much chance to the “The Martian” as Best Performer. That led me to look more closely at the model I’ve been using and to adjust some of my own expectations of it.
Regression Tests and Expectation Adjustments
I spent some time this weekend performing regression analysis. We’ve now played 25 weeks of FML so there’s quite a lot of data to mull over to look for patterns and what I came to realize is that some weeks, many weeks even, weird things happen. Sometimes those things can be anticipated, like in Summer Week 11 when “Mr. Holmes” grabbed Best Performer despite not being forecasted by any web site but was scheduled to have a significant uptick in theater count. Other times, those things are tough to predict, like in Summer 13 when the Perfect Combination tipped on “Pixels” having a better 4th week of release than anybody thought.
What I found when performing these regression tests is that, much like a professional baseball player, you have to accept in FML that you’re going to swing and miss often. Even professional forecasts of individual films have a large degree of variance, 75% being within +/- 15% of their forecast is the most up to date number I have. By extension, finding the Perfect Combination every week is unlikely, but even finding it 30% of the time would make you a top player. I found what worked the best in the regressions was to take the top 6 Best Performer candidates based on the average of ShowBuzzDaily and ProBoxOffice.com forecasts and run them through a variance simulator +/- 15%. When you do that for the past 25 weeks the Perfect Combination pops out 8 times, 32% of the weeks tested.
So from now through the end of Awards Season, I’m going to run two accounts. On my main one I’ve used since I started playing FML, I’m going to look at what that model spits out every week and apply common sense to it before choosing one of the top 10 candidates it recommends. On the second account, I’ll only use the top pick that comes out of the model, blindly. At the end of 16 weeks, we’ll see which one comes out better.
Week 11 Perfect Combo Probabilities
Here’s what that model recommends for Week 11:
The model suggests that the top Best Performer candidates are “Hotel Transylvania 2”, “Bridge of Spies”, “Peanuts”, “The Martian”, “Goosebumps”, and “Burnt”. Usually there is a new film in the list of candidates the model finds, but for this week that’s not the case. Instead there are films that are as long in the tooth as 8 weeks in theaters and there will be no Thursday data to provide assistance (although the Thursday BoxOfficeMojo forecast should prove useful).
“Peanuts” shows up often to lead lineups of multiple screens of the other candidates, so really it comes down to how you think “Hotel Transylvania 2”, “Bridge of Spies”, “The Martian”, “Goosebumps”, or “Burnt” might do relative to one another and there is some built in hedge with “Peanuts”. For what it’s worth, here’s how each of the remaining films declined a week ago and how much the average of the professional forecasts think they will drop this week:
“Hotel Transylvania 2” – 38.5% last week, 33.4% this week
“Bridge of Spies” – 30.4%, 23.4%
“The Martian” – 22.5%, 20.1%
“Goosebumps” – 31.1%, 29.4%
“Burnt” – 42.3%, 42.9%
Of those, “Bridge of Spies” seems a bit aggressive but the others are relatively small margins. With just three weeks to go, selections will be critical this week to remain competitive in your leagues. Good luck to everyone!