Offensive Holding in the B1G and Beyond: Part Deux
As a follow up to my earlier thread here, I performed a deeper dive on offensive holding. In the initial thread, I found that in 2014-2017, Michigan's defense had only drawn 10 offensive holding calls, significantly less than all other B1G teams.
TL;DR: Defensive line performance is a poor predictor of offensive holding calls, but Michigan and Alabama both draw holding at a much lower clip than other NCAA teams.
About the Data
I downloaded the 2010 – 2017 Play by Play data from the ESPN API. This contained 1,234,271 plays, during which, 82,570 penalties were called. Additionally, I correlated this data with Football Outsiders’ Offensive Line and Defensive Line statistics, found here.
Each of the 1.2M plays has a “free text” field which describes the play, and if there is one, the penalty and whom it was called on. As an example, this means Michigan could be referred to as Michigan, UM, UofM, Mich, etc. I very carefully developed an algorithm to assign the penalty called, and who it was called on and drawn by, based on this field for all NCAA games involving an FBS team. While I’m sure it wasn’t 100% accurate, I do think I did a pretty good job and designed some good tests to check this process. Probably 99%+ accurate.
Are there any good predictors of Offensive Holding being called?
Honestly, none that I could find. I found evidence that holding calls are generally random…with a slight edge given to the underdog team or team currently losing. This means the argument “Michigan’s D-Line has been amongst the best in the country, we should be getting more holding calls than everyone else” probably lacks merit based on the data.
Comparing Football Outsiders Offensive and Defensive Line metrics (and score differential at the time of the holding call) to holdings drawn. All three lines are downward sloping – suggesting a slight edge is given to the underdog. However, the r-squared value is low; you can see visually it is generally quite random.
Michigan – we’re getting screwed, right?
I’m going to go with a yes. As it turns out, Michigan and Alabama are in a special kind of unlucky streak, stretching for many years. The below is a graph of all FBS teams 2012- present (B1G teams are represented by their logos) for all games played. Michigan is towards the top right – in the “we’re getting screwed” category. The only team that has fared worse is Alabama. A lot worse. Since 2012, Alabama has drawn a holding once every 260 plays. That is nearly 2.5 times the average.
If you zoom in on the B1G for in-conference games over the same timeframe, you see Michigan has drawn half as many holding calls as their peers.
Why limit the analysis to conference games?
Because if there is a bias within the conference, it wouldn’t perpetuate itself in nonconference games. Let’s take a look at holdings drawn in conference vs. out of conference games in the B1G, 2012-2017:
The red teams (Illinois, Michigan, and Nebraska) each have holdings drawn much more frequently in the nonconference schedule vs. B1G play. In reality, this chart likely has as much to do with scheduling choices than anything else. But you see Michigan has a very unfavorable variance, while Indiana and Purdue have a very favorable variance for in-conference play.
Alright. Let’s grab the torch and pitchforks. How long has this been going on?
Well, there is a reason why I keep showing 2012-present. Beyond this reflecting 5 full years (plus the partial season currently being played), it also reflects how long Michigan’s misfortunes on drawing offensive holding has been going on. The below table shows the rank within the B1G each year on plays per offensive holding. A “1” would represent the team that least frequently drew an offensive holding call in the current year. Michigan finishes in the top (err…bottom) 3 in 5 of the last 6 years. Before that time, Michigan fared much more reasonably.
What else did I find?
Not much else, but not for a lack of effort. I reviewed other attributes of when holding was called: Down and distance, quarter within the game, where the play was on the field, etc. – nothing jumped out. I even looked at other penalties to see if Michigan has had similar misfortune elsewhere…And couldn’t find much evidence behind it. So I’ll leave you with one final visual – the total number of penalties (holding or otherwise) called for each pairing of teams since 2010.
…And penalties called per game…as not every team has played an equal number of times:
Where do we go from here?
I don’t think the Big Ten is actively out to get Michigan. I probably should have led with that. There might be some subconscious bias, however. The way you combat that is by providing visibility behind trends in this data, and increased accountability for referees. Finding a way to better compensate officials - and not have this be their weekend job – is probably a good first step. But another step is to have this information be more readily available.
About umich1
Umich1 is a data scientist in his professional job, and graduate of the University of Michigan. He spends a lot of time on Saturdays yelling at officials from row 67 in the big house.
December 29th, 2017 at 4:02 PM ^
which plots sack rating against plays/holding call shows a positive correlation. One would expect that teams that get more sacks would be able to draw more penalties (ie lower plays/holding call and thus negative correllation). This tells me there is a deeper bias here than just against michigan. What do you make of this?
also, it would be cool if you called out who the other outliers were in the graph and showed a couple other big-name programs, just for context
December 29th, 2017 at 4:47 PM ^
The top 5 most screwed programs were Alabama, Michigan, Georgia, Utah, and Boise State.
Notable lucky programs? There really aren't any - they aren't big names. Top 5 are Coastal Carolina, USF, Temple, Arizona, and Tulane.
To your point - It seems the better defenses tend to get less calls, while the lesser defenses get more.
December 30th, 2017 at 12:29 AM ^
December 29th, 2017 at 4:02 PM ^
Alabama is a crazy outlier. Any thoughts on why this might be? One wild guess that comes to mind is that offensive holding against the OL tends to get called in situations where the OL and DL are relatively equal and the OL appears to get an unfair advantage that impacts a one-to-one battle. Where the OL and DL are unequal, either the OL is unlikely to hold (doesn't need to) or the DL "wins" its battle anyway in spite of the hold.
Of course, that would likely show up with other teams like OSU (traditionally with a strong DL) also getting disadvantaged with these calls, and I don't see that here. I suppose one way to test the hypothesis would be to compare games against comparable OL and DL vs. games where one team's line is significantly superior to the other team's opposite line to see whether there is a significant drop in offensive holding.
Or could be bias/bad luck.
Thanks again!
December 29th, 2017 at 4:52 PM ^
Alabama's numbers get especially bad in the nonconference.
December 29th, 2017 at 5:08 PM ^
I was hoping someone can do this for a long time. Unfortuantely, I am not a data scientist so it would have costed me 5x time to bring up my rusty programming skill. Regardless the conclusion I would appreciate your effort.
I would like to share my thoughts (my bad English, not a native English speaker)
1) I don't think no holding call alone would be able to be conclusive enough to tell the story. The referees are smart enough to balance it out throughout the game, so by seeing the number is not necessary enough. A good example, which led to me giving up watching any NBA game (regular season, playoff, any team) and most NCAA (most BIG) regular season basketball game (except for march madness), basketball refs can balance the call very effectively. Think about this way, the timing that the refs call a foul is important too. An offensive foul (withouth reaching the team limit) will only result in pocession change, but a defensive foul under the basket can results in two points (free throw). Ref often can do make up calls, but the impact is different, some can lead to points, some other calls may not (chaning possesion doesn't always lead to points). Go back to football, some call can keep the drive alive, and lead to points, some other calls, like holding can easily kill a drive and swing the momentum. a PI call can lead to moving the chain. However, refs can call the opposite team another PI call/holding call while there is not impact at all (either it get offset or got declined. but this penelties number still would show up in the stat sheet.
2) That being said, I think it's better to link this penalty to its impact, for example, did a holding call directly lead to a 3 and out, that would be a high impact? did a PI call lead to opponent to get points. There may be some sort of impact factor for theose penalties, then we shall see how that affect the outcome of the team.
3) I think it's crucial to find how the referees are asigned by BIG. And compare to their assigning critieria for regular season game. It's based on geographic proximity? or something else? How the win/lose percentage, and penalty percentage for these referee towards different team. The reason I would like to see this is that I'v noticed John O'neil has been assigned to many of our lossing game. and Fans noticed this already. both our Home game vs MSU in 2015, 2017, and away game vs. IOWA in 2016 are assigned to John Oneil. and we got a lot of bad calls in those games, and npot strong enough to win the uphill battle. The way how those refs affecting the game is astonishing to me. They can allow a lot of no call and initiate the physical contacts from oppositing team in 1st qrt, and make our players and coaching gets to a uncomfortable zone. The ref calls IMPACTs play call, people! If you throw a slant and the opponent DB hold/or PI your WR the hell out of it, but no call? Would you call a slant again? in the next few play?
December 29th, 2017 at 4:23 PM ^
Thanks for putting this together! Wish we had more diaries like this one.
This is the sentence that stands out to me most:
"Michigan has drawn half as many holding calls as their peers."
Ya gotta think something is up there. I mean damn that x/y chart is an eye opener. We are indeed getting screwed and it's not just an outlier. It's like double every other team in the entire conference! WTF!
December 29th, 2017 at 4:38 PM ^
To whom at the B1G can we send this? This is the exact sort of thing that ref's need to see, and they need to be accountable.
December 29th, 2017 at 4:48 PM ^
And the contents would be simply the B1G graph with Michigan at the top right, all alone.
December 31st, 2017 at 1:09 PM ^
It looks really bad but its also a tiny sample. You are looking at a dataset of approximatly 100 total calls over 14 teams per year. Its less than 2 calls per game and like Umich1 pointed out if you go back to 2010-2011 there is no longer a bias against Michigan. It would take one game of a hold happy ref to complete skew a dataset for an entire year.
Michigan is getting extremely unlucky but despite being visually stunning I don't think the graph is quite as convincing as some would suggest that there is a problem given the tiny number of total calls.
It is quite stunning how much holding teams get away with though, if you are on offensive coach and look at these numbers you know your guys can get away with anything short of ripping jerseys from behind directly in front of refs.
December 29th, 2017 at 4:59 PM ^
Does the analysis take into consideration holding calls that were declined? Could this answer a little bit of Alabama's bad luck? If they still get the sack and get a holding call they would decline the play. Obviosuly, this would scew the numbers that much and I am sure Alabama is still unlucky here, but it may bring them a little closer to the average...
Also, with the sack rate corrlation, I would assume the defences that are getting the most holding called against the offense are losing out on possible sacks because of the hold, thus giving a possible answer to positive correlation of the sack rating against plays/holding cal.
Just some thoughts...
December 30th, 2017 at 12:17 PM ^
In my initial thread, I had done an analysis for both declined and accepted calls. They painted a similar story. For this analysis, to try to simplify things, I included both accepted and declined.
Great point on the sack rate correlation. If it was a hold, it likely didn't result in a sack. I did look at other metrics, such as overall defense, and it was similar - negative correlation with very low r-square.
December 29th, 2017 at 7:40 PM ^
Pretty interesting! A couple critiques:
- Could it be that that when michigan plays the refs uniformly do not call holding for both of the teams? (for whatever reason). Under this scenario, Michigan would have few holding calls against and for them.
- Suggesting bias from the refs (unconcious or otherwise) seems to be a bit weak given just this analysis, as you did not address/rule out other sources. For example, opponents might be scheming around Michigan's dominate defensive line, causing few oppurtunities for holding. This would also explain Alabama discrepency, as they also usually have a dominating D-line. It could also be a Michigan coaching issue. For example the play calls (ex. unblocked lineblacker blitzing and sacking qb) might not lend itself enough time for holds to be commited/called.
December 30th, 2017 at 12:15 PM ^
- Michigan was closer to the middle of the pack regarding offensive holding calls against Michigan. So I don't think that is an applicable rationalization. Even if it were, calling Michigan games substantially different from others has its own host of issues.
- I think it is fair to suggest that bias is one possible explanation. I agree that there are a lot of potential factors at play, and a lot of those factors are more subjective and less data-centric. For example, what if our opponents just "show up" and block well? With that said, I do think the trend is strong enough where it is worth observing. And regarding the sniff test, It definitely feels like there were plenty of missed holding calls this year (I'm a data guy, not a football X's and O's guy - so take my biased opinion with a grain of salt here).
December 29th, 2017 at 9:03 PM ^
December 29th, 2017 at 9:55 PM ^
These are the type of posts that make me love this blog.
Something is totally wrong about the officiating, and the data makes it clear now.
December 30th, 2017 at 12:24 PM ^
Good stuff.
Very much appreciate someone with your knowledge breaking these down.
When watching games with a dispassionate eye (when Michigan is playing I can't do that) it seems that in most any game played, there is a bad call/noncall, or three, where one team is getting the shaft. If is accepted that mistakes are part and parcel of the whole and teams are coached to play past it.....where it gets unfair is if the bad calls/noncalls are game changers. TDs and first downs are either given or negated based on these calls and at what point can teams overcome incompetence... or a fix? Perhaps a fair conclusion is that there is an unfortunate amount of poor officiating in college FB today.
December 30th, 2017 at 2:09 PM ^
December 31st, 2017 at 10:22 AM ^
December 31st, 2017 at 12:41 PM ^
January 2nd, 2018 at 8:59 PM ^
January 4th, 2018 at 11:33 AM ^
Thank you!!!
January 4th, 2018 at 5:48 PM ^
If I can ever get home from Connecticut, I will e-mail you that file.
Comments