Mistreating PFF grades and others as gospel drives me nuts!

They are not gospel. Whats worse is their use here at seahawks.net has created far more confusion than clarification. The reason is that too many users of PFF data don't understand what it is they are playing with. New age analytic sites including PFF are works in progress. The data produced on players, coaches and officials are works in progress. They do not produce, by way of analogy, the equivalent of definitive SAT or ACT score on anybody.

During this past past season, and by way of analogy, too many .net members were shooting PFF grades back and forth as if they had never been on a range before ..... much less participated in a gun safety course.

Although administrators and/or moderators may chose to move this to the NFL forum after a time, I've posted it in the main forum because of the growing challenges we have recently seen at .net over poorly understood analytic sites. Since the introduction of analytic sites has been unleashed on the public and is growing, it is my hope that we can all learn and benefit together in this thread.

With the off season upon us, I present to you a couple of links to off site discussions and opinions for your review and edification. Please feel free to contribute additional links on the subject.

Link >>> [urltargetblank]http://www.replyall.me/stephanie-stradleys-cast/the-impact-of-advanced-stats-in-the-nfl/[/urltargetblank]

Link >>> [urltargetblank]http://mmqb.si.com/2014/03/06/nfl-offensive-linemen-grades/[/urltargetblank]

The closest thing to NFL stat gospel is DVOA from Football Outsiders. It's not gospel in the traditional sense, because no stat is perfect. That being said, DVOA is a lot better than PFF's grading in my opinion, and I'm a premium subscriber at both sites. I mainly like PFF for the raw stats you can get from them; i.e., not the grades they give which are sometimes wildly off, but for just looking up factual stats that are not, and cannot be, inaccurate; snap counts, how many snaps a player took at one side or the other, or a myriad number of other fact-based pieces of information.

I agree with what you're saying, in general; but I'd also say that offensive, defensive, and special teams DVOA grades from Football Outsiders can be trusted to a pretty high degree, overall. (Their O-line rankings, not so much.)

Unlike baseball where statistics lie very little, statistics in the NFL will be just like scouting: subjective. For me, the point of a statistic should be a measurable shorthand manner of expressing the eyeball test. I think PFF matches the eyeball test most of the time, but they have different people grading different things and some of them are better at their jobs than others.

I still remember one of the preseason game this year when Seattle rushed for something like 7 yards a carry and Seattle's OL graded out with abysmally bad run blocking grades at PFF. I personally thought the run blocking was mostly excellent in that game, and even if I hadn't watched I would have a hard time imagining 7 yards a carry behind atrocious run blocking.

Situations like that are pretty rare though, PFF gets it right more often than not.

It's not perfect, but I do think it's a nice tool when the scores match the eyeball test. When I read a PFF score for a player I haven't watched, I generally trust it while also knowing that PFF is not infallible. PFF also does a good job of breaking down where the points come from, so for example in the case of Antoine Winfield they were very upfront about his high score coming from his excellent run defense.

You both have invested the time to understand the material and it's limits. And, I thank you for your insight. My admitted frustration and motivation is that of producing a thread for those who have not. My interest is in how can we, as a collective forum, better understand and interact with such sites.

Consider this linked concern of PFF >>>>>

Part of the issue I think is that PFF has no control over how people use PFF data. I know initially Neil was dead against even having overall rankings, because people would just look at them 1-50 and say "player x is 20th, player y is 25th, so player x is better".

Click to expand...

Football is such a team and scheme dependent sport, there will never be a perfect stat to objectively measure how good a player is. Even the most isolated positions like special teams players do not play on totally even footing across the board.

I agree that PFF grades are not gospel but they should not be lumped in with advanced statistics. According to PFF, their grades are the results of knowledgeable eyeballs watching tape: "We look beyond the stat sheet at game footage". There are still reasons for skepticism here; their process is not open for external review and they do not have access to team playbooks or schemes. Of course PFF's other statistic metrics "Signature Stats" do belong with the rest of this conversation.

Football statistics can be very useful when they are applied broadly. The sample size problem diminishes as you widen the scope and charting something like league-wide YPA over several seasons can provide indisputable evidence of offensive capabilities across eras. For exampel, Burke's adjusted win probabilities for a single game are very good as is something like DVOA across an entire team.

The problems come when attempting to narrow the target because of the interconnected nature of the sport. For example, looking at the defense of an individual team while disregarding the special teams or offense misses the impacts of field position, ToP, and offensive turnover differential creating negative momentum. The temptation for sports economists is to assume all of these away to keep the analysis feasible (ie.. no such thing as clutch, no such things as momentum), but these are not reasonable assumptions when it comes to football. This problem is exacerbated when you try to break down individual position groups and especially individual players. DVOA for individual players is worthless because the benefit accrued from a positive correlation between good players and increased numbers is offset by the noise introduced.

All the advanced stats are somewhat flawed, including DVOA.

However, if you look at DVOA, PFF, and Cold Hard Football Facts metrics, they all tend to overlap in places. Those overlaps also seem to be the most accurate predictors though.

Other posters have stated the same sentiment but here it is again in a smaller package:

Statistics are tools to be used to enhance our judgement, not replace it.

BirdsCommaAngry":jkux3e5l said:

Other posters have stated the same sentiment but here it is again in a smaller package:

Statistics are tools to be used to enhance our judgement, not replace it.

Click to expand...

Here's the other side of the coin: Stats are tools that free our judgment from confirmation bias.

Both sentiments are correct, Sac/Birds. The problem with stats is you can make them say whatever you want.

My thought is that PFF grades are probably too biased toward action around the ball. I think they say that they only grade the point of attack on runs and where the ball is thrown on passes. For example, what if Thomas and Sherman are playing their coverage assignments perfectly for 95% of the snaps but aren't thrown to? Those are a lot of + plays that aren't being factored in to their overall grades. If they get beat the one time their thrown to, then suddenly their slapped with a negative grade for the game. Additionally, if they're covering up for someone else's blown assignment and get closer to the ball, then they get thrown under the PFF bus. I think this is why ET doesn't get much love on PFF.

Still, I've been guilty of touting PFF scores to enhance a point, and I do think the grades are useful. It's just important to remember that's it's just one of many stats that help paint a picture. And as kearly and others suggested, we're really just looking to confirm what our eyes are already telling us.

People that solely use stats and not understand schemes and roles of players are always way off the mark, it takes a broad scope of game understanding to really see what a given player can do, eyeball test with a semblance of that gives a good scouting report.

PFF is massively overrated.

Their system doesn't take into account situation or opponent. For example, Seattle's pass protection. It gets a disastrous grade because a line without Okung, Unger and Breno doesn't fare too well against J.J. Watt in a one off game.

Well scratch my back with a hacksaw.

Yet a fully healthy line will get a fantastic grade for competing against a pitiful opponent with zero pass rush. These are pointless assessments not worthy of any grading scale.

Penalties have way too much impact on their rankings too.

Earl Thomas was something like the 29th best safety in football according to PFF in 2012.

I wouldn't use a PFF grade to argue the sky is blue.

Anyone who wants to know about the brains behind PFF, read the SB prediction article. With the exception of one, it was regurgitated media drivel.

RolandDeschain":16a3dhwc said:

Both sentiments are correct, Sac/Birds. The problem with stats is you can make them say whatever you want.

Click to expand...

Stats don't lie, but statistical analysis often does.

sc85sis":36pups84 said:

Stats don't lie

Click to expand...

Sure, they do.

Scottemojo":2oeq3dnb said:

Anyone who wants to know about the brains behind PFF, read the SB prediction article. With the exception of one, it was regurgitated media drivel.

Click to expand...

https://www.profootballfocus.com/blog/2 ... wl-xlviii/

Wow, brutal. Not so much the predictions, which basically said "too close to call", but some of the supporting arguments. 2013 was a down year for quarterbacks? I want what that guy is smoking. Funny enough, he followed shortly after by saying he expects Russell Wilson will be the NFL's very best QB within a year or so.

sc85sis":22pni57d said:

RolandDeschain":22pni57d said:

Both sentiments are correct, Sac/Birds. The problem with stats is you can make them say whatever you want.

Click to expand...

Stats don't lie, but statistical analysis often does.

Click to expand...

Stats say on a per game basis, Foles was the second best qb in the league.

But I get what your saying. It's often how we interpret stats that gets us in trouble.

.
Thank you. And not to call anyone out or point a finger, but the second best qb in the league assertion highlights the common error in interpretation and conclusion so many of us are prone to make ...... from the referenced conversation >>>>>

Part of the issue I think is that PFF has no control over how people use PFF data. I know initially Neil was dead against even having overall rankings, because people would just look at them 1-50 and say "player x is 20th, player y is 25th, so player x is better".

Click to expand...

Are not these rankings and ratings tied to player combinations within a team specific system rather than attributed to the isolated individual? >>>>>

The issue of context also goes well beyond the micro-level issues Eric identified. For example, Eagles WR Riley Cooper played six games with Mike Vick as his starting quarterback this year. Here were his grades:

-0.6
-0.2
-2.7
-0.1
-1.2
-1.3
---------
-6.1

He also played seven games with Nick Foles as the starter:

1.7
2.0
2.2
1.9
0.5
0.7
2.1
---------
+11.1

I'm not sure I could find a better illustration that what PFF is grading is not "Riley Cooper" but rather "Riley CooperWithVick" and "Riley CooperWithFoles."

Click to expand...

RolandDeschain":3nve1vgc said:

sc85sis":3nve1vgc said:

Stats don't lie

Click to expand...

Sure, they do.

Click to expand...

amill87":3nve1vgc said:

sc85sis":3nve1vgc said:

RolandDeschain":3nve1vgc said:

Both sentiments are correct, Sac/Birds. The problem with stats is you can make them say whatever you want.

Click to expand...

Stats don't lie, but statistical analysis often does.

Click to expand...

Stats say on a per game basis, Foles was the second best qb in the league.

But I get what your saying. It's often how we interpret stats that gets us in trouble.

Click to expand...

Exactly. Stats are simply data. It's what we do with that data that matters.