Analytics, not statistics, driving NHL evolution

In the fall of 1917, the Montreal Wanderers and Toronto Arenas played the first NHL game; with the sport in its infancy, forward passes weren't yet legal.

Every June, executives and columnists wax poetic about the traditions that make hockey what it is; the handshake line, the passion and pain threshold of the players, the Stanley Cup.

What has truly made hockey into the exhilarating sport and thriving industry it is today, though, has been its ability to adapt, fusing playmaking concepts from European soccer with the rugged athleticism of rugby and the structure and tactical teachings of lacrosse.

The evolution continued this summer with the addition of analytical specialists in the hockey operations departments of several teams, highlighted by Kyle Dubas being named assistant general manager of the Toronto Maple Leafs.

Such developments suggest hockey may be in the process of learning its most important lesson, one baseball has only recently come to terms with itself:

"If we weren't already doing it this way, is this the way we would start?"

That's the question Paul DePodesta asked Billy Beane, the architect of the Oakland A's roster, when they met. Was Beane's process guided by principles of accuracy and efficiency or by convention and tradition?

For Beane, who would soon face the loss of three big-time free agents, the question made an impression. Less than three years after he and DePodesta joined forces, the 2002 A's won 20 games in a row, an American League record. Despite having the lowest payroll in baseball, Oakland tied for the best record in the majors.

Beane and DePodesta revolutionized baseball with those accomplishments. Now, more than a decade later, hockey is facing a similar tipping point.

SUNDAY LONG READ

Adult league thriving in San Jose

By Corey Masisak | NHL.com Staff Writer
San Jose has become one of the strongest hockey towns in the NHL, and the massive adult league has played an integral part. READ MORE ›

• Columbus has become first-rate hockey town
• Oral history of the 2010 World Junior final
• Avalanche's Gabriel Landeskog a born leader

MORE SUNDAY LONG READ STORIES ›

There has been an attempt to distance the sport from the "Moneyball philosophy," which was laid out in the book, "Moneyball: The Art of Winning an Unfair Game," by Michael Lewis.

Hockey insiders suggest statistics only lend themselves to slower games which stop and start following every play.

But "Moneyball" wasn't about advanced statistics, or even really about money; it was about thinking differently, asking questions, and never being satisfied with doing things the way they have always been done.

"In such a tight market for talent, you needed to look beyond the conventional means of thinking," hockey executive Mike Gillis told Bruce Dowbiggin in "Ice Storm: The Rise and fall of the Greatest Vancouver Canucks Team Ever."

Gillis was general manager of the Cancuks when they reached the 2011 Stanley Cup Final.

"In hockey, people have not wanted to go beyond their experience to find new solutions," he said. "Some hockey people dismissed 'Moneyball' because Oakland never won using it. But the A's were a small-market [team], and when they taught the big guys how it works, their advantage was gone. The Red Sox did adopt those principles and won two World Series."

The principles of "Moneyball" apply to hockey as much as any other sport, and in order to claim an understanding of the game, executives must be willing to look at it from every angle. That is where analytics come into play. The willingness to discard preconceived notions which don't stand up to strict scrutiny are already playing a role in separating the most successful teams from the rest.

The first step in embracing analytics is to understand them. Only with that base can the misconceptions and biases that skew hockey decision-making at the highest level be appreciated.

Last season during a "Hockey Night in Canada" broadcast, when the Edmonton Oilers lost to Vancouver 6-2, analyst Glenn Healy commented on the Oilers' performance in a manner which echoes how many sports fans feel about the analytics movement.

"When you look at the stats, every stat was in Edmonton's favor," said Healy, who played goaltender in the NHL for more than a decade. "Faceoffs, they were better in faceoffs. They were better in hits. They had more blocked shots. But they weren't even in the game."

There seems to be an idea among those who oppose the use of analytics that Healy's words present a flaw in the practice, that those who believe in analytics think adding up the numbers in a box score should produce the game's winner like a seventh-grade math quiz.

In this instance, Healy was arguing against a caricature. There is a crucial difference between statistics and analytics.

A statistic is simply a piece of data. The San Jose Sharks defeating the Los Angeles Kings 2-1, that's a statistic. Tyler Seguin scoring his 20th goal to put the Dallas Stars up 4-3, that's a statistic. Anything can be a statistic, and that is why, as they say, statistics can be manipulated to fit any narrative.

Analytics are distinct, the study of statistics to find meaning. Analysts look for reproducible patterns in large samples which illuminate important lessons about the game.

"If [a metric] is consistent over time, then it's a reliable gauge of skill," said A.C. Thomas, co-founder of the popular advanced-stats website War-On-Ice.

Any analyst would say a player's performance on Thursdays in March or a team's home/road power-play splits are largely meaningless. The misuse of statistics, therefore, is poor analytics, just as taking a player's quote out of context is poor journalism.

"For an analyst to suggest a metric has value, we need to be able to explain how or why it relates to outcomes we desire, most generally goals for or against," Sportsnet.ca analytics writer Stephen Burtch said. "Once we can identify how it relates to winning, we need to be able to show that it describes something meaningful, that it relates relatively strongly to goals/wins."

Hits are a prominent example of a dividing-line statistic. Purists cite it as a difference-maker; analysts dispute its importance. To evaluate their claims, analysts test every number under a high degree of scrutiny. What is a hit? How is it tracked? Have more hits historically led to more wins? Are there problems with the statistic?

As it turns out, hits are an unreliable metric. The team that has the puck the most has the fewest opportunities to record hits. Ultimately, there is very little correlation between hitting and winning. A hit is an example of a play that can make an impact but also of a statistic that analysts have shown to be misleading.

Analysts and non-analysts are on the same side in the struggle to counteract the abuse of statistics. There are a lot of numbers out there, but if a statistic isn't a true indicator of performance or in large samples tends to predict future success, it can lead to misinformation. Without thinking analytically, it is nearly impossible to determine which numbers are important and which are noise.

Trusting analytics is difficult without a fundamental understanding of the metaphysics involved in sport.

Most conventional analysis treats hockey as a solely deterministic entity: The conditions which lead to every occurrence in a game, deterministic theory dictates, couldn't have resulted in any other outcome. In other words, a game's result is the product of physical manifestations that are independent of past or future games.

Under a strictly deterministic lens, winning is the only metric that matters because every game is a clean slate and, under fresh conditions, anybody can defeat anybody and anything is possible. As commentators often state at the beginning of an important power play, "Throw the percentages out the window here."

Deterministic logic does hold up in a hockey game. Hockey is deterministic; every game is a fresh opportunity for success in an ever-changing environment. But the sport is not exclusively deterministic, and that is the key behind much of the disagreement between analysts and purists, between those who have played the game at a high level and those whose expertise rests in quantitative fields.

Consider a game of Texas Hold 'em, a form of poker where players combine privately held and publicly shared cards to form the best possible hand, placing bets along the way. On the surface, Hold 'em seems like a very different game from hockey because it's all about odds. A player estimates his chance of winning a particular hand based on the information available and bets accordingly.

The game is probabilistic in nature; in the long term, those probabilities bear out. Even if a player sees his full house defeated by a straight flush in one instance, he should bet on a full house winning the next time around. It would seem as though hockey and poker are metaphysically opposed -- one deterministic, the other probabilistic -- but there is a problem with that view.

Poker is an example of a game that is deterministic and probabilistic. The result of each hand comes from the physical process of shuffling, but because none of the players know the result of that process beforehand, the game is one of probabilities. Hockey is very much the same way.

When Washington Capitals forward Alex Ovechkin decides to wind up for his patented one-timer from the left circle on the power play or to pass to the slot, he has to take into account a number of factors. The timing of his windup, the direction of his swing, the weight of the pass, and the texture of the ice decide in absolute terms where a shot would end up.

If the puck doesn't go in, critics would argue it was because he did something wrong. If he opts to pass and the play doesn't connect, they would ask why didn't he take the shot? But whether the effort went bar-down and in or struck the wrong side of the post would largely be a result of variance, or as it's called in metaphysics, chaos.

Because Ovechkin doesn't know how variance will impact his shot (how the deck will be shuffled) he has to trust in probabilities; he has to go with the play he thinks gives the Capitals the best chance to score.

During the course of thousands of repetitions, those probabilities will remain constant. It is why the best scorers during long periods tend to stay constant, but why in individual games or seasons there may be inexplicable hot and cold streaks.

Players will make the right decision a certain percent of the time based on their hockey IQ, and their physical makeup will allow them to deke, pass, check, clear, deflect and score a certain fraction of the time. But the determination of when they are able to do those things successfully is largely out of their control.

So, yes, each power play is a fresh chance to do the little things right and score a critical goal, but through enough of those chances, the true talent of the group will be revealed, whatever it may be. So, no, someone can't throw the percentages entirely out the window.

Fans generally think of luck in its most blatant form: a strange bounce, a bad call, a broken stick. Variance, though, affects the result of a game in more critical ways, and that is why winning is not the only metric that matters. A good process, across a large number of games, will lead to the most wins. Though winning a game in the present is nice, having sustainable success -- whether that means making the Stanley Cup Playoffs, winning four playoff series in a row, or ascending to dynasty status -- is what every team and player is seeking. Those successes are dictated by an accumulation of probabilities; the teams most likely to win the most games will eventually win the most games.

The aspect of variance that is most difficult to understand is that outcomes are not evenly distributed.

For instance, Ottawa Senators forward Bobby Ryan was left off the U.S. Olympic team last year, and a story from ESPN's Scott Burnside revealed that Ryan was disparaged in the process, with Calgary Flames executive Brian Burke metaphorically painting him as someone who "can't spell intense."

In the five weeks following the decision leading up to the 2014 Sochi Olympics, Ryan had seven points in 17 games, causing commenters to opine that Burke was right and Ryan missed an opportunity to prove the executive wrong.

It certainly wasn't the first time a player was criticized for a scoring drought, but it echoed a failure by many, from fans to executives, to understand variance and how it manifests itself in hockey and more broadly.

When Apple's iPod came out with the option to shuffle songs, customers complained the software must be broken because they kept getting two songs by the same artist back to back, or several in a row of the same genre. They didn't think the built-in randomness was "random" enough.

In reality, randomness "creates counterintuitively dense clusters," and the mind is programmed to read patterns even when none exist, as Alex Bellos wrote for the Daily Mail. According to Nobel Prize-winning psychologist Daniel Kahneman, we "understand sentences by trying to make them true." If there is the potential for a causal connection, we naturally cling to that explanation.

In an experiment done at a university in Barcelona, students were asked to predict a sequence of five coin tosses. In the aftermath, one student was identified as having predicted the most tosses correctly, and one the least.

Audience members were told they would either be betting on the student who had been the least successful for a second round of flipping, or they could pay to switch to the student who had done the best. Anonymously, 82 percent of the audience paid to switch. In a simple game that was so clearly decided by chance, the audience was fooled by randomness.

In the end, Barcelona students lost money betting on randomness disguised as reproducible success. Steve Jobs had to change the iPod shuffle feature to manufacture false randomness in a way that would appease his customers. Bobby Ryan had to address the media to make excuses for a sudden inability to score that was more likely the result of variance than of criticism or skill level.

Hockey is a fast, fluid game of small margins. Even across samples as large as a season or two, the bounces can go one team's way more than another's. One player may see his shots tip in off of a stick or the post, or may benefit from fortunate screens, and another player may not. Variance is difficult to catch with the naked eye, because a shot off the crossbar and in looks like something a player could repeat every time if focused.

Analytics have a ways to go in differentiating variance from talent, but an inability to recognize clustered randomness in extreme cases has led to some of the biggest management mistakes in hockey history. Being wary of chaotic concepts can give a team numerous wins a season. Falling prey to them can waste millions of dollars.

From the time Bill James published his first "Baseball Abstract" in 1977, purists have accused analysts of not watching the games, suggesting that they treat the spreadsheet as a manuscript and the pleasure derived from sports is drawn only from calculations. This, by and large, couldn't be further from the truth.

Massive amounts of the most cutting-edge analysis in hockey have been recorded by watching game film repeatedly and tracking metrics in an attempt to make sense of what goes on at ice level. But there are also reasons only watching the games isn't sufficient, and why conventional scouting alone is flawed.

The first problem is that nobody can watch every game. Even with an implausibly packed schedule, a scout could see 300 or so games a year, split between teams and leagues.

A general manager or coach will watch mostly his team's games, seeing snippets of other players as a result. Analytics allow the ability to attain a level of insight into a team or player that can't be gleaned from sporadic viewings. The numbers can see every game and provide a better idea of how a certain team or player is doing than by going off past viewings, hearsay, the odd shift, or, worst of all, reputation.

The more statistics that are available, and the better those statistics are at measuring value, the better the insight. That's why some advanced statistics are better than goals, assists, plus-minus, etc. They get you closer to an all-encompassing understanding of a game in a sport that tends to be dictated by selective viewing. Advanced statistics provide a more precise understanding of the broader game.

Beane was asked a question about trusting the eye test and he responded, "The idea that I trust my eyes more than the stats, I don't buy that because I've seen magicians pull rabbits out of hats and I just know that rabbit's not in there."

Sports are obviously different from magic tricks, because players aren't intending to fool the viewer at every turn, but there is so much happening during any play that a similar effect is present.

"The eye test isn't sufficient for analyzing a fast-moving game like hockey," Burtch said. "There's just far too much going on at far too fast a pace for any one individual to easily track and store the information they're seeing via memory."

Hockey's pace makes numerical analysis and eyeball scouting more difficult when compared to baseball, but it doesn't make either method less valid. It just means each needs to be handled with more scrutiny.

Watching the game can give us information but it can't get us all the way there. Our preconceived notions, our biases, and our inability to capture everything that happens make the eye test imperfect as a means of evaluating performance. There needs to be an objective layer. That is what analytics provide.

The top three Corsi, or SAT, players during the 2013-14 season were all Los Angeles Kings, who won the Stanley Cup championship. (Photo: Dave Sandford/NHLI)

Around the time Beane's Oakland team was becoming a contender, hockey's wave of analytics was beginning to take shape in the comments section of blogs like Irreverent Oiler Fans. Impassioned fans who understood the analytic principles searched for areas of the game that most strongly correlated to winning, attempting to syphon the variance or chaos that clouded decision-making and analysis at the highest levels.

They found that shot-attempt differentials, with a sample size that accumulated far quicker than a plus/minus of goals or regular shots, were able to do that. This led to the creation of Corsi, a metric that has infiltrated the mainstream media and front offices, and on NHL.com is called Shot Attempts after the site's statistical pages were recently overhauled and expanded through a partnership with the software corporation SAP.

They discovered that uncharacteristically high or low shooting and save percentages, even during a full season, aren't sustainable for players or teams. Therefore, adding them together and comparing the number to past team performance or league averages can work as a decent proxy for variance. This became known as PDO, which is called SPSV% on NHL.com.

Several of those pioneers, including Tim Barnes (Capitals), Tyler Dellow (Oilers) and Sunny Mehta (New Jersey Devils), now work for NHL teams. They find ways to account for the parts of the game that puck-possession metrics like Shot Attempts miss, and to reconcile the newfound importance of controlled-zone entries and deployment optimization with challenges including imperfect data recording, changing environments, and noise.

For some teams, that has meant sweeping systematic changes. The Minnesota Wild have gone from a neutral-zone-trapping, dump-and-chase behemoth to promoting controlled entries and faster play, leading to vastly improved even-strength numbers recently undone by disappointing goaltending and special teams.

For others it has meant an evolving lineup structure. The Maple Leafs waived physical players Colton Orr and Frazier McLaren and signed undervalued targets Daniel Winnik and Mike Santorelli this summer. (Santorelli was traded to the Nashville Predators last week.)

Ultimately, judging analytics on the short-term results of recent converts is an affront to the analytic process itself. Analytics aren't any more of a magic bullet than a new general manager or a first-round pick in the NHL Draft.

"Because numbers are involved, there is the perception that statistical predictions have to achieve perfect accuracy," said Rob Vollman, a columnist for ESPN Insider and author of "Hockey Abstract." "But the results [only] need to be compared with traditional analysis."

In other words, analysts aren't trying to take the unpredictability out of sport but are attempting to improve on evaluative practices.

Adopting an analytical mindset in a salary-cap world can't overcome the challenges created by fielding a bad team, but by extracting a fraction more out of every player and situation through optimal deployment, and then working to improve by acquiring undervalued assets and avoiding costly mistakes, are the ways a team can pave the way for a successful future.

"[Analytics] are about effective decision-making with a high reward," said Thomas, the War-On-Ice founder. "If hiring someone for $100,000 right now can get you a free-agent-value savings of a million dollars and more flexibility under the cap, they've paid for themselves right there."

Dallas Stars general manager Jim Nill told Travis Yost of TSN.ca this summer, "We are all trying to get 3-5 percent better. It's a cap world and we are limited. We are always looking for the next thing."

A team can win without analytics, and many will lose employing them, but the additional information and a scrutinized process provide a greater chance at success. As the metrics improve and attitudes shift, this will become more apparent.

Analytics have led hockey executives, journalists and fans to pose the same question DePodesta asked Beane prior to joining him in Oakland, and they have precipitated a massive shift in the way teams do business.

Rather than exclusively trusting the eye test, condemning players for misfortune in small samples, or labeling players as lazy or enigmatic based on reputation or hearsay, analytics has provided the opportunity to scrutinize decision-making and avoid those characterizations. Dismissing analytics means settling for an obsolete method of evaluation and using an inefficient business practice.

"Hockey analytics are simply the objective analysis of hockey," Vollman said. "Teams are bringing in outsiders to challenge [conventional wisdom], and there's an opportunity to gain an edge."

NHL has experimented with SportVision to work on advanced player tracking with microchips placed in pucks and jerseys, and that could lead to an entire new world of data to be analyzed.

The so-called "Summer of Analytics" doesn't represent the beginning of analytical adoption in hockey; analytics have been in use for some time. It proves, in no uncertain terms, that Beane's progressive thinking is flowing into hockey.

The inclusion of enhanced statistics on NHL.com and a partnership with SAP that will continue to expand and is expected to produce further innovations could help. The NHL also has experimented with SportVision to work on advanced player tracking with microchips placed in pucks and jerseys, and that could lead to an entire new world of data to be analyzed.

If a 28-year old former sports-management major, a university statistics professor, and a practicing lawyer -- none having played professional hockey -- can successfully replace longtime executives, then the door is open for other innovative thinkers to follow them into positions of influence.

Executives, journalists and fans will continue to revere the tradition behind the Stanley Cup, but the future will bring new challenges to teams hoping to win it, and that's good. The spirit of evolution that has crafted the hockey we enjoy remains alive, well and back on track.