Smiling from ear to ear, he gestured toward the five Stanley Cup banners hanging on the wall at the north end, specifically, the 2016 and 2017 ones.
"You see those," the Penguins general manager said, nodding in their direction. "Those are special. No one had won back-to-back Cups in a long time.
"People ask me if I ever thought about one day getting into the Hockey Hall of Fame. It never even entered my mind until we won those. And even then, I never really expected it to happen."
Until his phone rang on June 25.
The voice on the other end of the line was Hockey Hall of Fame chairman Lanny McDonald, who informed Rutherford that he would be part of the Hall's Class of 2019.
"He deserves it," Penguins captain Sidney Crosby said. "Just really happy for him. Amazing honor. He's been involved in hockey for a long time and played a major role in our recent success. He's won three Cups as a GM and influenced so many people in the game."
Under Rutherford's watch, the Penguins became the first team in the NHL instituted the salary cap in 2005-06 to win consecutive championships, a feat that hadn't been accomplished since the 1996-97 and 1997-98 Detroit Red Wings. He's the first GM to win the Cup with two teams since the 1967 NHL expansion, the first coming with the Carolina Hurricanes in 2006.
"No doubt, you look at that and say, wow," Winnipeg Jets coach Paul Maurice said. "But with Jimmy, it goes well beyond the numbers or the fact that he won three Cups.
"I mean, he's going into the Hall as a builder. Most guys who fit that description go in because they've built winners. Jim did more than that. In Carolina, he literally built a franchise in that market from the ground up, including the building of an actual arena.
"It's been a fascinating road for him to get here."
Rutherford's hockey journey began as a boy who regularly would tear up the kitchen of his house on Lilly Street in Beeton, a town with a population of 700, about 50 miles northwest of Toronto.
He estimates he was about 5 when he developed his own game to hone his goalie skills. He'd drop to his knees and throw a baseball-sized rubber ball as hard as he could at the wall 10 feet away, then try to stop the ricochet from getting past him.
"I'd be in front of the kitchen table, which was the perfect size for a net," he said.
It was a hockey-based ritual that he would indulge in for the next five years, much to the chagrin of his parents, John and Dorthea.
"I'd do it for hours. Every day," Rutherford said. "I used to knock the paint off the walls and the chairs and everything, which my mother wasn't too happy about. Or if they had a late night out the night before, I'd be down there at seven in the morning bouncing the ball while they were trying to sleep upstairs.
"Sometimes I'd find a broken stick and use it. I tore up the kitchen floor that way."
The in-house practice paid dividends.
By his late teens he'd become an accomplished goalie prospect that had drawn the interest of NHL teams. He was selected No. 10 by the Detroit Red Wings in the 1969 NHL Draft.
There was no phone call telling him that he'd be picked. Instead, he and his parents drove to Toronto and waited five hours outside the Globe & Mail newspaper offices for a copy with the draft listing inside.
"The internet was still decades away so that's how I found out," he said
Rutherford played 457 NHL games with Detroit, Pittsburgh, the Toronto Maple Leafs and Los Angeles Kings, going 151-227-59 with a 3.65 goals-against average and 14 shutouts. He retired in 1983 thirsting to stay in the game.
While running a hockey school several months later, Rutherford's work ethic caught the eye of Detroit-based computer magnate Peter Karmanos Jr. He was soon hired as the director of hockey operations for the Compuware Sports Corporation, a multi-tiered organization that included leagues for youngsters all the way up to a Junior 'A' team that competed in the North American Hockey League (NAHL).
What followed was a working relationship between Karmanos and Rutherford that lasted for the next three decades.
Rutherford served as GM for Karmanos-owned Ontario Hockey League teams in Windsor and Detroit/Plymouth. He became GM and part-owner of the Hartford Whalers with the Karmanos-led group that acquired the NHL team in 1994.
"This is the point where Jim's Hall of Fame career as a builder really started," said Maurice, who took over as Whalers coach in 1995.
It would not be easy, especially at the start.
Karmanos can't attend Rutherford's induction Monday in Toronto because of health issues. That won't stop the 76-year-old from being there in spirit.
"I admire him so much," Karmanos said from his Florida home on Saturday. "I mean, the way he kept his focus in those crazy first couple of years in Carolina is amazing."
Following the 1996-97 season, Karmanos moved the Whalers to Raleigh and renamed them the Carolina Hurricanes. Because the arena there had yet to be built, the team played in Greensboro, 70 miles away.
"Like Paul Maurice said, we did have to build from the ground up, and Jim was determined to make the crazy move that I wanted to make, successful," Karmanos said. "It was tough. We didn't have much of a choice."
On Oct. 3, 1997 the first regular-season NHL game in North Carolina drew a crowd of 18,661 to Greensboro Coliseum. Several days later, Karmanos was in his Detroit office when he got a call from Matt Brown, the managing director of the arena.
"He told me we'd only sold 13 tickets for our second game (season tickets excluded)," Karmanos said. "Thirteen! I said, 'Don't worry about it, Matt, everything will be all right."
"Then I hung up the phone and said to myself, 'Oh Man.'"
Karmanos then called Rutherford.
"I told him about the 13 tickets," Karmanos said. "He asked me if I was kidding. When I told him no, I wasn't, his answer was: 'Uh oh, this is going to be bad.'
"I just told him, 'Hang on tight buddy, we'll get through it."
Somehow, they did.
The Hurricanes moved into the new 19,722-seat PNC Arena in Raleigh in 1999 and reached the Stanley Cup Final three years later, losing in five games to the Red Wings. Maurice was fired in 2003 and replaced by Peter Laviolette, who coached the team to a Cup title three years later.
For Rutherford, who ended up leaving the Hurricanes after the 2013-14 season, the Cinderella 2006 Cup title will always hold a special place.
"Think about it," he said. "We moved a major-league sports team from one market to another in probably a six-month period. We did it but it's hard to do. So we end up in Raleigh, without an arena. We're trying to promote the sport of hockey and we're playing 70 miles away.
"To be able to go from that to a Cup, well, man."
Karmanos holds equity stakes in the Hurricanes, having sold the majority of the team to businessman Tom Dundon in December 2017. He was inducted into the Hall of Fame as a builder in 2015.
"The thing is, when I went in, I think Jimmy was more excited than I was. And when he goes in, I think I'll be more excited than he is.
"There isn't anybody that knows how much he's earned that honor more than I."
That Rutherford is still an NHL GM five years after leaving the Hurricanes was not the original game plan.
The blueprint was to retire. At least that's what it was about a week after he'd left the Hurricanes in 2014.
"I was sitting on my porch in Carolina when Dave Morehouse, the CEO of the Penguins, called asking if I could fly to Pittsburgh the next day," Rutherford said.
It took one meeting with Morehouse and team owners Ron Burkle and Hall of Famer Mario Lemieux to sway Rutherford.
"I remember my wife Leslie asking: "We're going to Pittsburgh? Why?" Rutherford recounted. "I said: "Look at that roster."
With players like Crosby, center Evgeni Malkin, defenseman Kris Letang and goalie Marc-Andre Fleury, Rutherford knew there was a window of opportunity to win. He was named Pittsburgh GM on June 6, 2014. The Penguins subsequently acquired forwards Phil Kessel and Patric Hornqvist via trades and replaced coach Mike Johnston with Mike Sullivan on Dec. 12, 2015.
Six months later the Penguins defeated the San Jose Sharks in six games in the Stanley Cup Final, the first of their back-to-back titles.
"Him and I hit it off right away," Sullivan said. "And I think because there's no fluff. We tell each other what we think. We don't have to worry about being politically correct. And no one ever has their guard up. He has courage too. And that impresses me when you're the guy who has to sit in the chair and pull the trigger.
"I'm thrilled for him going into the Hall. Look how hard it is to win in this League. The parity. And to see him win three Cups with two different teams, that's impressive."
For Crosby, the 2016 and 2017 Stanley Cup banners at the practice rink serve as symbols of Rutherford's success.
"He came in and was eager and hungry," Crosby said. "You'd have to ask him, but I don't think he expected to be a general manager for that much longer. The fact that he's been here as long as he has means the team has been winning.
"When you have great people like him that are true to their word, loyal, and treat people right, and you are respected not only in your own dressing room but throughout the League, I think that's the best legacy you can have."
As the clock clicks down to his induction, Rutherford admits all this reminiscing is something he doesn't usually do.
"I've always lived by looking ahead, not back," he said. "Always focus on what's in front of you.
"Here's an example. After we won Game 7 of the 2017 Eastern Conference Final in double overtime against Ottawa, I was driving home when my car got hit by a deer. I just parked the car in the garage when I got home even though there was plenty of damage. I was too excited about going to the Final."
Just like he's excited about going into the Hall.
"Biggest honor of my career," he said. "Not bad for a kid from Beeton."