Assistant general manager Patrik Allvin will serve as GM while the Penguins search for a replacement.
"It has been a great honor to serve as general manager of the Penguins, and to hang two more Stanley Cup banners at PPG Paints Arena," Rutherford said in a statement. "There always has been so much support from everyone involved with the Penguins, both on the hockey and business staffs, and, of course, from a special group of players led by Sidney Crosby. The fans here have been tremendous to me and my family. I know it's a little unusual to have this happen during a season, but just felt this was the right time to step away."
The 71-year-old did not rule out working in the NHL again.
"It's just time now to take it easy," Rutherford told The Hockey News. "My contract is up in June, and when that's up, I may retire or I may consider jumping back in."
A specific reason for the resignation was not provided, and Rutherford said it was not related to his health.
"(The pandemic) hasn't worn me down, and, in fact, it's been better for me," Rutherford told The Hockey News. "I've lost weight. I've exercised more. I probably feel better than I have in a long time."
Rutherford and Penguins president and CEO David Morehouse spoke Tuesday night, at which point, Rutherford had his mind made up, Morehouse said. That was still the case when they spoke again Wednesday morning.
"It's a personal decision that Jim made," Morehouse said. "I think that says what you need to know. It's personal. Shocking? I'm not sure 'shocking' is the right word. I think he's accomplished a lot here. I think he still has things he wants to do."
Video: Discussing Jim Rutherford resigning as Penguins GM
Rutherford was named general manager on June 6, 2014, replacing Ray Shero. The Penguins made the Stanley Cup Playoffs in each of Rutherford's six seasons and won the Stanley Cup in 2016 and 2017.
"It's a good organization and I was treated well here," Rutherford told The Hockey News. "When I first came here, I was hoping to have a chance to win a Cup in a couple of years and ended up winning two and ended up staying almost seven years."
Shortly after hiring Rutherford, the Penguins acquired forward Patric Hornqvist in a trade from the Nashville Predators on June 28, 2014. That was the first of several trades the Penguins made leading to back-to-back championships.
"I want to personally thank Jim Rutherford for everything he's done for the Pittsburgh Penguins," Morehouse said. "He won two Cups here, two consecutive Cups, a feat I don't think will be replicated anytime soon in this salary cap era. He made the playoffs every year he's been here. He's in the Hockey Hall of Fame. He's a fabric of the community of Pittsburgh already.
"The greatest compliment a Pittsburgher can pay someone from outside of Pittsburgh is he's a true Pittsburgher. He says what he means, and he means what he says."
Under Rutherford, Pittsburgh hired coach Mike Sullivan on Dec. 12, 2015, and added forwards Phil Kessel, Carl Hagelin and Nick Bonino, and defensemen Justin Schultz, Trevor Daley and Ian Cole, among others.
Forwards Jake Guentzel and Bryan Rust, and defenseman Brian Dumoulin, became regulars on the Penguins roster during Rutherford's tenure.
In searching for the next GM, Morehouse said the Penguins will look for someone who shares Rutherford's outlook on building a roster with the main criteria being a desire to win the Stanley Cup in the immediate future.
Since winning the Cup in 2017, the Penguins (4-2-1 this season) have not advanced past the second round of the playoffs. That includes getting swept by the New York Islanders in the first round in 2019 and losing in four games to the Montreal Canadiens in the best-of-5 Stanley Cup Qualifiers last season.
"We're not in a rebuilding mode," Morehouse said. "We're in a win-now mode, and we're going to continue to be in that mode until we're in a rebuilding mode. But for right now, we're looking for someone who can come in and continue having us work toward winning another Cup."
Rutherford was elected to the Hockey Hall of Fame in the Builders category on June 25, 2019. He also was GM of the Carolina Hurricanes when they won the Stanley Cup in 2006.
Rutherford was a goalie for 13 NHL seasons from 1970-83 with the Penguins, Detroit Red Wings, Toronto Maple Leafs and Los Angeles Kings.
Allvin, who was named assistant GM in November, is in his 15th season with the Penguins. The 46-year-old was director of amateur scouting from 2017-20.
Morehouse said Allvin will be a candidate to become GM and will not be held back when it comes to shaping the roster for the remainder of the season.
"He's also going to have (Penguins co-owner) Mario Lemieux as a backup," Morehouse said. "So I think we're OK in that area. Patrik will have full authority, but Mario's still, along with Ron Burkle, the general partners of the ownership team here. Mario's had some experience around this stuff. So we think we'll be OK."