Bylsma won a franchise-record 252 games behind the bench but failed to produce a bookend to the championship he captured with stars Sidney Crosby and Evgeni Malkin in 2009. The Penguins were just 4-5 in playoff series since raising the Cup, with each loss coming to a lower-seeded team.
Pittsburgh's latest defeat came last month when the Penguins fell to the New York Rangers in seven games in the Eastern Conference semifinals.
The 65-year-old Rutherford played goalie for the Penguins in the 1970s before spending 20 years with the franchise that began as the Hartford Whalers, moved to North Carolina in 1997 and won the Stanley Cup in 2006.
Rutherford stepped aside in April.
AP Sports Writer Joedy McCreary in Raleigh, North Carolina, contributed to this report.