Gostisbehere scored an extra-attacker goal at 17:56 of the third period to cut the deficit to 3-2. Lundqvist had to make two saves seconds before the final buzzer to seal the win.
The Flyers outshot the Rangers 42-23, including 21-8 in the third period.
"Their goalie played a good game, made some key saves for them, but I think we stuck with it and at the end we were right there to tie it up," Flyers captain Claude Giroux said. "I think we played a solid game. The last few games, they've been really good games, just we can't find a way to get the win. I think we've got to stay positive here."
Stepan and Puempel scored 1:34 apart in the first period to give the Rangers a 2-0 lead. Hayes scored in the second to make it 3-0. Stepan and Hayes scored off of defensive zone turnovers by the Flyers.
"Critical errors," Philadelphia coach Dave Hakstol said.
J.T. Miller knocked down and stole Scott Laughton's pass behind the goal line and found Stepan cutting into the slot for a one-timer at 13:16 of the first. Nick Holden intercepted Mark Streit's errant pass at the right point and set up Hayes for a redirection goal at 12:33 of the second.
"The little details in the game we need to pay attention to, but at the same time I think we're playing a lot better than the start of the year and we're not doing as many turnovers as the start of the year," Giroux said. "We play like we played [Friday], we're going to win a lot of games."
Puempel scored off of a 2-on-1 with Jimmy Vesey at 14:50 of the first period. He had no points in 13 games with Ottawa before he took Mika Zibanejad's spot on the Rangers' active roster.
Zibanejad is out with a broken left fibula.
"Confident kid," Rangers defenseman Marc Staal said of Puempel. "Picked his spot and got it. That's huge when you have injuries to have guys come in and contribute. Big for us."