Viktor Kozlov, who earlier had a goal, circled around the Rangers net to feed Rafalski for the winning shot from the left circle.
Alexander Mogilny also scored for the Devils and Martin Brodeur made 33 saves.
Jason Ward and Martin Rucinsky scored for the Rangers. Henrik Lundqvist made 25 saves in his NHL debut.
Trailing 2-1 in the third period, Mogilny tied the game with a power-play goal at 2:14 when his shot from the slot trickled past Lundqvist.
New York had taken the lead in the second period on Rucinsky's power-play goal at 8:32. The Rangers dominated the first two periods, outshooting the Devils 26-8.
In the first period, Ward gave the Rangers a 1-0 lead with an unassisted goal at 4:57. Ward knocked the puck away from Brodeur behind the New Jersey net and scored before the Devils goalie could recover. A video replay confirmed that the puck crossed the goal before Brodeur swatted it away.
The Devils tied the score at 1 on their only shot of the period at 15:39. Kozlov got the goal with a deflection off Rafalski's blueline shot.
|Jason Ward (1) Wrap-around - ASST: NONE|
1 - 0 NYR
|Viktor Kozlov (1) Tip-in - ASST: Brian Rafalski (1), Zach Parise (2)|
1 - 1 Tie
|PPG - Martin Rucinsky (2) Wrist shot - ASST: Michael Nylander (3), Martin Straka (4)|
2 - 1 NYR
|PPG - Alexander Mogilny (2) Wrist shot - ASST: Scott Gomez (2), Brian Rafalski (2)|
2 - 2 Tie
|Brian Rafalski (1) Wrist shot - ASST: Viktor Kozlov (2), Paul Martin (4)|
3 - 2 NJD
|Brian Gionta Cross checking against Darius Kasparaitis|
|Darius Kasparaitis Diving against Brian Gionta|
|Brian Rafalski Delaying Game-Puck over glass|
|Alexander Mogilny Slashing|
|Fedor Tyutin Hooking|
|Jamie Langenbrunner Hooking|
|John Madden Hooking against Martin Rucinsky|
|Jamie Langenbrunner Unsportsmanlike conduct|
|John Madden Hooking against Jaromir Jagr|
|Marek Malik Hooking against Grant Marshall|
|Michal Rozsival Interference against Zach Parise|
|Daniel Mcgillis Cross checking against Dominic Moore|
|Ryan Hollweg Boarding against Paul Martin|
|Fedor Tyutin Delaying Game-Puck over glass|
|Jamie Langenbrunner Hooking against Jaromir Jagr|