Kevin Hayes put the Rangers ahead for the first time when he picked the pocket of David Savard at center ice and broke in alone and beat Bobrovsky five-hole to put the Rangers up 2-1 at 4:49 of the third period.
"I kind of cheated and thought he was going to go D-to-D so I just guessed and luckily he did," said Hayes, who had an assist on Girardi's goal in the second period. "Got a breakaway, saw the five-hole open up and put it in."
But Columbus would answer on a goal by Nick Foligno with 8:07 remaining in the third, before Vesey's game winner less than two minutes later.
The 23-year-old rookie began the night on the team's fourth line with Oscar Lindberg and Jesper Fast, but head coach Alain Vigneault made the move to put Vesey with Rick Nash and Mika Zibanejad midway through, and it was a decision that proved fruitful.
Raanta started for the first time since Jan. 14 in Montreal, when he left the game due to an injury. He had made just one relief appearance in that time.
"Really good feeling. We have been working hard in practice and when Hanky is playing that level, it's tough to get games there," Raanta said. "But it was really fun to get there and just play. It's been a long time since I've played last time. I just wanted to get that good feeling going."