Blue Jackets goaltender Curtis McElhinney, who played seven games for the Senators in 2010-11, made 34 saves, and Nick Foligno, the Senators' first-round pick (No. 28) in the 2006 NHL Draft, scored the winning goal.
Foligno, traded to the Blue Jackets for defenseman Marc Methot in 2012, scored his 21st goal of the season, a career high for the 27-year-old.
"I finally scored on a breakaway this year, so that was good," Foligno said.
He muscled his way past Senators defenseman Cody Ceci and forward Mark Stone to break in and put a backhand through the pads of Senators goaltender Robin Lehner to make it 2-0 at 8:51 of the third period.
"It's meaningful (to score against the Senators) for sure. It's a team you used to play for and you get up for those games. You want to do well against them," Foligno said.
The Blue Jackets (23-25-3) won back-to-back games, following a 7-1 win against the St. Louis Blues on Friday.
Columbus passed Ottawa into 13th place in the Eastern Conference. The Blue Jackets are 14 points out of the second wild card into the Stanley Cup Playoffs from the Eastern Conference but haven't given up hope, Foligno said.
"We just have a mentality right now where we have to do whatever it takes to win whether the game is going our way or not," he said. "It says a lot about the guys and our commitment to winning right now and doing the right things.
"We're not in a position to feel sorry for ourselves. We just [have] to win. That's pretty much what it comes down to."
McElhinney is getting an opportunity with No. 1 goaltender Sergei Bobrovsky out with a groin injury. McElhinney made his best save with the score 2-1 and 3:55 left in the third period when he got the paddle of his stick on a shot by Stone off a scramble.
"I've gotten into a pretty good groove right now and I think the guys are responding to me in net," McElhinney said. "It was nice to build on the game from last night, just a very important win for us on the road here."
The Senators (20-22-9) have three regulation losses in a row for the first time this season and are 9-11-4 under coach Dave Cameron, who took over from Paul MacLean on Dec. 8.
They have been held to one goal in each of those three losses.
"There were chances for our line to score that need to go in the back of the net, especially during this dry spell we're going through. Whether it's me or anybody … I just need to put the puck in the back of the net," Stone said.
"We're falling out of it every game right now. We lose three in a row and teams are winning. It's frustrating only scoring one goal a game. We're getting chances … we can't just be getting chances right now. It's time to start scoring on them."
The Blue Jackets took a 1-0 lead 4:13 into the third period on a goal by Cam Atkinson, who was shuffled off the top line with Scott Hartnell and Ryan Johansen in the second period but was reunited with them for the third.
Ottawa's Erik Condra scored at 10:22 to make it 2-1.
Hartnell scored an empty-net goal with 1:16 left, and Mark Letestu shot over Lehner's blocker with 12.1 seconds remaining.
"It's tough for us. There's a lot of odd-man rushes," Lehner said. "We're talking about it. The first two goals, in a 0-0 game, shouldn't happen.
"And if you look at these teams that make the playoffs every year, it doesn't happen. It does not happen. They play a simpler game in the third and they make sure they have three guys at home all the time. We've got to get better at all these small detail things."
Lehner, who made 29 saves, started his fifth straight game with No. 1 goaltender Craig Anderson out with a hand injury. Cameron said Anderson's blocker hand had improved and there was a good chance Anderson could be the backup Tuesday when the Senators visit the Buffalo Sabres.
Columbus entered the game Saturday with an NHL-leading 317 man-games lost.
"There were some things I didn't like, but we found a way to win a hockey game," Columbus coach Todd Richards said. "[McElhinney] made some great saves. We left him exposed a few times and not covering up in our zone, a couple of penalties where they generated some momentum and some opportunities, but he was excellent."
The Blue Jackets host the Los Angeles Kings on Monday.