Atkinson scored in the first period to end an eight-game drought.
"It felt like I hadn't scored a goal in a couple of years," he said. "It's good to contribute. That's my game. It's nice to be a part of it and get back to the way I'm playing."
Nick Foligno scored his first goal in 14 games with 6:33 left for a 4-2 lead, and Tyler Motte scored into an empty net with 50 seconds remaining to make it 5-2.
"I felt like I was doing a lot of good things tonight to find myself in the right position," Foligno said.
Markus Nutivaara scored his first goal of the season, and Sergei Bobrovsky made 24 saves for Columbus (15-7-1).
Tom Pyatt and Mike Hoffman scored for the Senators (8-7-6), who are 0-4-1 in their past five games. Anderson made 27 saves.
"The first 10 minutes of the third period were outstanding," Ottawa coach Guy Boucher said. "The power play looked great, just couldn't score. Their goalie played terrific. The [penalty kill] looked good too. We were right there, 2-2."