Ullmark, playing in his first game since Dec. 22 after missing the previous 18 because of a back injury, made 34 saves.
Brandon Hagel broke a tie early in the third period, and Nikita Kucherov also scored for the Lightning (28-20-4), who had lost three of four. Victor Hedman had two assists, and Andrei Vasilevskiy made 25 saves.
"That was a grind," Tampa Bay coach Jon Cooper said. "If we want to get ourselves back to where we were to be, these are the games you have to win. Not only win them, but make sure we weren't sharing points, and the boys did it."
Claude Giroux and Drake Batherson scored for the Senators (29-21-4), who had won seven of nine. Thomas Chabot had two assists.
"I thought early on our execution wasn't great," Ottawa defenseman Nick Jensen said. "It wasn't snappy and tape-to-tape as it felt the prior stretch of games. There were times tonight where we gave ourselves a chance to win, but we left it up to chance."
Michael Amadio gave the Senators a 1-0 lead with a power-play goal at 14:09 of the first period when he knocked in a loose puck on a rebound.
Luke Glendening tied it 1-1 at 1:44 of the second period with a redirect from inside the right circle off a shot from Emil Lilleberg.
"It wasn't always pretty, but we grinded it out," Glendening said. "It's not been easy this year, but I'm happy to get one. This was a big game to get two points in. We've got another one coming up and this whole week is against our division. So we need to keep working."
Kucherov’s power-play goal put Tampa Bay ahead 2-1 at 7:38, a one-timer from the left circle off a pass from Hedman.
"We had our looks and their goalie kept them out, it was tough being down 1-0 for a bit," Cooper said. "But it was big for us to get one and then we kept building off that."
Giroux tied it 2-2 at 9:30 with a wrist shot from above the hash marks with Cole Reinhardt providing a screen.
Hagel gave the Lightning a 3-2 lead at 1:56 of the third period with a wrist shot from the high slot that beat Ullmark glove side.
"We started the third period with 10 forwards, so I think there was a lot of urgency and a lot of adversity as well," Hagel said, "But we just kept pushing."