Kyle Palmieri, Bo Horvat and Emil Heineman scored, and Ilya Sorokin made 29 saves for the Islanders (2-3-0), who trailed by a goal entering the third period.
"Exciting. It was exciting," New York coach Patrick Roy said. "We had both teams generating offense on both sides of the ice, turnovers. But hey, we find a way to win and that is the bottom line to me. I think guys in this dressing room know that we need to be better on our exits, we need to be better in our D-zone coverage, but you know what? Offensively, we were good."
Tim Stutzle had a goal and an assist, and Shane Pinto, Dylan Cozens and David Perron scored for the Senators (2-4-0). Ullmark made 18 saves.
"Disappointing, for starters," Ottawa coach Travis Green said. "I thought our game got too loose when we had the lead, and a lot of things we took pride in last year where we were good, we weren't good tonight."
Perron's quick shot from the left face-off dot got by Sorokin on the power play to give the Senators a 1-0 lead at 17:01 of the first period. Arthur Kaliyev got his first point with the Senators with the secondary assist.
Pinto made it 2-0 at 5:32 of the second period with a one-timer from the top of the left face-off circle on a delayed penalty. It was his seventh goal in six games this season, and he has scored in five of those games.
"Should've won that one," Pinto said. "There's not much else to say."
Heineman put a rebound into an open net in front to cut it to 2-1 at 6:50.
Horvat tied it 2-2 at 9:57, finishing a passing play with Lee and Ryan Pulock.
Stutzle gave the Senators a 3-2 lead at 16:46 when he took a drop pass from Nick Jensen on the rush and beat Sorokin under the glove with a wrist shot.
"Turnovers cost us," Roy said. "We made some bad decisions, almost, at times, when they were on the rush that we could have played a little different. But we found a way. We scored goals offensively, we battled hard offensively, we got those goals, but what I loved was how resilient we were."