Tanguay converted a pass from Jarome Iginla on an odd-man rush after Columbus nearly scored moments before.
"Actually, it started with Leland," Tanguay said. "I lost my guy in the defensive zone, they made a great play, Leland made a stop. And then [T.J.] Brodie started the play off in the corner, beat one of their guys and then we were off on a 3-on-2.
"Iggy drove to the net and made a great pass to me. I just had to shoot and it was pretty much if I hit the net then I'd get a good chance to score. And the puck went in."
Irving made 26 saves as the starter in place of Kiprusoff, who suffered a lower-body injury during the second period of a 4-1 win at Detroit on Tuesday. Irving, a 24-year-old who was the Flames' first-round pick (No. 28) in the 2006 NHL Draft, won for the second time in his nine NHL appearances. Kiprusoff has played in at least 70 games for seven straight seasons.
"There's nothing better. It's pretty exciting," Irving said. "The boys really came through tonight, they played great. I'm glad they were rewarded with two points."
Calgary won its second straight road game after starting the season 1-3-2, with five home games among those first six.
"The whole team is playing better and better and hopefully we can keep on building confidence and winning more games," Tanguay said.
Columbus tied the game 3-3 when Vaclav Prospal scored with 11 minutes remaining in the third period. The Blue Jackets ended regulation and started overtime on a power play but could not come through.
They have lost eight of 10 since winning the opener.
"We have to figure it out," Columbus coach Todd Richards said. "For 20 minutes it's good and then we lose our game. And when it's gone, it's gone. It's not one shift or two shifts; it's for an extended period of time."
Lee Stempniak's power-play goal, with an assist from Irving, put the Flames ahead 3-2 at 8:33 of the third. Stempniak raced up the right-wing boards and fired a slap shot past Columbus goalie Sergei Bobrovsky.
The Flames tied the game 2-2 when Mikael Backlund sent a wrist shot past Bobrovsky at 13:06 of the second period. Flames rookie Roman Cervenka scored his first NHL goal when he deflected Dennis Wideman's point shot at 4:19 of the first.
"A gutsy effort, that's the first thing I'm going to say," Flames coach Bob Hartley said. "We scored the first goal and they came right back. We didn't really get the first period that we wanted, but the second and third periods, we owned the puck. We made some great plays, like a great penalty kill at the end of (regulation). And then we got the winner with Tang."
The Blue Jackets responded 25 seconds after Cervenka's goal when Matt Calvert sped into the left wing circle and put a wrist shot past Irving's glove side.
Columbus grabbed a 2-1 lead after some nifty triangular passing on a power play. Derrick Brassard sent a pass from right wing to Jack Johnson at the blue line. Johnson sent it back to Brassard, who hurried it to Mark Letestu on the left wing. Letestu's one-timer found a near-empty net at 9:37.
Material from team media and wire services were used in this report.