Price made 30 saves for the Canadiens, two nights after allowing all six goals in Ottawa's 6-1 victory in Game 3.
He was injured on Conacher's tying goal late in the third by Budaj.
Game 5 goes Thursday night in Montreal in a series that has seen a little bit of everything through four games.
Down 2-0 nothing in the third period and with little going their way offensively, the Senators finally beat Price when Zibanejad's goal at 11:55 from in close off a pass from Chris Neil stood after video review.
Ottawa continued to push and tied it with 22.6 seconds left in regulation when Conacher scored off a scramble in front with Anderson on the bench to send the crowd of 20,500 at Scotiabank Place into a frenzy.
After carrying the play for long stretches of the first period, the Canadiens were finally rewarded in the second. Subban, who took 25 minutes in penalties in Sunday's fight-filled affair and was booed every time he touched the puck, opened the scoring at 2:52 after taking a feed from Tomas Plekanec off the rush and roofing a shot past Anderson.
The flashy defenceman celebrated by mimicking a high-five to a Canadiens fan seated by the glass before being mobbed by his teammates.
Galchenyuk then made it 2-0 at 3:04, again beating Anderson up high, much to the delight of the large contingent of Montreal fans in attendance.
The Ottawa goalie made a big save to keep the deficit at two a couple of minutes later with a big save from in close on pinching Montreal defenceman Andrei Markov.
Senators defenceman Eric Gryba returned to the lineup for Ottawa after serving a two-game suspension for his hit on Canadiens forward Lars Eller. Gryba caught Eller with his head down at the Montreal blue-line just as he received a pass in Game 1, sending the Habs forward flying.
Despite a raucous crowd that sensed the Canadiens were ripe for the taking, it was Montreal that came out the more determined of the two teams early.
Anderson held the fort as Ottawa didn't record a shot until after the eight-minute mark of the first period, an effort off the rush from Game 3 hero Jean-Gabriel Pageau.
The Canadiens said after the morning skate that they wouldn't be looking for retribution on Gryba, but Montreal forward Brandon Prust took an undisciplined roughing penalty late in the period, only to see the Senators power play fail capitalize.
Notes: The Canadiens were missing forwards Brian Gionta and Ryan White with upper-body injuries. Jeff Halpern and Gabriel Dumont drew into the lineup in their place. ... Gryba took the spot of Senators defenceman Patrick Wiercioch, who left Game 3 with a lower-body injury. There were no other lineup changes for the Ottawa. ... Eller, who was sent to hospital with a concussion, broken nose and some lost teeth after the Gryba hit in Game 1, remains out of the Canadiens lineup. ... Former Ottawa goalie Ron Tugnutt was in attendance and got a loud cheer from the crowd in the first period. ... Keila Penner and her family were the guests of Senators owner Eugene Melnyk at the game. The 11-year-old was sent home from her Montreal-area school last week for refusing to remove her Ottawa sweater on 'Habs Jersey Day.' Keila's father grew up in Ottawa and is a die-hard Senators fan.