Roberts hit a left-footed drive from 15 yards in the second minute of injury time and central defender Fitzgerald deflected the ball past Kasper Schmeichel in the Bury goal. The strike condemned Bury to a second straight league defeat and kept Swindon in the promotion hunting pack.
In a dull opening to the game both teams found it hard to settle and cancel each other out in midfield.
Right-back Paul Scott broke the deadlock in the 20th minute when he headed home Thomas Kennedy's corner. The Swindon defence were caught napping and let Scott get in the free header that he powered into the net.
Roberts hit back for Swindon on 34 minutes with a well taken goal.
Left back Andy Nicholas's cross picked out Roberts 12 yards from goal. He turned and struck a first time left-footed shot into the top left-hand corner leaving Schmeichel with no chance.
Swindon striker Lee Peacock should have extended his side's lead but he put his free header wide of the right-hand post from Aaron Brown's free kick.
Schmeichel emulated his famous father, Peter, just before the break when he saved Roberts' shot with an instinctive save with his feet.
Bury almost took the lead after the break when Richie Baker put a free header wide of the right-hand post from Marc Pugh's corner.
Swindon striker Roberts almost scored with a fabulous curling free kick. He hit the ball from 25 yards from the left-hand side of the penalty area and his effort whistled just wide of the right hand post.
It looked like both sides had settled for the draw, but Roberts had other ides.
He picked up the ball on the left-hand edge of the penalty area and hit a shot that Fitzgerald could only deflect past his goalkeeper.