Hit-man Williams equalised for his side with a right-footed finish after Stuart Fleetwood's cross and then struck a second-half winner to leave Swindon heartbroken.
Swindon missed a hatful of opportunities in the second-half and failed to give caretaker boss Ady Williams his first points since taking charge.
Swindon captain Lee Peacock had the first opportunity of the day on six minutes, but his header was cleared off the line by Hereford midfielder John Eustace.
Peacock should have done better on 17 minutes when he met Aaron Brown's cross with a firm header that went agonisingly wide.
Swindon striker Christian Roberts, back in the side after an injury lay-off, showed his team what they had been missing when he opened the scoring on 32 minutes. Swindon midfielder Michael Pook crossed from the right-hand side and Roberts headed into the roof of the net from eight yards out.
Hereford hit back through Andy Williams on 36 minutes.
Swindon lost the ball in Hereford's half and from a clearance Stuart Fleetwood went clear. He crossed for Williams to hammer into the roof of the net past Swindon new boy Andy Lonergan from 12 yards out.
Eustace was prompting his side all over the pitch and showing his former Premiership pedigree. The former Coventry star hit a right-footed volley from 20 yards that Lonergan managed to gather at the second attempt.
Swindon came out after the break reinvigorated and threw everything they had at the Hereford defence as Roberts hit a right-footed drive on 53 minutes that went just wide.
Peacock then headed wide after Roberts had crossed from the right-hand side. He was left holding his head in his hands as he knew he should have done better.
Swindon midfielder Ricky Shakes was the next to test the Hereford defence on 62 minutes when his right-footed drive went just over the bar.
Paul Evans hit a speculative 25-yard right-footed shot that just cleared the bar and Roberts went close with a right-footed shot that was well saved by Wayne Brown in the Hereford goal.
Swindon were left to count the cost of their missed opportunities when Hereford grabbed the winning goal.
Hereford full-back Neil MacKenzie put striker Williams through on goal and his clever shot went underneath Lonergan and into the back of the net.
Swindon pressed hard for an equaliser but the closest they came was in injury time but Evans' 25-yard free-kick went over the bar.