Yeovil took the lead in the 10th minute when Luke Rodgers won a corner and from Danny Schofield's flag-kick, Craig Alcock headed past Phil Smith.
Swindon's cause was not helped when skipper Hasney Aljofree limped off to be replaced by 17-year-old Sean Morrison.
A fumble by Josh Wagenaar almost provided an opening but the ball fortunately dropped to a defender's boot.
The visitors went two up on 28 minutes as the Swindon defence waited for an offside decision.
Schofield played in Lee Peltier, who fired in to the bottom corner.
Swindon had plenty of possession but mounted few constructive attacks and their best effort was a Simon Cox header which skimmed the bar from Anthony McNamee's cross.
Swindon were back in the game in first-half added time.
Lee Peacock was brought down by Terrell Forbes, who earned a booking in the process.
Worse followed for the visitors as Michael Timlin found the top corner with his free-kick.
It got better for Swindon immediately after the break when Jon-Paul McGovern lashed the ball into an empty net as a high ball caught the visiting defence in some confusion.
There were several chances for both sides as the game wore on but neither team could produce quality finishing.
But it was Yeovil who regained the lead in the 70th minute when the home defence went to sleep and a bouncing ball into the goal area appeared to deflect off Swindon defender Jerel Ifil.
Cox for Swindon had two excellent chances late on which he spurned and for the visitors, Rodgers was free in a one-to-one with Smith but his attempt to chip the goalkeeper gave Smith an easy save.