Abbott struck his 14th goal of the season and was a constant threat to the home side as the Terriers held on for a deserved win.
Andy Holdsworth fired a warning shot over the bar after three minutes before Huddersfield took the lead on 19 minutes when Abbott seized on a cross by Rob Edwards to fire home from six yards.
The visitors doubled their lead 11 minutes before the break when home defender Jerel Ifil miscontrolled a long kick from goalkeeper Paul Rachubka and Brandon ran in to slot home the second.
Ifil's evening got worse moments after the break when he received a straight red card for pushing over Abbott.
Swindon defender Sean O'Hanlon flashed two headers over the bar from Steve Jenkins corners as they went in search of a consolation, before Abbott almost added a third for the away side when his curling 25-yard effort sailed just wide of the post.
Referee Kevin Woolmer evened up the sides with just three minutes remaining when he sent off Huddersfield midfielder Lee Fowler for a two-footed lunge at substitute Grant Smith.
Smith was again involved in the action a minute from time when he poked home a Jenkins cross from six yards to give the home side hope.
But despite four added minutes Swindon failed to breach a stubborn Huddersfield defence again, as they fell to defeat in front of their lowest league crowd of the season.