Swindon went behind after only 3 minutes against the run of play when Richard Barker scored from close range for his 13th goal of the season.
Ady Williams came close to getting an equaliser when his shot hit the bar, but it went out of play rather than Swindon getting another chance to score.
On 38 minutes Brown made it 2-0 to Mansfield going around Phil Smith who was off his line. Smith had earlier just made an excellent save to keep Swindon in the game.
The second half was much like the first with Swindon having plenty of the ball, but not having many chances in front of goal. Christian Roberts was replaced with Royce Brownlie.
Overall a fairly average game which Swindon just didn't look on form. Mansfield only had another 5 shots on goal, and scored two. Swindon now make the short trip to top of the table Walsall on Saturday needing a win to keep them in the top 3.