On the Norman conquest of Glamorgan, Caradoc, the eldest son of the defeated prince, Iestyn ab Gwrgant, continued to hold this lordship, and for the defence of the passage of the river built a castle whose ruins still lie above the town on Mynydd Dinas. His descendants established, under line protection of the castle, a chartered town. In modern times this charter was not acted upon, the town being deemed a borough by prescription, but in 1861 it was incorporated under the Municipal Corporations Act. From 1832 it belonged to the Swansea parliamentary district of boroughs, uniting with Kenfig, Loughor, Neath and Swansea to return one member; later it acquired its own MP.
Michael Caradoc had the family house, Corlannau, built from around 1830, in the Tudor Gothic style. It incorporates stone from the medieval manor house that preceded it, and it is thought that this in turn drew its materials from yet older buildings.