Life after the Lords of Gower

His successors, the Earls of Worcester, Marquis of Worcester and Dukes of Beaufort remained in control as absentee landlords until the 20th century. After the Civil War Oliver Cromwell was granted the Lordship of Gower in 1647. By the time a formal survey of the assets was undertaken in August 1650 the castle was described as "a decayed Buildinge called the Castle of Swanzey ".

On the first known sketch of the castle, created by Francis Place in 1678, we find that the square tower was  used as a glass works producing wine bottles, samples of which have been found during excavations here. In 1700 a town hall was built in the castle courtyard.  Where the great and the good had once been entertained in the Great Hall, the poorest of Swansea's inhabitants were later incarcerated; in 1750 the Great Hall became the town's "poor house" (workhouse). A market filled the courtyard by the 1770s. Other buildings cosied up to the castle walls along Castle Lane, whilst on the site of the old castle fine Georgian residences such as Worcester House, with its bay windows and gardens looking out over the river, were built.