The first castle in Neath was located west of the river near the Roman fort of Nidum, and was a timber fortification in a motte and bailey structure. When Richard de Grenville founded Neath Abbey close by, he abandoned this original castle, and it may have been used by the monks as a source of building material.
A second castle on the opposite bank of the river, in what is now the centre of the town, is first documented in 1183; shortly afterwards, William de Cogan, son of Miles de Cogan, was appointed constable. This second castle was built by Gilbert de Clare, 5th Earl of Gloucester. During the 13th century, being a Norman stronghold, it was subject to attack by the Welsh, notably by Llywelyn the Great, who captured it in 1231 with help from a local Welsh lord, Morgan Gam. Following this, it was substantially rebuilt by Richard de Clare, 6th Earl of Gloucester, Gilbert's son.
It was taken again, and this time destroyed, by Humphrey de Bohun, 4th Earl of Hereford, in 1321, during the rebellion against King Edward II of England. Its owner, Hugh Despenser the Younger (who had gained the lordship through his marriage to the heiress Eleanor de Clare), rebuilt it, possibly with the addition of a gatehouse. In 1376, one Roger Kyngot was the constable, and the castle was rebuilt in stone in 1377; that is probably when the great gatehouse was built, which is the main surviving feature. The castle was in use until the 17th century, and has been a recreational area for the town since the 18th century.References:
First record of Kastelholma (or Kastelholm) castle is from the year 1388 in the contract of Queen Margaret I of Denmark, where a large portion of the inheritance of Bo Jonsson Grip was given to the queen. The heyday of the castle was in the 15th and 16th centuries when it was administrated by Danish and Swedish kings and stewards of the realms. Kastelhoma was expanded and enhanced several times.
In the end of 16th century castle was owned by the previous queen Catherine Jagellon (Stenbock), an enemy of the King of Sweden Eric XIV. King Eric conquered Kastelholma in 1599 and all defending officers were taken to Turku and executed. The castle was damaged under the siege and it took 30 years to renovate it.
In 1634 Åland was joined with the County of Åbo and Björneborg and Kastelholma lost its administrative status.