In the third century AD, the city of Vannes, called Darioritum, acquires the right to fortify. Thus, west of the Gallo-Roman city, a castrum was built. During the Middle Ages, the castrum becomes the centre of the city. Extended in the 14th century, it is reinforced in the 15th century. Over the following two centuries, the ramparts are modernised with the construction of the Garenne buttress. Several medieval gates has been survived, like Tour du Connétable and Château de l'Hermine (former castle, transformed into a palace in the 17th century). Today there is a nice view from the park Jardins des Remparts to the walls.
First record of Kastelholma (or Kastelholm) castle is from the year 1388 in the contract of Queen Margaret I of Denmark, where a large portion of the inheritance of Bo Jonsson Grip was given to the queen. The heyday of the castle was in the 15th and 16th centuries when it was administrated by Danish and Swedish kings and stewards of the realms. Kastelhoma was expanded and enhanced several times.
In the end of 16th century castle was owned by the previous queen Catherine Jagellon (Stenbock), an enemy of the King of Sweden Eric XIV. King Eric conquered Kastelholma in 1599 and all defending officers were taken to Turku and executed. The castle was damaged under the siege and it took 30 years to renovate it.
In 1634 Åland was joined with the County of Åbo and Björneborg and Kastelholma lost its administrative status.