The Varbola Stronghold was the largest circular rampart fortress and trading centre in Estonia in the 10th-12th centuries. The first record of Varbola is written by Henry of Livonia, who mentions the Castrum Warbole being besieged in 1211 for several days by Mstislav the Bold of Novgorod. The conflict was resolved with a payment of seven hundred Marks.
During the Livonian crusade Livonian Brothers of the Sword invaded the territory and the people from Varbola asked for the terms of peace. The terms offered by Livonian Brothers of the Sword were accepted by Varbolians, but later the people of Varbola became the subjects of the King of Denmark. On the basis of the Danish Census Book the estate surrounding the Varbola trading center remained a possession of the Lode family, nobility of Estonian origin at the time. The Varbola stronghold lost its importance only in the second quarter of the 14th century, after having played in important part in the anti-Christian and anti-German St. George's Night Uprising of 1343.
In the 16-17th centuries the stronghold was used as a cemetery. The first known fort plan dates from 1786 and was drawn by Ludwig August von Mellin. Archaeological excavations at the site have been undertaken in 1938-41, 1953 and 1974. Among the archaeological finds were dice made of bone.
Parts of the ruins of the 580 metre long and 8-10 metre high limestone wall of the fortress stand still this day. The long gateways with multiple gates were built to defend the entrances. In these sections higher defensive towers were erected. There was a 13 metre deep well in the middle of the fortress and the territory held about 90 structures with furnaces for accommodation built with limestone floors and foundations.
Ängsö Castle was first named as "Engsev" in a royal charter by king Canute I of Sweden (r. 1167-1196), in which he stated that he had inherited the property after his father Eric IX of Sweden. Until 1272, it was owned by the Riseberga Abbey, and then taken over by Gregers Birgersson.
From 1475 until 1710, it was owned by the Sparre family. The current castle was built as a fortress by riksråd Bengt Fadersson Sparre in the 1480s. In 1522, Ängsö Castle was taken after a siege by king Gustav Vasa, since its owner, Fadersson's son Knut Bengtsson, sided with Christian II of Denmark. However, in 1538 it was given by the king to Bengtsson's daughter Hillevi Knutsdotter, who was married to Arvid Trolle.
In 1710, the castle was taken over by Carl Piper and Christina Piper. Ängsö Castle was owned by the Piper family from 1710 until 1971, and is now owned by the Westmanna foundation. The castle building itself was made into a museum in 1959 and was made a listed building in 1965. It is currently opened to visitors during the summers.
The castle is a cubical building in four stores made by stone and bricks. The lower parts is preserved from the middle ages. It was redecorated and expanded in the 1630s. The 4th storey as well as the roof is from the expansion of Carl Hårleman from 1740-41. It gained its current appearance in the 1740s.