Varberg Fortress was built in 1287-1300 by count Jacob Nielsen as protection against his Danish king, who had declared him an outlaw after the murder of King Eric V of Denmark. Jacob had close connections with king Eric II of Norway and as a result got substantial Norwegian assistance with the construction. The fortress, as well as half the county, became Norwegian in 1305.
King Eric's grand daughter, Ingeborg Håkansdotter, inherited the area from her father, King Haakon V of Norway. She and her husband, Eric, Duke of Södermanland, established a semi-independent state out of their Norwegian, Swedish and Danish counties until the death of Erik. They spent considerable time at the fortress. Their son, King Magnus IV of Sweden (Magnus VII of Norway), spent much time at the fortress as well.
The fortress was augmented during the late 16th and early 17th century on order by King Christian IV of Denmark. However, after the Treaty of Brömsebro in 1645 the fortress became Swedish. It was used as a military installation until 1830 and as a prison from the end of the 17th Century until 1931.
It is currently used as a museum and bed and breakfast as well as private accommodation. The moat of the fortress is said to be inhabited by a small lake monster. In August 2006, a couple of witnesses claimed to have seen the monster emerge from the dark water and devour a duck. The creature is described as brown, hairless and with a 40 cm long tail.