There was originally a defensive tower on the site of current Nørlund Castle in the 14th century. It was demolished in 1355 and the first castle was built to the site. It was however destroyed by Queen Margareth I in the late 1300s. The current castle construction started in 1570 by Ludvig Munk and it was completed in 1597.
References:The Citadelle of Quebec is an active military installation and official residence of both the Canadian monarch and the Governor General of Canada. It is located atop Cap Diamant, adjoining the Plains of Abraham in Quebec City. The citadel is the oldest military building in Canada, and forms part of the fortifications of Quebec City, which is one of only two cities in North America still surrounded by fortifications, the other being Campeche, Mexico.
The first fortifications in Quebec were built by the Governor General of New France Louis de Buade, and completed just in time for the Battle of Quebec in 1690.
After the British conquest in the second half of the 18th century, the problem of Quebec City's defences grew more acute.