During the twelfth century a round defensive tower was built on Kalmarsund and a harbour constructed. At the end of the thirteenth century King Magnus Ladulås had a new fortress built with a curtain wall, round corner towers and two square gatehouses surrounding the original tower. Located near the site of Kalmar's medieval harbor, it has played a crucial part in Swedish history since its initial construction as a fortified tower in the 12th century.

One of the most significant political events in Scandinavia took place at Kalmar Castle in 1397, when the Kalmar Union was formed - a union of Denmark, Norway and Sweden (including Finland), organized by Queen Margaret I of Denmark. During the Swedish rebellion against Denmark in 1520, the fortress was commanded by Anna Eriksdotter (Bielke), who at the death of her spouse, Johan Månsson Natt och Dag, in the middle of the rebellion against Denmark in 1520, took control over his fiefs and defended Kalmar against Denmark.

The fortress was improved during the 16th century under the direction of King Gustav I and his sons King Eric XIV and King John III, who turned the medieval fortress into a castle fit for a renaissance king. Kalmar Castle suffered heavy damage during the Kalmar War of 1611-13 and was badly damaged by a fire in 1642. Repairs were begun but from the end of the seventeenth century the castle was allowed to fall into disrepair.