Standing between the confluence of the Eresma and Clamores rivers, the first documented record of the fortress dates back to Christian writings in the 12th century.
It was built on top of a rock, a testament to its original military status. It was used as a residence by Alfonso VIII. In the 13th century the building took on an air of Gothic elegance thanks to the Spanish architects working for John II and Henry IV. The last architectural phase was completed in 1587 by the architect Francisco de Mora, possibly in collaboration with Juan de Herrera, who designed the main courtyard and the Escuela de Honor building. In 1764, Charles II founded the Royal College of Artillery here. The edifice has a number of secret passageways that lead to the river and connect several of the city's palaces.