The present Visconti Castle replaces the older castle, located near the Church of S. Maria Vecchia, which was destroyed by Emperor Frederick Barbarossa in 1167.

Strategically placed close to the Naviglio canal, it was built in the late 13th century and was enlarged and renovated probably at the behest of Azzone Visconti (1329- 1339). In 1354 Galezzo II Visconti gave the fortress to his wife Bianca di Savoia, and from then on it would be the tradition of the dukes of Milan to give the castle to their wives as a wedding gift.

In 1381 with Gian Galezzo, the castle was transformed from a defensive fortress into a stately home. It will always be Gian Galezzo who will give birth to his first heir, Giovanni Maria, here in the castle. The event of September 7, 1388 will be remembered with the dedication of the Church of "Santa Maria Nascente," today known as S. Maria Nuova. It would be with Filippo Maria, the last heir of the Visconti family, that the castle would see its greatest splendor with new decorations, of which only a few frescoes remain today in the courtyard and interior rooms with the motto "a bon droit," demonstrating his absolute power. On the top floor of the castle, where the prisons were, one can see engraved on the wall the inscriptions and drawings left by political prisoners from the time of the Ambrosine Republic 1447-1450, which ended with Francesco Sforza's seizure of power.

Filippo Maria's natural daughter (later legitimized) Bianca Maria Visconti bride of Francesco Sforza, born and raised here will always hold this village dear, as will her first heir Galeazzo Maria Sforza, who will also stay for a long time to follow the work on the Convento dell'Annunciata he wanted. Engraved in the room at the end of the second floor is the date of birth of Gian Galeazzo Maria Sforza, the first son of Galeazzo Maria and Bona di Savoia: "June 20, 1469."

Instead,Ludovico il Moro, Galeazzo's brother, preferred the Castle of Vigevano, having his widowed sister-in-law Bona of Savoy imprisoned here in Habiate. Gradually the castle lost importance, being reduced to a fortress, with the end of the Sforza dynasty. It is the site of clashes between the French and Spanish (1524 -1527), resulting in the ruin of part of the building. During the Spanish era the castle went into gradual decay, until 1658, when it was partially demolished and the only remaining tower, out of 4, lowered.

Reduced to a noble residence, it remained in private hands until 1862, when the municipality bought it, to place first schools, then municipal offices. Currently, after a restoration completed in 1994 and one completed in 2009, it houses the R. Brambilla Civic Library, other municipal offices and is home to cultural events