People living nearby were evacuated and spent the night in the town hall. No one was hurt.
Works on loan by Albrecht Durer, Hercules Segers, Jacob van Ruisdael, Anselm Kiefer, and Richard Long have all been lost.
The museum is the permanent home for works by Armando, 78, one of the Netherlands's most important post-WWII artists.
"I'm not thinking so much about my own work," he said. "A number of the works on loan are irreplaceable."
The fire started in the tower of the 19th century Elleberg church, where the museum is located.
Officials have not yet determined cost from the damage, but media report the insurance value of the destroyed artworks at nearly $4.3 million.
An investigation has been started into the cause of the blaze. The church roof was undergoing construction when fire broke out.
The blaze is under control but the church tower may yet collapse.