A museum was first a place of study, not a place to look at things.
The word comes from Greek mouseion, which Etymonline says originally meant a temple or shrine of the Muses. It later came to mean a place of study, a library, or a school of art or poetry. Latin borrowed it as museum, "library, study."
The Muses were the nine daughters of Zeus and Mnemosyne and protectors of the arts. The Greek name Mousa is linked to the Proto-Indo-European root men-, "to think."
The most famous mouseion was in Alexandria. Wikipedia says the Musaeum there was an institute for philosophy and research built under Ptolemy I Soter around 280 BC. It was essentially the same institution as the Library of Alexandria, and it inspired Renaissance thinkers, so libraries were at one point called museums.
That is the sense English took first. Etymonline records museum in the 1610s, meaning the university building in Alexandria. By the 1640s it referred to libraries for scholarly study. The sense we know, a building set aside to display objects of art, literature or science, dates from the 1680s.
That date lines up with a real event. Wikipedia says the Ashmolean Museum at Oxford was founded in 1677 from the collection of Elias Ashmole and opened on 24 May 1683, with Robert Plot as its first keeper. It was one of the first museums open to the public.
Public access came slowly. The British Museum opened in 1759, but visitors had to apply in writing and only small groups were admitted each day. The Louvre opened in 1793, during the French Revolution, giving people of all social classes access to former royal collections.
So the word started with the Muses, passed through a library, and only then reached the building with the glass cases.