To rummage once meant the opposite of what it means now. The word began as the tidy, careful packing of cargo into a ship's hold.
Its source is Middle French arrumage, from the verb arrumer, which Wiktionary glosses as "to arrange the cargo in the hold." Modern French spells them arrimage and arrimer, and Merriam-Webster uses those spellings for the Middle French words too. English dropped the first syllable. Dictionary.com calls the result an "aphetic alteration" and dates it to the 1520s.
The French verb has a surprising back story. Merriam-Webster traces its stem to Middle English rimen, "to open up, make room," from Old English rȳman, "to make room." If so, an English word crossed the Channel from English into French and then came back.
The University of Texas Indo-European Lexicon places rummage in a family built on an old root meaning "wide, roomy." Room, roomy and rural belong to the same group.
Packing a hold well takes moving things around, and the search sense grew out of that work. Wiktionary still lists a nautical verb meaning to search a vessel for smuggled goods, the kind of job a customs officer does after a long voyage. Dictionary.com says the idea of rummaging as searching was first recorded in the 1620s, though Merriam-Webster dates its verb sense of a thorough search to 1607.
From there the meaning slid toward disorder, and a rummage became a search that leaves a mess. The noun had a messy side early on: Merriam-Webster's earliest noun sense, from 1598, is "a confused miscellaneous collection." The verb now covers any haphazard poking around, like rummaging in the refrigerator.
The rummage sale keeps the harbor in its history. Dictionary.com explains that the first one was a sale on the docks of unclaimed items. The jumble of goods on offer fits the word's later sense of a confused collection.
So the word that once meant packing a hold neatly now describes turning one upside down.