An impromptu speech sounds unprepared, but the phrase behind it means the opposite: "in readiness."
Latin in promptu combined in with promptu, a form of promptus, "ready, prepared" or "brought forward." Promptus was the past participle of promere, "to bring out." That verb was built from pro-, "forward," and emere, "to take." So something in promptu was brought out and kept at hand, ready for use.
French fused the phrase into a single word, impromptu, in the 1650s. Merriam-Webster says the French used it to mean "extemporaneously." English borrowed it in the 1660s, first as an adverb. The noun followed in the 1680s, and the adjective in 1764, according to Etymonline.
The shift in meaning is small but telling. Being ready for anything means you can respond without warning. An impromptu remark is one you had ready, or appear to have had ready, when the moment came.
Impromptu has a close relative in prompt. Merriam-Webster notes that both go back to promere, and that the adjective prompt comes from promptus. A rarer cousin is promptuary, a book of ready reference. Another near-synonym, extempore, makes the same point with a different image. Latin ex tempore means "out of time," and English used it for spur-of-the-moment acts from the 1550s.
Musicians took up the word in the 19th century. Wikipedia, citing an 1815 report in the Allgemeine musikalische Zeitung, says Johann Baptist Cramer had begun publishing piano pieces subtitled "impromptu." Jan Václav Voříšek is credited with the first pieces published under the title Impromptu, in 1822.
Franz Schubert produced two sets of four impromptus for piano, Op. 90 and Op. 142. Frédéric Chopin wrote four, including the much-loved Fantaisie-Impromptu. These were written-out compositions. Wikipedia describes the genre as free-form pieces that give the impression of an improvisation, as if inspired by the moment.
That gap suits the word. From Latin onward, an impromptu has always depended on being ready in advance.