English has two apostrophes, and the one most people never think about came first.

Both come from Greek apostrophos, "turning away," from the verb apostrephein: apo, "away," plus strephein, "to turn." Both reached English through French in the 1500s, half a century apart.

The older English sense belongs to rhetoric. An apostrophe in this sense is a speaker's brief turn away from the main audience to address one particular person. Grammarphobia quotes the OED's earliest citation, from Sir Thomas More's Apology of 1533: "With a fygure of apostrophe and turning his tale to God criyng out: O good Lorde." Etymonline dates this sense to the 1530s.

The punctuation mark followed in the 1580s. Its Greek name was prosōidia apostrophos, the "accent of turning away." The idea was that a letter had been turned away, or left out. According to Grammarphobia, the OED's first example comes from Shakespeare's Love's Labour's Lost, dated 1588: "You finde not the apostraphas, and so misse the accent." The Latin-looking spelling "apostrophus" lasted into the 18th century.

When the mark arrived in English, it showed where a letter or syllable had been dropped. That is how it ended up in the possessive. In Old English most nouns made the possessive with -es, so "John's house" began as something like "Johnes house." Once the e fell silent, writers marked its absence with an apostrophe.

Then the rule outgrew its reason. Etymonline notes that by the 18th century the mark was being extended to all possessives, whether or not they had ever had an e. The possessive "its" went the other way and dropped the mark, possibly because "it's" was already established as short for "it is."

Not everyone loves the result. In a 1989 article in English Today, quoted by Grammarphobia, Adrian Room called the apostrophe "a cumbersome name for an awkward object." Still, the name fits. Every apostrophe marks a place where something turned away and left.