A throat lozenge is named for a shape, not a medicine. A lozenge was first a diamond, and nobody is sure where that word came from.
The geometry is the oldest sense. Merriam-Webster defines a lozenge as a figure with four equal sides, two acute angles and two obtuse ones. The word came into Middle English from French in the 1300s. The OED has some questionable examples from early in that century. Its first clear one is from Chaucer's House of Fame, around 1384, which describes crowns "wroght ful of losenges."
Heraldry kept the shape busy. In English practice a woman's coat of arms has long been displayed on a lozenge rather than a shield. A 1995 ruling by the English Kings of Arms still lets unmarried women bear their father's arms that way.
The French word's origin is a puzzle with two main suspects. The American Heritage Dictionary suggests it may be related to Provençal lausa, a flagstone or roofing slate, possibly of Celtic origin. Its other option is Arabic lawzīnaj, an almond confection. That came from Middle Persian and ultimately from an Aramaic root meaning almond.
The almond theory has real support. The Oxford Companion to Food says the scholar Maxime Rodinson argued for it persuasively in 1956. Lauzinaj was a marzipan-like sweet from the Levant, cut into small pieces to serve. In fourteenth-century European cookery manuscripts, "losanges" were shapes cut from spiced dough, sometimes made with almond milk.
Either way, the word was soon tied to food and medicine. By the sixteenth century, the Companion says, lozenge meant a medicinal sweet, and it has kept that meaning since. The Companion notes that the word still mainly describes a shape, square or diamond, so even biscuits and marzipan can be cut into lozenges.
The word also has a regional cousin. The OED and the Dictionary of American Regional English record "lozenger," with an r on the end, across the United States, chiefly in the Northeast.