For twenty years the United States government officially spelled Pittsburgh without its "h." The city mostly refused to go along.
The name dates from the French and Indian War. The French had founded Fort Duquesne in 1754, naming it for Michel-Ange Duquesne, governor of New France. In 1758 British forces under General John Forbes took the site. Forbes renamed it for William Pitt the Elder, the British statesman.
The earliest known uses are Forbes's own letters of 26 and 27 November 1758. The Pennsylvania Gazette printed "Pittsburgh (formerly Fort Duquesne)" on 14 December 1758.
The ending reflects the man who chose it. Forbes was a Scot, and burgh is the Scottish form of borough. Wikipedia suggests he probably said the name like a Scot saying Edinburgh, roughly "PITS-bər-ə." Etymonline notes that this Scottish ending is rare in American place names.
Behind both spellings is the Old English burh, a dwelling inside a fortified enclosure. Over time the word came to mean a fortified town and then any town.
Spelling wobbled for a century. Early municipal records usually included the "h," but the 1816 city charter left it out. Then the federal government stepped in. The United States Board on Geographic Names set out to standardize -burgh endings as -burg. Its 1891 decision made the city "Pittsburg" for federal purposes.
Pittsburgh resisted. The city government, the Pittsburgh Gazette, the stock exchange and the University of Pittsburgh all kept the "h." On 19 July 1911, after a Chamber of Commerce campaign and pressure from Senator George T. Oliver, the board reversed itself.
The short form did not vanish overnight. The Pittsburg Press kept it until 1921. Elsewhere it stuck: Pittsburg, Kansas, and Pittsburg, California, still spell their names without the "h."