Every soldier who heard "over the top" knew what came next, and it was not a figure of speech.
Etymonline describes the phrase as First World War soldiers' slang for starting an attack. It means climbing over the top edge of the trench, the parapet, to advance on the enemy. A close synonym, "over the bags," is recorded in the Evening Express of Aberdeen on October 9, 1916, in a report of a city chaplain's address.
The phrase reached a wide audience through an American. Arthur Guy Empey, who enlisted in the British army, published "Over the Top" by an American Soldier Who Went in New York in 1917. According to wordhistories.net, his glossary, "Tommy's dictionary of the trenches," calls it a famous phrase of the trenches, usually the order to charge the German lines. That book put the phrase in front of readers who had never seen a trench.
Almost at once the phrase started to leave the battlefield. In December 1917 the Liverpool Post & Mercury quoted Canada's finance minister saying Canada had "gone over the top" with its Victory Loan. This is the positive sense of beating a quota or goal, and it survives in fundraising.
The "too far" sense came next. The Manchester Guardian of March 12, 1919, quoted a church leader at a Sheffield assembly saying the Church must go "over the top" and dare all things. In June 1919, a Dundee paper used it for a bold collective action in a housing rent protest. By September 23, 1920, the Leeds Mercury reported West End theatres had "gone over the top" in a fight over selling chocolates, which is close to the modern idea of going beyond what is usual.
Dictionaries date the final shift later. Etymonline attests the figurative sense of going too far from 1968, and Merriam-Webster gives 1980 for the hyphenated adjective "over-the-top," meaning extremely or excessively flamboyant. That gap suggests the sense of excess lived in speech and newspapers for decades before dictionaries recorded it.
Etymonline notes that the word top has meant the rim of a ditch since about 1300, so the military phrase was built from old parts.