The popular story says Romans gossiped at crossroads, so small talk became "trivia." It is neat, and it is only partly supported.
The Latin part is solid. A trivium was a place where three roads meet, from tri-, "three," and via, "road." Etymonline says the adjective trivialis meant "common, commonplace," tracing it to the crossroads through the idea of a public place. That is the root of English "trivial." The Romans also used the name for Hecate, the goddess of three ways, who was linked to crossroads.
There is a second, rival route. In the Middle Ages the trivium was the first stage of the seven liberal arts: grammar, rhetoric and logic. The Word Origins site traces "trivia" and "trivial" to that medieval trivium, treating the crossroads as the older Latin meaning. Etymonline records English "trivialle" for the school subjects in the 15th century, and Thomas Nashe used "trivial" for ordinary or unimportant things in 1589.
The noun trivia in its modern sense is much younger. In 1716 the poet John Gay published a mock poem called Trivia, named for a goddess of city streets. In 1902 the American-born essayist Logan Pearsall Smith borrowed the title for a book of short, light pieces. Etymonline credits his book for the modern sense, "trifling facts of little consequence," recorded by 1932. Word Origins finds the noun in the Glasgow Herald on 21 July 1920, for the minor details of a holiday.
Then came games. By 1965 American college students were competing to answer questions about obscure popular culture, which was already being called trivia. Trivial Pursuit arrived in 1982 and turned the word into a household brand.
So the road imagery is real, but the gossip story is not documented in these sources. What the record shows is a word for "common" that became a book title and then a pastime.