Nepotism is, word for word, nephew-ism. It comes from the Italian nepotismo, built on nepote, "nephew." That in turn comes from the Latin nepos, which could mean either a grandson or a nephew.
The nephews in question belonged to popes. From the Middle Ages to the late 1600s, popes regularly handed offices, estates and cardinals' hats to relatives. As celibate clergy they usually had no legitimate children, so nephews took the place a son might have.
The habit was close to universal. Wikipedia notes that every Renaissance pope who created cardinals appointed a relative to the College of Cardinals. The favoured relative became powerful enough to hold a recognised post: the cardinal-nephew, who handled the pope's correspondence with his envoys.
The examples run in chains. Callixtus III, a Borgia, made two nephews cardinals. One of them, Rodrigo Borgia, later became Pope Alexander VI. Alexander raised his mistress's brother, Alessandro Farnese, to cardinal. Farnese in turn became Pope Paul III and made two of his grandsons cardinals, aged 14 and 16. Merriam-Webster singles out Sixtus IV, in the late 1400s, for his many favours to his nephews.
A popular version of the story says "nephew" was a polite cover for a pope's illegitimate son. That is partly true. Etymonline calls "nephew" a euphemism for a pope's illegitimate son, and Wikipedia records children of broken vows being passed off as nephews. But most cardinal-nephews really were nephews. And when Alexander VI made his own son a cardinal, no disguise was involved.
The word came into English through French népotisme and is first recorded in the 1660s, or 1670 according to Merriam-Webster. Merriam-Webster links its spread to Gregorio Leti's 1667 book Il nipotismo di Roma, whose title shows the word in Italian.
The Church eventually shut the practice down. On June 22, 1692, Pope Innocent XII issued the bull Romanum decet pontificem. It abolished the office of cardinal-nephew and limited each pope to making one relative a cardinal. The last cardinal-nephew had been named in 1689.
By then the word had left the Vatican behind. Today it covers any boss who hires family, and the slang "nepo baby" has given it a second life.