Before it described gossip and headlines, "Tabloid" was a brand of pills.
In 1884 the British firm Burroughs, Wellcome & Company used the name for its compressed, concentrated medicines. It joined tablet to the suffix -oid, which means "like" or "resembling." A tabloid was a tablet, or something close to one.
The name caught on so fast that it became a general word for a drug in this form. Within a few years it was being used for other things squeezed into a small dose. Etymonline records a figurative sense of "any compressed form" by 1898.
Journalism borrowed the idea. Etymonline records "tabloid journalism" in 1900 and the newspaper sense in 1901. Word Histories ties the journalistic use to Alfred Harmsworth, the British publisher. For New Year's Day 1901 he edited an experimental issue of the New York World. It called itself "the busy man's paper" and promised "all the news in sixty seconds." He described the approach as "condensed or tabloid journalism."
At first, the word was a compliment. It referred to news in a concentrated, easy-to-swallow form. Whether it described the short stories or the smaller pages was unclear from the start, because the 1901 issue linked the two. Etymonline notes that some people read the word as referring to the smaller page.
Page size then took over the dictionary definitions, followed by tone. Word Histories reports that the 1989 Oxford English Dictionary added "sensational" and smaller pages. The 2008 edition defined a tabloid as a paper with pages about half the size of a broadsheet, usually popular in style and dominated by sensational stories.
There is one caution on dates. Word Origins cites an earlier metaphorical use, in the Pall Mall Gazette on 24 January 1891, for a condensed reference book. So the figurative use began earlier than the newspaper sense, but the pill name is the starting point in every source.