I just read a book on Biblical archaeology, and IIRC the deal was that Israel and Judah were at war, and the king of Judah made a deal with the king of the Assyrians in which the Assyrians would wipe out Israel. The Assyrians had a general policy of relocating all captured peoples and dispersing them throughout the enormous Assyrian empire, to diffuse any sense of national identity and thereby make revolts less likely. As a result, the conquered Tribes of Israel were, in a word, Lost, as they assimilated into the Assyrian melting pot. (This is all well-attested by archaeology- there is a bas-relief of the king of Judah bowing to the king of Assyria, which is the only known portrait of a Jewish king.)
Then, the Babylonians took over the Assyrians. The homogeneity of the Assyrian empire made it easy for them to seize the reins of the whole thing at once. It seems reasonable to assume that yes, indeed, the Babylonians deposed the vassal king of Judah at this point. Apparently things weren’t so harsh under the Babylonians, slavery-wise. Remember, the book of Daniel and (again, IIRC) the Psalms about the captivity were written long afterwards. (Daniel was written roughly 500 years later, during the Maccabean revolt.) Of course, the Babylonians destroyed the temple and stole its gold (that is, the gold remaining after the Jewish kings had sold much of it off to finance wars and bribe neighboring kings with tribute.) But, OTOH, I don’t think the Jews had to make bricks while being bullwhipped.
My source for all this is Magnus Magnusson’s book Archaeology and the Bible, which is a bit out of date (mid-70s) but takes a balanced view of its subject.