IIRC, for example, Dred Scott was a slave owned by a middle class white person.
Dred Scott was born a slave in Virginia around 1799.[18] Little is known of his early years.[19] His owner, Peter Blow, moved to Alabama in 1818, taking his six slaves along to work a farm near Huntsville. In 1830, Blow gave up farming and settled in St. Louis, Missouri, where he sold Scott to U.S. Army surgeon Dr. John Emerson.[20] After purchasing Scott, Emerson took him to Fort Armstrong in Illinois. A free state, Illinois had been free as a territory under the Northwest Ordinance of 1787, and had prohibited slavery in its constitution in 1819 when it was admitted as a state. In 1846, Dred Scott returned to St. Louis and tried to purchase his freedom from Mrs. Emerson, who refused. The Scotts then decided to pursue their freedom through the courts.
So he was originally owned with 6 other slaves, then sold to a doctor where there is no other indication a more slaves in the household other than Dred’s wife. Her tale is even more convoluted:
Born into slavery in Virginia, Harriet Robinson lived briefly in the free state of Pennsylvania before being taken to the Northwest Territory by Indian agent and slaveholder Lawrence Taliaferro. In 1836 or 1837, Harriet married Etheldred, an enslaved man who had been brought to Fort Snelling in present-day Minnesota by Dr. John Emerson, a military surgeon. Their civil wedding ceremony was officiated by justice of the peace Taliaferro, who never actually sold Harriet to Dr. Emerson, since slavery was illegal there. In 1838, Harriet Scott gave birth to their first child on the steamboat Gipsey as it traveled north on the Mississippi River, in free territory, back to Fort Snelling. In 1840, the Scott family moved to St. Louis in the slave state of Missouri with Irene Emerson, who eventually hired them out to her brother-in-law, Captain Henry Bainbridge, at Jefferson Barracks. After Dr. Emerson’s death in 1843, Dred accompanied Captain Bainbridge to Louisiana and Texas, leaving Harriet and their two daughters behind in St. Louis. Harriet was hired out to Adeline Russell, wife of grocery wholesaler Samuel Russell, most likely working as a laundress.
So from that it seems pretty common that there was a class of slaves above field worker, who were relied upon as household servants, or even - like Scott - actually manage a business for their owner. That Scott could even offer to buy his own freedom - or his wife’s - suggests a form of lifestyle as a slave far above the common (mis)conception of slaves as simply plantation fieldhands and uneducated physical labourers. That he could arrange the lawsuits to try to win his freedom buttresses this view. History is always far more complex than we think.
(Speaking of Huckleberry Finn, as I understood the character Jim was also the single slave owned by Miss Watson, who freed him in her will.)