While it wasn’t your original question, it is established fact that most Southerners did not own slaves, there are census records which show this. So from now on you won’t have to say you have “often heard” you can simply know it as fact ;).
Here is a link to all kinds of interesting data from the 1860 U.S. census.
In total, there were 17 states where slave owners resided in 1860. In those 17 states there were 8,171,786 white people, and 393,975 slave owners (roughly 4.8%.)
In those 17 states, the state with the most slave owners was Virginia with 52,128. The state with the highest rate of slave ownership was South Carolina, where 9.1% of the white population owned slaves.
It’s worth noting that in the state with the highest rate of slave ownership, it was still well below 50%, meaning it is quite factual that the majority of whites did not own slaves.
It also raises the interesting question of why so many whites fought and died in order to keep rich men their slaves. It’s a complex issue, by and large the Southern plantation owners were simply the power elite. Slave owners were a very small portion of the population.
Even more remarkable, is that a small portion of the slave owners owned a majority of the slaves. The system of slavery was actually economically bad for whites who didn’t own slaves, and wasn’t very advantageous even for the whites who owned one or two slaves.
It tended to be very profitable for the very small number of people who owned a great number of slaves. These people were the most powerful people in their societies, and used their power to shape state politics. While only a small portion of the population, these types of people typically dominated the state house, tended to be the class from which U.S. House Representatives and Senators were chosen from.