A lot depends on the type of musket and how much you shot it before the ball got stuck.
Black powder leaves a lot of residue which very quickly fouls up your barrel. This makes it harder to both load and unload the musket.
A musket ball moves fairly easily up and down a clean smooth-bore musket. Hunters back in the day tended to use tighter fitting balls for more accuracy. Keep in mind though that a smooth bore musket always fires curve balls. The ball will randomly contact the side of the barrel on one side or another as it travels down the barrel’s length, which will make the ball spin in some random direction. It will go straight for roughly 50 to 75 yards or so. After that, where it goes is anyone’s guess. A tighter fitting ball helps your accuracy somewhat, but the thing is still spitting out curve balls.
Military muskets tended to use smaller, looser fitting balls as they cared more about rate of fire and quick loading than they did about accuracy. A line of musketeers firing in unison was basically like a huge shotgun-like blast at the enemy.
After you’ve fired a few shots, a looser fitting ball will travel much more easily up and down the barrel. A gentle push/pull is enough force for a loose fighting ball in a clean barrel. A bit more force is going to be required for a tighter fitting ball in a heavily fouled barrel. Either way, you’re nowhere near “mighty yank” forces. A mighty yank is just going to rip the screw right out of the ball anyway.
For older rifles, before you had the Minie ball (pre-1840s), the bullet/ball had to fight tightly in the barrel so that it could grip the rifling, otherwise you didn’t get any benefit to having a rifle and you might as well be shooting a smooth bore. Militaries usually didn’t use rifles much (Napoleon refused to have any at all in his armies) because the barrel would foul too quickly and make reloading all but impossible. Hunters could easily stop and clean their barrels in between shots. Soldiers didn’t have that luxury. If you have something like your typical Kentucky/Pennsylvania rifle and it’s been fired quite a few times, getting a round in or out of it is going to be significantly more difficult.
The whole point of a Civil War era Minie ball is that it has a hollow skirt. This makes it smaller than the barrel for easy loading and unloading, even with a fouled barrel, but when you shoot it, the skirt flares out and grabs the rifling, making it accurate. This isn’t going to be too much more difficult to load and unload than a smooth bore musket, though again it does get significantly more difficult when the barrel is heavily fouled.
In movies, they always ram the bejeezus out of the round whenever they load a musket, which you don’t need to do (typical Hollywood - looking cool is more important than accuracy). This gives people the impression that it’s a lot harder to shove the round down the barrel than it actually is. In reality, getting the round past the first couple of inches can take some effort (that’s why they make ball starters for muskets), but after that, all it generally takes is a fairly gentle push to get the round all the way to the back end of the barrel. You want to give it a firm push once you get it all they way in to the breech, but that’s just because you want to make sure there’s no air gap between the bullet and the powder, or your musket can become a pipe bomb that blows up in your face. But overall, there’s a lot less force required than most people think to make the round move up or down the barrel.