“…who of course always went with cheaper non-extra background talent.”
That should have been “non-union background talent.”
I was an extra on the TV movie Category 7: The End of the World. For eight hours on set, they fed me really well (eggs and suasage for breakast, brie and chicken sandwiches for lunch and veal for supper) and paid me about $140 after deductions. I was only working for maybe 45 minutes of that time (mostly standing around in a group, talking); the rest of the time we were watching TV or reading.
This is probably the most important lesson for anyone who’s thinking about doing something like this: It’s almost always really, really boring.
It sounds glamorous to work on a movie, but until you’ve actually been on a film set, you don’t really know how much standing around doing nothing there is (unless you’re one of the few people who’s busy every minute, which definitely doesn’t include extras), and how much it can suck. It will feel disorganized and unprofessional to the inexperienced performer (“man, we’ve been here all day and we’ve only been in front of the camera for ten minutes; these guys don’t know what they’re doing”), but that’s the nature of the business. If a production hires a big crowd of extras (as opposed to cardboard standees or CGI to fill in the distance), they will usually organize their shooting schedule to get the angles with all the people first, so they can let the majority of the mob go, because they know from experience that the extras will get restless and grouchy as their day wears on without having much of anything to do. Note that TV is usually better in this regard, because they work on a tighter schedule and have to get through a lot more setups in a shooting day, so extras are consequently in front of the camera more.
If you can handle this aspect, and prepare for it (lots of reading material, quiet socializing that doesn’t interfere with set management, quick and furtive monkey sex behind the generator truck, etc.), the money is… okay. But be prepared for a lot of downtime, and distance yourself from the people who start grousing after a couple of hours of inactivity, because they’ll be among the people dismissed first.
You definitely need a high tolerance for boredom, can’t argue with that, but, based on my experience, not knowing what the main cast a crew are up to is not always mutually exclusive with them having no idea what they’re doing. 
The above-mentioned “Lethal Weapon IV”, where I played distant background in the aftermath of the final explosion scene that is the staple of the American action film (I don’t know the specifics as I refused to see this turkey after spending two nights working on it), was a prime example. There was one assistant, who had no other job in the world but to stand near where we were, headset on, let us know where to be, and to tell us when cameras were rolling, since we were a good 1/4 mile away from the foreground. Never managed to pull that off once, but did yell at us quite a few times after Richard Donner had yelled at him because we were off mark during every take.
In addition to needing a lot of patience, you must also be able to suffer fools gladly.