Ten years of community theater experience, here.
Of course they paid for the rights to put on these shows. That’s how community theaters operate. Nobody but the rawest of beginners would try to stage a show without securing the rights. You might fly under the radar once or twice, but eventually you would get caught at it, and rights owners do not have sense of humor about those sort of violations.
It’s not possible to say exactly what the royalties to a particular show are, since it depends on a lot of variables. These include the size of your theater, how many performances you’re going to do, whether any of the proceeds will go to charity, whether you’re paying any of your actors, stuff like that. But for a small theater, with presumably a limited number of seats and maybe a single weekend of performances, royalties might not be all that high, unless they’re trying to do Rent or something!
Community theaters will work out a budget for their season, figuring out how much each show will cost to put on (which includes both royalties and expenses for things like sets and costumes), how many ticket sales they expect to generate, how much they can make by selling popcorn and sodas and stuff, and then at the end of the day, hopefully they show a profit from that. It can be a delicate balancing act sometimes. Do we do an additional weekend of performances? That might bring in more money from ticket sales–but it will also add to the cost of the royalties. Where is the break-even point?
If the theater has been around for a few years, they (hopefully) have money left over at the end of every season, which they will have in a bank account somewhere, earning interest. When next season comes along, they can use that money to start paying for rights to their next batch of shows. That money will also go toward things like keeping the theater lit and heated, upkeep on the building, and so forth.
In addition to money generated from ticket sales, community theaters often also get money from donations (look on your local theater’s website, if they have one–I’ll bet you can find a link you could use to donate money to them!). Look through the program when you go to a show, and you will likely see advertisements from local businesses. All of those bring in funds. Sometimes a business will donate so much money that they will be identified as a sponsor of a particular show, or even the whole season. Businesses do this either for the advertising benefits, or the prestige of being associated with local theater, or sometimes just because their owner has an interest in the arts and wants to support them.
As to the actual question, could you just adapt Silence of the Lambs into a musical, the answer is usually no. That would be a derivative work, and the creation of these is generally controlled by the copyright holder. Silence! likely gets away with it because it’s a parody.
N.B., this is not legal advice. I am not your lawyer–indeed, I am not anyone’s lawyer, as I do not have a law degree. I have, however, appeared in a community theater production of Legally Blonde. 
