I do a lot of potlucks. Comes with being a community-minded hippie chick, I guess. I agree that free-for-all leads to disaster (I literally attended a potluck once with 12 spinach dips and no protein course), and sign-up is not always practical or can be too restrictive.
If you’re going to be doing a regular potluck with the same group of people, a good compromise is to assign not dishes, but categories, by name or birthdate or some other random attribute. The next time, switch categories (this, over time, eliminates someone always getting the cheap soda while someone else buys the expensive roasted chickens.) I put in on the invite or sign announcing the potluck.
So it might look something like this:
Last name starting with:
A-G, bring a protein dish (meat or vegetarian is fine)
H-K bring a vegetable dish
L-O bring a starchy side dish
P-S bring a desert
T-V bring beverages
W-Z bring supplies (napkins, plates, plasticware)
Other categories I’ve used, just to be fun is Astrological sign (Water signs bring drinks, Air signs bring desert, Fire signs bring side dishes, Earth signs bring protein), Birthdates, Hair color, First digit of your phone number after your prefix, Live on a Street, Avenue, Blvd. or Place. Be creative!
It’s not always perfect, and some people ignore the request and bring what they want anyway, but I’ve not had a disaster yet with this method.
ALSO: Consider having people write the name of their item and the recipe for it (if homemade) on an index card and leaving it next to the item on the table. This does two things: alerts anyone with food allergies or preferences as to ingredients, and provides the recipe to copy down if someone really likes a dish.
The other thing I do is use red serving utensils for omnivore dishes, white serving utensils for lacto-ovo vegetarian dishes, and green for vegan dishes. I do this because there are a lot of vegetarians and vegans at my potlucks. YMMV.