Going directly to the airport to book a flight at the airline counter itself, no prior reservation - is this really a thing?

I used to take the AA shuttle from Boston to NYC regularly, and they fly frequently enough that what you say is true, but all can book in advance, and certain times like 7 and 8 am will sometimes sell out.