Are you sure? Post office ATMs close at 9 PM at the latest. After that point you can’t even use your post office ATM card at other ATMs. I’ve never seen an exception.
The one in Fukuoka (next to Hakata Train Station) is open 24 hours as I used it about 11:30 at night two weeks ago. From the road, the entrance is around the left side of the building.
In my experience, the conversion rates in Nagoya and Narita airports are vastly better than those in SFO or LAX. The rates I’ve gotten converting cash from USD to JPY in Narita were always cheaper than using my ATM or credit card when you take the bank fee into consideration. However, do not exchange at an American airport unless you’re willing to lose 20% of your money