That’s true, although all of the states have chosen to appoint their presidential electors via some sort of public election, and where states have chosen to hold elections, there are Constitutional protections about who is eligible to participate. (See the 15th, 19th, and 26th amendments.)
But what about primaries or caucuses? Do they count as “voting” for which the above amendments apply? The Voting Rights Act says that primaries do, and this has been upheld under 15th amendment grounds by the Supreme Court: Although primary elections are internal contests for party nominees, because the state takes an active roll in administering them and setting their standards, the Supreme Court held in Smith v. Allwright that a whites-only primary for the Texas Democratic Party violated the 15th amendment’s protection against racial discrimination at the polls.
Caucuses are more of a gray area, because the state government is not directly involved in administering or regulating them.