Nobody wants to run an .exe file they downloaded off the internet. That’s obviously very risky.
Nonetheless, I want to make a self-extracting file available. But for the above reason, it can’t be an .exe file. Is there a non-proprietary self-extracting file format that won’t cause a user to immediately consider the risk factor of getting viruses (the way .exe might)?
A self-extracting file needs to be a format that can run.
Which means, anything self-extractable is also a potential virus vector.
On the Mac, the names of Self-Extracting Archives often end in “sea”. I suppose you could use Stuffit to make an SEA, then instruct your recipients to extract them inside their Macs or Mac emulators… 
Self-installation packages are .msi and .msp files. But I don’t know how one makes these.
You can play games with extensions, but that won’t placate the more intelligent of your audience: By their very nature, self-extracting files have to contain executable code, and all executable code is a potential vector. Going from .exe to .sea or .msp or .foo won’t change that fact.
It comes down to trust: Do your patrons trust you? They obviously don’t trust their OS to protect them, so do they trust your files to do what you say they’ll do and nothing more? Unless that trust is there, the problem remains.
This is a social problem, not a technical one.
hm. that’s disappointing. well, thanks!
Your other option is to not make it self-extracting. You could, for instance, distribute a zip file. While a zip file can contain a virus (heck, it can contain anything), it won’t automatically run the virus when you open it. The downside is that your users must already have a program to open zip files. But this shouldn’t be too big a deal: Recent computers are almost guaranteed to already have a program on them for dealing with zip files, possibly even built into the operating system.
Yes, any good virus program can scan a zip before opening it.