What is there to fight? I think you described how it works quite well. I suppose what you need to bear in mind is that it really is a matter of style over ‘correctness’. The purpose of language is to communicate effectively. If using ‘wrongly’ in those instances feels awkward, then the rule is - don’t use it. There really is no right and wrong.
The reason we have standards at all is to ease communication. This is especially useful in it’s written form, because standard grammar and spelling help you understand what is being written. Spoken language doesn’t really have rules, unless you’re speaking formally about a certain subject… If everybody understands what you mean clearly, then it’s right.
Stylistically I think most would agree that, “You accused him wrongly” and “You did it wrong” work best. Just because ‘wrong’ can be an adverb, doesn’t mean it has to be used as one all the time. Word categories aren’t set in stone. A noun can sometimes work as adjectives for example.