I suppose that’s right. Instead of saying:
When they questioned my citizenship, I quickly pulled out my passport.
You could say:
*When they questioned my citizenship, I quick pulled out my passport.
Everyone will understand the idea. The problem is that syntax weighs in its ugly head. Syntax prods us to put an adjective before a noun. If this were German, it wouldn’t matter. But syntax itself starts to define what is an adjective vs. an adverb, in many cases:
*The fast car drove away.
The car drove away really fast.*
These two sentences don’t have the same meaning; the adjective and the adverb have the same form, but the syntax is what makes the difference.
Do we really want to give up the adverb?