Are you specifically referring to only grammar and lexicon? Because you mention other things in the OP, like pronunciation and orthography, as well as usage. For example:
[QUOTE=Sampiro]
I’d always assumed the lack of formal/informal and the lack of gender in English would be a welcome simplification to those learning it, but apparently it’s confusing.
[/quote]
This is something much more complex than grammar. Moreover, English does NOT lack formal/informal distinctions, it just doesn’t involve pronouns. In fact, the ways that English speakers mark register is rather complex and difficult to master (shifting tenses, modals, ect.). But if you’re just asking in general, I’d agree with:
My masters is in teaching ESL/applied linguistics, and I’d agree that this is probably what generalizes most to speakers of all other languages. Article use and specific pronunciations generally challenge speakers of specific language groups. However, phrasals pose to ALL L1 speakers especially difficult and idomatic grammatical (syntactic and collocative, i.e, separable vs. not separable, transitive or not) restrictions, but probably more challenging are the semantic dimensions.
Take, for example, knock off.
The base verb, knock, has a fairly straightforward meaning, but the phrasal knock off takes on a wide array of meanings, which have differing grammatical restrictions, depending on the meaning:
As a transitive:
*The mob knocked off Greasy Higgins shortly thereafter.
The mob knocked Greasy Higgins off shortly thereafter.
The mob knocked him off shortly thereafter.
*
However:
*The mob knocked off him shortly thereafter.
[* = not grammatical to native speakers]
Another meaning distinct meaning, which is also separable and transitive:
*They knocked $50 off from the price.
They knocked off $50 from the price.
*
Or:
The factory in Hebei Province knocked off the pirate DVDs in two weeks.
The factory in Hebei Province knocked the pirate DVDs off in two weeks.
The factory in Hebei Province knocked them off in two weeks.
BUT
*The factory in Hebei Province knocked off them in two weeks.
It also has transitive vs. transitive restriction:
*It’s late. Let’s knock off now. (intransitive)
It’s late. Why don’t we knock off work now. (transitive)
*
BUT:
**Yeah. Let’s knock it off now.
*Yeah. Let’s knock off it now.
*
This particular phrasal also plays a part in an important colloquial idiom in English:
*Hey! Knock that nonsense off, right now!
Hey! Knock off that nonsense, right now!
Hey! Knock it off, right now!
*
BUT:
**Hey! Knock off it, right now!
*
Phrasals are extremely important in conversational English, and in that way, they can mark informality when used in lieu of standard verb equivalents (knock off instead of reduce, quit, produce quickly, or kill.)
To complicate matters, it can be easy to confuse different phrasals:
*knock out
knock around
knock over
knock up
*
..and of course, all of these can in theory be used as nouns:
*The coat was a cheap knock-off from China.
That chick’s a real knock-out!
*
Of course, phrasals can be avoided simply by choosing the standard verb equivalent: (using instead reduce, quit, produce quickly, or kill. etc.)
With that in mind, I’d say the one thing that challenges all learners the most form ALL language backgrounds is the correct use of prepositions, in particular as they collocate with specific verbs. Working with international graduate students at UCLA I found that no matter how good their English was in other aspects, the use of prepositions was the last thing they mastered, simply because prepositions are so idiomatic.