In my opinion it’s mostly grammar and registers, with a heaping helping of writing on the side.
For one thing, Japanese is an SOV language, whereas English is SVO. There are more languages which use SOV word order than SVO, but English only uses that kind of construction for poetic license (or when talking like Yoda you want) but even then the structure is dissimilar to SOV languages since English uses prepositions instead of post-positional markers, and meaning is often coded in word order, as opposed to being more free-form and using various markers to clarify meaning.
One of the things that still trips me up sometimes is that the extra information comes before the thing you’re explaining, rather than after. For example, “The car I bought yesterday is green.” In Japanese order that would be something like, “Yesterday bought car green is.” You have to remember that the details come before the topic, and sometimes my mouth runs away with the subject before my brain remembers to switch the order, so the sentence is all garbled and back-asswards.
Going the other way, I’ve gotten lost because the person takes forever to get to the bloody subject (if it’s even stated instead of being left out or implied) and I’ve forgotten what the point is by then. At least I’m not completely alone there. Native speakers sometimes lose track of what the subject is; I’ve personally seen it happen a few times.
Then you get into the levels of politeness and formality that people mentioned before. Keigo and sonkeigo can be so complicated that native speakers have to formally study to use them properly. I actually have a better grasp of it than some of the teenagers and college-aged people around me. Employers in the service industries typically have etiquette classes for new employees to review proper polite forms in speech, among other things.
The only similarity Chinese and Japanese have is the writing, which was an import superimposed upon a language that is from a completely different family of languages from the Han-related languages. One of the reasons there are so many pronunciations for the characters is because this fundamental difference between the languages, and because there were a couple of different time periods when they were adopted.
Since the War, simplifications have changed the respective written forms, so many of the characters are not mutually intelligible anymore. Some characters are combinations that exist only in Japan. Things written with the old pre-War characters are very roughly understandable across the languages, but you have to know the old forms, which many younger people do not.
I don’t have any personal knowledge of Chinese, but from what I understand, Chinese uses the same word order as English most of the time, and the grammar is reportedly less complicated in some respects; no definite/indefinite articles, no plural forms. There are also many fewer ways to read characters. I believe that most characters have only one possible reading in Mandarin.
Sage Rat’s example for Japanese is not atypical. The fewest number of readings is usually two, but there could be a bunch of different meanings, all of which depend on context or in-depth knowledge to get right.
And even then it’s guesswork. Names or regional pronunciations are sometimes so idiosyncratic that even native speakers won’t have a clue about how to read it without asking a local. My Japanese wife didn’t know how to read some of the names in the area I live in when she first came here, and while it’s rural I’m in the same general region as Tokyo. There’s a regular game show where (admittedly not very bright sometimes) celebrities have to guess the reading of various words, or write the proper characters from a pronunciation.
Even fairly well-educated people screw up occasionally, or just plain don’t know all the possible characters you could use in a situation. A friend of mine who worked in a Japanese company related how one of the guys who studied Chinese at university screws with co-workers’ heads by writing notes that no one else can read. Admittedly, that’s not too far off from deliberately writing something in academic or lawyerese, but in this case you wouldn’t even be able to sound out the words without a dictionary.
Oh yeah, and your dictionary would be organized in an arbitrary way that sometimes depends on a knowledge of the etymology and structure of the character. Have fun looking up something with a lost radical that’s listed in the traditional category instead of the simplified version.