hanaSEru is the potential form of hanaSU. It means “can speak”.
Actually it kind of means “can be spoken” even though it’s not a passive form. For example, you know those little sachets of silica gel that say “do not eat”? In Japan they say “taberaremasen” meaning “can not eat” but it would be clearer to say “can not be eaten”.
Anyway the point I was trying to make earlier is that it should be “nihongo GA hanasemasen” and not “nihongo O hanasemasen”. I was always taught that you say “X GA dekiru” rather than “X O dekiru”. (Dekiru is the potential form of “suru”, to do.)
But as others have pointed out, under some circumstances people do say “nihongo o hanasemasen” and there’s a slightly different nuance. So I guess it’s not so simple, but hey, that’s Japanese.