I think **Lemur886 ** and **Blake ** touched upon the real reason but it has got lost in the concentration upon firepower and weapon technology - as is the want of history buffs everywhere.
Now those factors are important of course but more important surely are the factors of command and control. Until very recently the mass of your army would be uneducated, relatively poorly training and relatively unwilling to be there. Your command and control of your men at individual unit level would depend on the technology of flags (“here is where the unit should be”), drums and other instruments to make a centre of noise (“follow the sound of the drums”) but also crucially to transmit orders (this drum pattern means form a square against cavalry, that pattern means charge) and the sight, example (“follow me”) and shouted orders of the officers and non-commissioned officers.
All of that could only be done at relatively short range. The proportion of available offcers to men was low and so you had to gather your men in close masses in order to get them to do what you wanted. Otherwise they would do nothing, do what they were last doing, or run away.
Firepower was the constraint on the exercise of that control and command. If and when you got slaughtered by being too close together you either took cover and/or spread out (both losing command and control ability) or if close enough tried to cover the distance between you and who was mowing you down to eliminate that source of firepower. You could then reform your unit to regain command and control and enable you to do something with it again.
Battles were generally won by the side that maintained command and control and morale (all are linked) which were eroded by disorganisation, losses, smoke, noise etc etc. Firepower itself was not decisive until it created one or more of those factors to reach a critical level.
As education levels rose, individual initiative followed and a professional army become possible. When the techology of command and control improved faster than the techology of killing the situation changed. So the nadir was perhaps WW1 when you had airplanes, artillery, gas, machine guns, rapid fire rifles, mines, barbed wire etc etc to kill or restrain versus only shouted orders, whistles, phone lines which immediately broke down and carrier pidgeons to command and control your men. The result of a slaughter that even when the men achieved their objectives could not be exploited due to the lack of knowledge up the chain of command of the local situation - the battlefield had had to spread out as reaction to overwheming firepower despite the lack of any mechanism to control the resultant battlefield.
Come WW2 and radio things changed radically.