Tradition is part of the answer, technology, another part and snobbery still another part.
The tradition part was due to Frederick the Great who instaured “drill” and achieved tremendous successes. So everybody copied his methods.
Then a little bit later came Napoleon, with his variants on these methods (shock columns, massed artillery, skirmishers, etc.) and since he was even more successful, his methods were copied.
Now you have to remember that 19th century America, was pretty much the boondocks of the world stage. And to show that they were up-to-date, Americans took Napoleonic studies very seriously at West Point where most of the main leaders of the Civil War studied.
Now here comes the technology part, until the middle of the 19th century, most muskets were smoothbore, which limited their range and accuracy, so it made sense to stay close together so as to deliver the maximum fire at a range of about fifty yards from the ennemy and then charge, leaving him with maybe one, maybe two shots.
Now at the time of the Civil War, the smoothbores were being replaced by rifled muskets, extending the range by a factor of 5, so the previous 2 shots became 10. And a new bullet was introduced, the Minié ball.
This explains part of the reason for the casualties level.
Another reason is that a large amount of the officer corps on both side was made of amateurs, who learned their trade “on-the-job” so to speak, and their prime teachers were the small bands of professional who had studied Napoleon at West Point, hence the mass groupings.
As the Civil War went on new tactics emerged, but not enough and not widely distributed and not well studied at the time. So, in cases of doubt, they reverted to the old standards, because they didn’t have anything else.
As for WWI, one of the reasons it went that way was because of the intellectual snobbism(?) of the Western armies. They could have studied the American Civil War, the Russo-Japanese war of 1904-05 for the effects of massed fire and fortifications, but they ignored them, after all what did these people (Americans, Russians, Japanese) knew about war ?