Yikes. You can probably build yourself an academic career figuring out the evolution of the Roman army, and all the whats and whys. I haven’t done that, so I’m certainly no expert. Paging Kobal2, I suppose. And if we get into it, we’ll be here all week. It’ll be a fun week, but kind of off-topic. Still, for those specific questions:
The reorganization from old-school legions into border troops / mobile field armies (which happens around the time of Diocletian and Constantine) is probably just a sensible change, considering what the army’s job is at that point. Which is: Mostly being border police along the (absolutely humongous) frontier, and then responding the fastest with the mostest to where that week’s crisis is. This is a different sort of job from, say, marching into a territory to conquer it. By the time of Diocletian, the army was probably already largely operating as vexillations, that is, parts of legions peeled off from the main legion ad hoc, rather than as full legions all the time. The limitanii / comitatenses thing is probably just a formalization of that into the army structure. And, again, the grouping into smaller forces rather than huge legions (although groups of about 1,000 soldiers were confusingly still called legions for a long time) is just reshuffling for flexibility. The field armies they were combined into would be 20,000 - 30,000 men strong, and the army as a whole was just as big when the fifth century got going as in the glory days of the Empire, as least as far as I can tell.
Another reason to put your main forces into concentrated field armies, and turning your border troops into a thin line of lighter troops (rather than having a legion for every x number of miles of frontier) is that you can then stick an Emperor in front of a field army. Especially in the third century, whenever a general won a battle (or, sometimes, successfully sneezed in the enemy’s general direction) his troops would go “General X for Emperor!”, and you’d have a usurpation or a civil war on. Better, if you’re Emperor, to be the Johnny On The Spot whenever a fire breaks out. This is part of the reason for the much larger reliance on cavalry in the later armies: They get to the fire real quick. The first Emperor to create a large mobile cavalry force is Gallienus, in the third century.
(Obviously, the Empire is still freaking big, so that’s not really going to work. Which is part of the reason for the various multi-Emporor or multi-Empire configurations that you get. The rest of the reason is probably another week-long topic to get into.)
As for the difference in look, weapons, helmets and armor from the old-style legions: I’m not entirely sure, to be honest, but my guess is that it has to do with who gets requited into the army these days. Which is: Germans. They would probably come with fighting styles and preferred gear already, and the army would adapt to them rather than the other way around. That is mostly a guess on my part, though, so y’all can feel free to correct me on it.
Another thing that happens to the army over time is that the difference between legions and auxiliaries disappear. It all just becomes the army, and, as far as I can tell, the later army is more or less all made up of what would all be auxiliaries in the olden days. Italians sure don’t want to be in the army anymore. I suppose that when the specialized jobs, manpower base, gear and tactics of the old-style auxiliaries get folded into the main force, then that leads to the army changing.