OK, let me try again. As you look around in all directions, your total field of view takes up 41,253 square degrees. If you go outside on a normal sunny day, 0.2 square degrees of that field of view will be taken up by a 6000 K body (the Sun), while all the rest of it is cold. That leads to an equilibrium temperature of a few hundred K.
With mirrors, lenses, or the like, you can increase the angular area that’s taken up by that 6000 K body. And if you’re sloppy (low-quality mirrors, say), you can decrease the effective temperature of that body (let’s assume that your mirrors are high-quality, so that’s negligible). But what you can’t ever do is increase the temperature of that body.
If you increase the angular area of the Sun to the max, instead of taking up just 0.2 square degrees of the sky, it’ll take up all 41,253 square degrees. With the Sun filling up 41,253 square degrees, you’ll come to an equilibrium temperature of 6000 K, the same as the temperature of the surface of the Sun. Use a smaller mirror setup, and you’ll get some amount of sky in between 0.2 square degrees and 41,253 square degrees, and hence a temperature somewhere between a few hundred K and 6000 K.
Now, let’s go back to our referee on the field. Before the game, he goes out and looks at Section A of the stands, and measures how much angular area that section of the stands takes up. If everyone in Section A aims a mirror at the ref, they can make the entire Section A look like the surface of the Sun. That’s not 41,253 square degrees, but it’s still a lot more than 0.2 square degrees, so the ref’s location is going to get pretty hot.
But if it’s only Section A that’s aiming at the ref, then that’s the limit. It doesn’t matter if sections G, H, I, J, and K, on the other side of the stadium, are all aiming their mirrors at Section A, for A to redirect that light to the ref. It can’t get any worse than all of Section A being sun-hot.