The Large Hadron Collider at CERN sends clumps of protons around a 17-mile circular path in both directions. When each clump of protons approaches each experiment’s location, the protons are deflected so as to cross paths exactly in the middle of the detector (CMS shown in the link).
Each proton that runs into a proton heading in the other direction is an independent trial. Protons are made up of quarks and the gluons that bind them together, so each trial is really a collision of a whole mess of particles at once, and the result is generally rather messy. The most interesting cases are where a single quark or gluon from each proton takes the brunt of the collision, and these two particles interact to produce a heavier particle. This heavier particle is usually an unstable one that decays almost immediately, and the decay products are detected (along with any spray from the “spectator” pieces of the protons) by the surrounding detector elements.
The production of a simple Standard Model Higgs boson at the LHC should happen in about 1-billionth of the proton collisions. The fact that it happens rarely is annoying enough, but it’s not the real issue. The real issue is that for every Higgs you produce, there have been a billion other collision producing other things that could mimic the Higgs’ decay pattern. The detectors and the analyses are designed to minimize this identification confusion, but in the end, almost all of the decays that look just like a Higgs decay are in fact not a Higgs decay. (The jargon is that these are “background events”.) As you say, you need to build up a very large number of trials to notice the tiny extra decays due to the presence of the occasional Higgs particle on top of all the uninteresting background.
Nature throws us a bone, though. If you had to just count Higgs-like decays and try to tell if you got more than you “should”, then we’d be miles from discovery. However, if you can measure the momenta of all the decay products, then you can calculate the mass of the particle that decayed into them. You can then look to see not just that you got extra Higgs-like decays but that these extra ones pile up at a particular calculated mass. This is very powerful is demonstrating that a new particle (and not a statistical fluctuation) is the source of the extra decays.
Here are the CMS and ATLAS plots that show the number of Higgs-like decays they see as a function of the mass they calculate for the parent particle using the observed decay products. Of note is that the background events lead to a wide smear of calculated masses. This isn’t because it’s a bunch of differently massed particles producing them but rather that the daughter particles detected in each background case are usually an incomplete set, which causes you to calculate a sort of random answer for the mass. The little bump of extra events you can see in the plot tells you both that there is something extra happening and what the mass of the parent particle is. And the figure shows that CMS and ATLAS see completely consistent values for the mass of this new particle.
These figures are showing only the cases where the Higgs was seen decaying into two photons. This is the easiest decay mode for removing background events since there isn’t as much that can mimic it. (Although as the plots show, there’s still plenty of background!) But unfortunately, it’s also a very unlikely way for the Higgs to decay. Take a look at this plot. It shows all the different ways a Standard Model Higgs can decay (colored lines) and the fraction of time it would decay to each (vertical axis), as a function of the mass of the Higgs. We now know the mass to be around 125 GeV, so we can read off how often it should decay to, say, a charm/anticharm quark pair (labeled “cc”, with a line over one of the c’s) or anything else. The two-photon case is labeled “gamma-gamma” and is way at the bottom. At 125 GeV, it is only 0.2% of the decays. But, it’s still the easiest one to use since all the others have tons of unrelated processes that mimic them, making them very hard needles to find in their respective haystacks.
Each search for one of these decay channels is an independent effort requiring somewhat different approaches. And, while the two-photon case is the most powerful, the others do help (with ZZ being the next best), and so the experiments combine the statistical power from each to get the best final answer possible. Also, a key question is whether the observed rate of decay into each of these different daughter particles follows the pattern expected for a Standard Model Higgs. So far it is all consistent, but the statistical power is very low in most of the decay modes so it is too early to say for sure.
