Who are refs talking to in a football game?

When you see a ref announcing a call in a football game, they appear to be facing a sideline. Are they addressing the coach of the team that got the penalty? Or do they just address some neutral point? Or is a television thing and they’re directly addressing the camera?

The press box side, which not by coincidence became the side where they set up TV cameras for broadcasting games.

The refs used to do their signals to both sides of the stadium. Here’s a Monday Night NFL game from 1971, on a play with a chop block penalty (I think). The ref didn’t have a microphone, hand signals only. And no arrows on the yard numbers. And goal posts on the goal line. And Howard Cosell.

Does anyone else think the players seem tiny in comparison to modern day players? Old school television had its advantages- you had to pay attention. No yellow line to show the first down, no constant reminder of the clock, down, yards to go, or the score for that matter And somehow, we managed to keep track of it all.