I admit to some reservations about Diamond Jim. However, you are all speaking from the perspective of people who live sedentary lives in residences and workplaces which are kept at optimum temperatures. And many of whom have to be careful not to gain weight (a battle I, too, fight).
One thing that automatically changes the dynamic is that at 40-45 degrees latitude in these here United States, a century or more ago, the warmest parts of rooms - next to the fireplace or pot-bellied stove, from around October 15th to April 30th might be around 80 degrees, but the coldest parts would more likely be in the 55-60 degree range. And many people did without heat - especially in the daytime - from April 15th until summer, and in the fall until mid to late October. Coal was expensive, and so was wood, unless you cut it yourself. If you cut it, that took considerable exertion, and burned up lots of calories, probably more than you’d burn by just eating more food instead.
In those circumstances, it took a lot more calories just to keep body temperatures up. And that doesn’t take into consideration the time spent traveling. When traveling by horse and buggy, the face, at least, was exposed to the weather (closed carriages were not nearly as common in the Americas as they were in Europe). When traveling by train (and one assumes that Brady did lots of that), it was a little warmer, but not by any means the even, dependable ~70 degrees we expect. Some railroad cars would be heated (another potbelly stove), but most wouldn’t, as best I understand it (I am subject to correction by any expert on 19th century trains). And if I am correct, there would be the same temperature gradient as would be found in a room heated by the same means.
I remember vividly that my grandparents, who were born in the 19th century, were very fond of fat (yes, the non-lean part of meat). Why? It’s the highest calorie density you can get, and people used to worry about getting enough calories.
For a more modern example, one set of cousins grew up on a farm in the 1940s and 50s. The father held a job, so the boys did the farmwork, planting and harvesting the cash crops, tending the livestock (cattle, hogs, sheep, chickens), and also tending the garden, where much of the family’s (vegetable) food was grown. As teenagers, the middle cousin (who is an excellent cook, who sometimes does gourmet dishes, so don’t imagine he doesn’t understand any of the concepts) has told me that he and his older brother usually ate at least 6,000 calories per day (four large meals, plus snacks) during planting and harvest (often more, in harvest; the work was harder). And they stayed skinny. How? They were working flat out from sunup till time to go to school, and from the time they got home from school until past sundown.
I, too, have trouble imagining that Brady ate as much as the stories relate in the height of summer (say June 15 to September 1), but in cold weather, he probably could manage to put away at least 3,000 calories three times a day. Dunno about the rest.