Precisely because the British army was an institution that had to clothe large numbers of - arguably - typical working class recruits in a standardised manner, military records are a major source for historians trying to trace changes in stature amongst UK males in exactly this period. In their survey of Health and Disease in Britain: From Prehistory to the Present Day (Sutton, 2003), the archaeologists Charlotte Roberts and Margaret Cox partially rely on several studies by Roderick Floud derived from this evidence for average male heights in the last couple of hundred years. In keeping with your eyeballing, Floud concludes that the mean height of a recruit was about 5’5" (165 cm) in the mid-18th century, rising to 5’9" (175 cm) by 1950.
However, for the late-medieval period (1050-1550) the summary figure derived by Roberts and Cox (Table 5.11) for average male height from skeletal excavations across multiple sites is 5’7" (171 cm). They actually don’t see much overall change in height across the period from the Bronze Age, though they do see fluctuations up and down over that timescale - including the rise traced by Flout - and believe that these are tied to changes in nutrition.