From my BIL, who works for the Vermont Historical Society, and posted with his permission:
The population table posted by [samclem] does not tell the whole story. One has to look at both what was going in the rest of the country and what was happening within the state. Vermont failed to keep pace with national growth. It went from six U.S. representatives in 1812 to five in 1822 and to four in 1842. The 1850 census showed that 42 percent of all native-born Vermonters lived outside the state. Within the state a persistent exodus from the land began as early as 1810. In 1840, Vermontโs population increased by 4% but five of her rural counties lost population. As other contributors have observed, Vermont was not an easy land to farm.