Like I said, drawing districts was not seen as a Federal concern, just as say, the employer employee relationship was not seen as a Federal concern. However the text of the commerce clause clearly makes that an area the Federal government can involve itself in if it choose, it just took well over 100 years before the Federal government exercised this power. That’s because of changing opinions about the proper role of the Federal government.
In the 1780s, it was not seen as a Federal concern how districts were drawn. In fact, House of Representatives members were not even necessarily elected by district originally as there was nothing requiring the modern day district system. The constitution specified only that they be elected by the people.
Something that must be understood if you’re going to talk about the early United States, is the primacy and power of the Federal government was not always respected or even real. While Federal supremacy in areas where lawmaking overlapped was clearly specified in the constitution, the reality was often far different. While not quite the same, it bears some semblance to the points in history where the Holy Roman Empire had weak centralized power. The Emperor and the Imperial Diet might pass laws that in theory applied to all, but any of the sovereign princes could ignore them, and most did.
The specific history of Article 1, Section 4 legislation highlights this point. Not until 1842 did Congress exercise this power at all, with legislation requiring House members be elected to represent drawn districts. This was a new thing, and as I said in the 1780s and 1790s there was no standard process nor did they have to be elected to represent a district.
Congress built on those requirements with more legislation that specified the nature of the districts, stated they needed to be compact to a certain degree, that they had to be contiguous, and other things.
These standards were done away with when Congress passed the 1929 apportionment act. Some of them probably would have made sense even today, as I do not believe there are currently any compactness standards that exist as Federal law.
Now, the reason they were deleted is the old mid-19th century laws were seen as a worthless historical relic. The reason? They were never enforced a single time in their history. The country was moving towards universally having drawn districts anyway, so that eventually became a norm, but district drawing and gerrymandering were just as popular then as now and continued unabated.
The reason is the mechanism of enforcement. Congress could pass laws stating this is how you have to elect Representatives, but a State could choose to ignore those laws and still send an elected Representative to Washington in defiance of Federal rules on district drawing. At that point, the House has sole discretion in looking at an incoming member and saying “alright, we agree the way you were elected comports with our requirements, and we welcome you to this body.” Or they can say the opposite, and get the hell out of here, you have no seat, and your constituents will remain unrepresented until your seat is filled in a manner we have deemed acceptable (through legislation and the process where the House reviews incoming candidates to make sure they met with those requirements in their election.) This doesn’t come up too often in the modern world, because we have much higher scrutiny of election law and the States don’t generally try this stuff. Additionally, modern law actually criminalizes some forms of election monkeying, so the State officials that would be involved in the modern era would face criminal charges.
It was brought up in the Rod Blagojevich scandal, there was initial opposition to letting Roland Burris be seated in the Senate because Blagojevich’s extreme corruption was considered to make any selection by him potentially invalid. Just like the House, the Senate has discretion in seating members and determining if someone showing up saying they’re a new Senator from Illinois actually meets the requirements the Senate and Federal legislation specifies for such a Senator. (This only makes sense, as theoretically otherwise me or you could just walk into either House of Congress claiming we had won election to a seat.) Burris ended up being seated, after some debate.
Back to the beginning of laws on districts in 1842, House members continued to be elected from districts that clearly violated those laws and regulations. But no challenge in the House ever once succeeded in preventing a single one of these elected Representatives from being seated in the House, so the laws had no power. This is generally a function of neither party being particularly serious about fair elections or fairly drawn districts. The next attempt to assert Federal power dealt with Senate elections and how State legislative deadlocks were causing problems with getting Senators seated. These regulations actually tended to work and improved things.
After that, you had laws in the 1870s to enforce the Fifteenth Amendment, these were powerless and never seriously enforced once reconstruction ended. Another example of how, earlier in our history Federal laws often were not worth the paper they were written on. Come to the 20th century and you generally see laws that actually were enforced, some to this day. Laws regulating contributions to political activities, stronger Voting Rights acts, a new apportionment act to replace the never-enforced laws from the 1840s and etc.