Given the Australian experience, I don’t believe that compulsory voting helps third parties. There is a greater third-party presence in Australia, but that is helped by the use of the alternative vote in most lower houses, and of proportional representation in the Senate and in some state parliaments.
And I should explain what compulsory voting means in Australia.
It is compulsory to be on the electoral roll if you eligible – basically an Australian citizen over 18 years old and living in Australia – and the relevant government agencies take active steps to get people on the rolls. However, I’ve never heard of people being fined for that: if you are found to be eligible and not on the roll, you are given the form to fill out to fix that condition.
Once you are on the roll, at each election you are required to go to a polling place (or cast a postal vote), get a ballot paper, and put it in the ballot box. You are not actually required to mark the ballot paper, and you can’t be, since it’s a secret ballot. If you don’t do that, you get a please-explain letter, and I suspect that if you give a half-reasonable excuse, that’s as far as it goes. However, few people do get to pay a small fine.
So, in practice, you generally get 90% plus voting in Australian elections at all levels (federal, state and local), though it tends to be a bit less in local elections. (The three levels of elections are always held on different days.) In addition, there is usually an informal vote of about 5%, and most of the informal vote is clearly deliberate and not the result of a mistake in filling out the ballot paper: either the paper is blank, or the voter has written something like, “They’re all crooks.” (I’ve spent a lot of time as a scrutineer looking at informal ballot papers.)
So my guess is that most of those who are eligible but don’t vote in the US would vote for one of the two major parties, and most of the rest would cast an informal ballot if the system allowed that. (I don’t know if voting machines allow you to deliberately vote informal.) Third parties would get very little extra support.