I don’t see how that’s the case. Again, the math works out such that you’re better off with the one box even if the computer is only 50.5% likely to predict correctly.
Let’s say that the computer is not infallible, just a pretty good judge of character (on this one very specific question). It can correctly predict how many boxes someone will take only 75% of the time.
There are four possible scenarios:
A - the computer guessed wrong; I take two boxes. I get $1,100,000.
B - the computer guessed right; I take two boxes. I get $1,000.
C - the computer guessed wrong; I take one box. I get 0 dollars.
D - the computer guessed right; I take one box. I get 1,000,000 dollars.
The only thing that I get to decide is how many boxes I take.
If I take two boxes, and the computer is right 75% of the time, then my expected value with two boxes is (.25)(1,100,000) + (.75)(1,000) or $275,750.
If I take one box, and the computer is right 75% of the time, then my expected value with one box is (.75)(1,000,000) + (.25)(0) or $750,000.
So even if the computer is not infallible, just decent at judging character, it’s better to go with one box.