I guess I was OK with the world being mystical and like a fantasy with religious artifacts and whatnot, but switching over to Sci-Fi with Aliens seemed way wrong to me. Plus, compared with the weight of the kinds of stories surrounding the Bible, (Ark, Holy Grail), having them look for Crystal Skulls was weak sauce. Why not just have him find Santa Claus or Big Foot?
Also, if the movie was really good up to that point and had earned the aliens I might have been more OK with it - but I had reached my breaking point by then based on the shit sandwich I had put up with already.