Based on your description and how you found it, I would suspect sodium bicarbonate or one of the hydrates of sodium carbonate. Calcium carbonate is not very soluble, and tends to precipitate as hard mineral-like scales and crusts. It’s the carbonate that is basic, so that it dissolves in acid (or reacts with it) is not diagnostic of which mineral it is.
I wouldn’t call it “crap.” It’s a perfectly ordinary dissolved mineral, and water with dissolved minerals generally tastes better. Pure distilled water tastes flat and stale. If you had a wonderful spring of mountain water, it would be chock full of stuff like this from its sojourn underground. (Another way to put this is that if what you have is one of the carbonate or bicarbonate compounds of sodium, you got it from the interaction of ordinary table salt and dissolved carbon dioxide from the air. You can’t get more innocuous and natural than that.)
The stuff that you would really want to worry about in your water is not especially amenable to home treatment.
Your single largest concern would be living organisms, bacteria, tiny worms, protozoans, et cetera, of the type that cause typhoid, giardia, cholera, et cetera. There is unfortunately no way to tell by eye or taste that water is contaminated with these buggers – the water can look clean and sparkly and certainly leave no residue – but give you a terrible case of the runs, parasitic worms, or in the worst case if untreated kill you. It’s for this reason that the primary concern of water treatment is always to make it sterile – kill anything in it. Almost all other concerns are pretty secondary, in part because it isn’t all that easy to make water sterile cheaply while not making it taste awful or worse. Home remedies are expensive and inconvenient – boiling, for example, or very fine filtering – or make it taste bad, e.g. iodine.
Your next concern would probably be heavy metal contamination, e.g. small amounts of dissolved lead, mercury, arsenic, cadmium. These are unusual, but if you’re downstream from some giant industrial or mining operation in a third-world nation, that might be a problem. Again, these contaminants are impossible to detect by eye or taste, and even much homebrew chemistry. They’re not promptly dangerous, but exposure over time (years) can have significant bad effects, particularly on children.
Finally you might worry about organic contaminants in the sense of PCBs, pesticides, solvents, gasoline, et cetera, stuff that might get into the water stream by various kinds of industrial, commercial and even residential pollution. These however are often detectable by eye or taste at about the same level where they become dangerous, at least promptly dangerous. Few people will willingly drink water with enough dry cleaning fluid e in it to be actually a problem, since it would taste foul.