Grout swelling between bricks on fireplace?

I have a 33 year old house, apparently built on the cheap. There was only 4" of insulation in the attic, for instance, and I doubt the walls are much better. Anyway, I have a brick fireplace which we do not use. We had a draft-blocking baffle professionally installed at the top of the chimney, and glass doors that further seal out heat and cold. That was about 5 years ago.

In certain places on the outside of my fireplace - but inside the house, so the stuff falls onto the hearth - the grout is swelling (about 3/8" out from normal) and the swollen parts flake away. Wherever it has flaked, the remaining grout has “flowered” with a thin layer of white stuff, like calcium creeping through.

Does anybody know why this is happening? Why it would happen at certain spots, and not other spots?

I do not know of any damage to the chimney from the last few earthquakes. No visible cracks. We do not feel cold air coming in anywhere it shouldn’t, either.

Last…is this a structural problem we need to address soon, or a cosmetic problem only?

Calcium?
Seems too be excess moisture…
But what do I know?
I live in Hawaii and watch too many home improvement shows! :wink: