I don’t know why you think you’d be labeled. Are you the party using the phrase, or not using it, to uniformed military personnel, or persons you otherwise learn are serving or have served? My opinion is that few who are eligible to have this phrase, or something similar, spoken to them are expecting it, or demanding it, or feeling entitled to it.
It falls in roughly the same general category as calling someone in military service (or similar career paths, such as police officer or firefighter) a hero. Anecdotal, but my understanding is that most such would reject such an assessment of themselves. I’ve run across many examples of ‘I’m no hero.’ I’d be willing to bet that many of those could tell you about someone they did consider a hero – say, a buddy who didn’t make it back.
Sorry for the length; obviously, it’s a personal peeve of mine. Full disclosure: I’m an Air Force veteran, 20 years, now retired. Most of that time I was an air traffic controller. I served in Operation Desert Shield/Storm (the first Gulf war). The only time I ended up saving a life (possibly) was in the normal course of my duties, at a stateside base. A civilian plane was low on fuel, in very bad weather, needing guidance to a nearby civilian airport. Long story short (too late!), I got the pilot down on the military base (something not normally done). After-report said the amount of fuel in his tanks was ‘immeasurable.’ Did I save his life? Maybe. Did I extend extraordinary measures to do so? Not really; the procedures were already there, ready to be used. Was I a hero? I’d vehemently deny it every time. Now, if you asked that civilian pilot…
Final point: if you never use the phrase again, no one is likely to give it any thought. IMO, anyone who might resent not receiving it never deserved it in the first place.