Well, it serves us right for wanting to use a vernacular language like Hindi with somewhat fluid orthographic rules, instead of sticking to Sanskrit where everything’s consistent!
What Aanamika is talking about is the alphabetic symbol for the “r”-sound in Indo-Aryan languages, which in written Hindi has several different representations. When it appears by itself it has one glyph, but when it’s combined with other consonants it has one of two other glyphs, depending how the combination is arranged. And sometimes the independent-form glyph is used, along with a diacritical mark that cancels vocalization, instead of one of the combination glyphs! So yeah, it can be confusing.
I can’t tell from the OP exactly what part of it is confusing you, but I assume you know that the “hook” above the line (“repha”) is used only when combining “r” with a following consonant. (If there’s a vowel-mark like long “a” or long “i” following that second consonant, it’s more proper to put the “r”-hook over that vowel than over the second consonant itself, at least that seems to be the way I usually see it.) And the sloping stroke that sticks out underneath and to the left is used only when combining “r” with a preceding consonant.
Easy-remember rule: “R-hook above a following consonant; R-stroke below a preceding consonant.”
When you are allowed to replace either of those with virama (the vocalization-cancel mark) plus the independent form of “r” isn’t clear to me. Personally, I never do if I can help it, but I started out with Sanskrit so the Sanskritized spellings are what look more natural to me.
Unfortunately, I haven’t been able to find an on-line reference that contains much more information about it than what I just said. I’m afraid your family is right that you need to be familiar with individual words (although IMO it’s more about looking at them than listening to them) to know for sure. Maybe somebody else will have a more helpful suggestion. Good luck!