Feng Shui uses a mechanic it calls “swerve”. You roll two d6 - one is a positive die, and one is a negative die. You add the numbers together (remember, one is a negative number), and the result is your “swerve”. In addition, both dice can “explode” - if you roll a 6, you roll again and add the result. Again, though, one die is a negative, so you can explode in either direction - or in both simultaneously!
I don’t think this is quite what you’re looking for, but Fantasy Flight Games’ “Genesys” system, which first appeared in their licensed Star Wars RPG, uses a variety of specialized dice with different symbols on the faces, instead of numbers. The symbols interact with each other in various ways, with some symbols paired so that one of the pair cancels out the other.
There are a lot of board games that have specialized dice like this, but in those you’re usually just looking for a particular symbol, not having some symbols cancel out others.
FUDGE, a freeware RPG that led to the popular FATE system from Evil Hat Games, uses another system with odd dice. FUDGE/FATE dice are six sided, but instead of numbers, they have two “+” faces, two “-” faces, and two blank faces. You roll (usually) 4dF, and the + and - cancel each other out, and you use the net + or - as your roll result (-4 to +4).
The old Alternity RPG from TSR used a d20 as a “Control die” - you always rolled a d20. But then, instead of static modifiers, you had a step ladder of die types, going through all the D&D polyhedrals, d4 / d6 / d8 / d10 / d12 / d20 / 2d20 / 3d20. The ladder also went in the opposite direction: -d4 / -d6 / etc, with the die result subtracted from your control die. If you had modifiers, you moved up or down the ladder, and rolled your control die with the modifier die, and added the result.
I’ve seen a couple of indie RPGs (which I can’t remember the names of right now) which used something like the FUDGE/FATE or Genesys systems, but using regular dice. You have an Ability die and a Difficulty die. You roll both, and try to beat the result of the Difficulty die with your Ability die.
Savage Worlds is probably the most prominent example of the exploding die mechanic. It also uses a “Wild Die” - when you make a Trait roll, you also roll a d6, which can also “explode”, and use whichever result is better. SW is also a bit unusual in RPGs in that Traits (attributes and skills) aren’t static modifiers or levels or dice pools, they’re die types. So your Strength is a d4, d6, d8, d10, or d12.
Godlike used pools of d6s, and you looked for matches. But, rolls had two dimensions. The higher the number of a match (the roll’s “height”), the quicker you acted, but the more matches you got (the roll’s “breadth”), the more effectively you acted. So, for example, if you rolled 10 dice, and got results of {1, 1, 1, 1, 3, 3, 3, 4, 6, 6} you could choose to use the 4 x {1}, and act slowly but very effectively, take the 2 x {6}, and act very quickly but barely eek out a success, or split the difference and take the 3 x {3}.