I have always understood “It’s down to you” to mean “There’s no one else left who can do this job; we’re relying on you to pull it off.”
Recently, however, I have become aware that there might be another meaning, perhaps in British English: “It’s my fault or the result of my doing.”
Is this another American v. British English usage difference?
Both seem correct given certain contexts.
‘We’re trapped. The escape tunnel is too small for anyone but Hodges. Hodges, it’s down to you.’
Or…
‘We’re trapped. Only one of us was here in the room with the big red button that says “PUSHING THIS BUTTON WILL CAUSE A CAVE-IN”. Jenkins, you were in the kitchen. Wallace, you were in the toilet. I was reading a manual. Hodges, it’s down to you.’
It sounds like the same meaning. Imagine a list of all the people in the situation. Each one is eliminated until you get to the one person who either can do the job or must be the one at fault. Ergo, it (the list) is down to that person.
These are both in the sense of “the list has been narrowed down to you.” The British sense is illustrated in the Rolling Stones’ song “Under My Thumb”:
In that sense, it means “It’s (written) down (next) to me (on a notional list of names where we also keep track of accomplishments)”. Compare “chalk that up to me” or “put me down for that.” The difference between usages is distinct: being singled out for something, or wanting to take the credit for something.
“It’s up to you” is perhaps the more common expression for ‘you’re the only one left’
Jurph, I immediately thought of Jagger when I saw the title of the OP, but I have to admit, I’d never interpreted the line the way you do. However, it suddenly makes sense. It’s because of me, it’s my doing, it’s my accomplishment, it’s down to me if she’s changed her ways, her style, her attitude and the clothes she wears. I always thought of It’s down to me as a kind of synonym for under my thumb, and as a twist on it’s UP to me – the clothes she wears, the way she speaks, it’s up to me, she does what I say, she’s under my thumb.
Anyway, it’s alright now. Thanks for setting me straight.
In the good old days there used to be a distinction:
It’s up to you - it is your decision, you choose; it’s no big deal: Shall I go by bus or train? Well, that’s up to you.
It’s down to you - it is your responsibility/fault: Who made a hames* of that? That’s down to Jim.
Nowadays, relative few people actually use “up to you” because they are too busy pretending to be American (see my rants elsewhere about sked-yule, mass-oose, sneakers et cetera, ad nauseam).
But this doesn’t reflect American usage. As illustrated by Jurph –
It’s up to you - It’s your decision
It’s down to you:
------ (U.S.) You are the only one remaining
------ (U.K.) It’s your responsibility
Sure, in some circumstances, the meanings might overlap, but they actually mean two different things.
What seosamh said. Those are the ways I would use each phrase. (In fact I am quite unlikely to use “down to you” at all, really.)
I don’t think I had heard of “hames” before, though. Ooh, all this ignorance. 
It’s probably an Irish saying: my mum, from Co. Laois, used it all the time. Hames are part of a horse’s harness, the “two curved bars of a draught horse’s collar” it says here. I have a feeling that the phrase appears somewhere in James Joyce’s Ulysses.