Can someone get your real name, or at least google account, from a google voice call?

I called and texted someone through my google voice account thinking that it’d give me some degree of anonymity (long story, not doing anything wrong) and they were able to figure out my name somehow from it. From my understanding, google voice will just display your google voice number on any sort of caller ID when you call with it.

Is there some way for them to get my google account name or anything like that from the google number, or did they find it some other way? That same google account that I use for voice does also have a gmail and g+ account, and those would have my name. But what are the steps from going from google voice number to getting a name?

It’s a little important to know if they got it through the number or some other way.

One thought: have you tried Googling your GV number to see what comes up? Is it possible that you’ve used the number in an online classified ad or something similar?