Yeah, that’s true. I would have pretended no-one’s home or no-one’s awake in a situation like that. Saves embarrassment, otherwise they know that you don’t want to talk to them.
So, have you gone over and knocked on his door since to find out what he wanted?
You could make up an excuse, “Sorry I didn’t open the door, but I had, uh, company, if you know what I mean.”