There are any number of things he could do to me. Maybe he couldn’t kick me out, but he could do his best to make sure I never finish. He could sabotage my job prospects (he’s a MAJOR deal in his field, lots of books and articles to his name) if he does let me graduate. I don’t think he would hurt me physically, but he’s certainly capable.
I was often like this; I keep looking out for signs or signals to measure the ‘strength’ of the relationship because I am not sure if my supervisor (boss or someone in a superior position over me, such as a friend I have come to rely on) really ‘likes’ me.
Though your supervisor could potentially hurt you, it is not a good idea to be obsessed over it. Is there any reason why he should?