I’m not sure what the rules are in your area, but there is a legal doctrine called “adverse possession” that may apply in your case, even if the neighbour can show that it was originally his land. He has been aware of your posession for 20 years or more and has not exercised his rights to remove you.