Irrelevant, since you put the fence on his property instead.
You are lucky he is not making you pay for the new fence.
Seems fair to me, if the trees are totally inside his boundary then thats it.
How can he install a new fence with trees in the way ?