Angkor is worth visiting for sure, but its a bit crowded and even touristy now. However, I enjoyed the temples spread out on the plain just outside of Pagan in Burma a lot more. Much more informal exploring on a bicycle and few if any locals around trying to sell you trinkets. And you are not too far away.
Non-stop flight time from Phnom Penh to Yangon is around 2 hours. You could do both in a week.