No. You would morally justified in helping him/her escape. If you feared white-lead reprisals from doing that, then just simply get/stay out of the slave’s way.
You try to stop/kill them? Then you are no longer even ‘an innocent bystander’ in my book; you even become a justifiable target (‘fair game’) for any anti-slaver wishing to make an example/statement.