How easy was it for citizens of the Confederacy to Defect (& Vice Versa?)

I was wondering this today whilst kicking around an idea for a short story on the plane; namely, if someone (a private citizen, for example) living in the Confederate States of America decides they want to leave and move to the USA (having seen which way the wind is blowing).

How easy to acheive would this have been? Was it simply a matter of selling one’s stuff and heading North, or was there a “Border” of sorts preventing people from doing this?

Similarly, what about a Pro-Slavery Northerner who supported the South and wanted to move there- how easy to achieve would this have actually been?

there was a system of passes which operated throughout much of the war. Jones, Rebel War Clerk’s Diary often adverts to it, usually unfavourably. Men of military age probably wouldn’t get one.
You could also transit through a foreign port, e.g. Bermuda, if you could run the blockade.

Come to think of it, Jones himself was a pro-slavery Northerner who moved south when the war began. He still had property in the North which his relatives and his agent managed for him.

There was somewhat free movement across the borders on both directions during the war. The closer you were to actual fighting, of course, the more difficult - and dangerous - it would be. Spies, smugglers and refugees were often in motion, one way or another. A Confederate War Department diarist wrote towards the end of the war that internal dissension was just as great a danger to the Confederacy as the Union armies; many in the Tenn. and N.C. hill country were very anti-Confederacy and were treated harshly. When both North and South adopted conscription, it gave many men an additional incentive to beat feet.