Am I allowed to simply move to say Puerto Rico or Guam and set up shop? Do I need to “immigrate” there and fill out paperwork or as a US citizen can I just go there and leave it at that? Can they (the possesions) kick me out (deport me) if I’m not desirable or break the local laws or whatnot?
“I guess one person can make a difference, although most of the time they probably shouldn’t.”
Yes.
The latter.
No.
Works for me 
I thought maybe there was some sort of tax paperwork at least to sort through.
“I guess one person can make a difference, although most of the time they probably shouldn’t.”
Nope, other than: when you move from, say NY to CA, you would have to fill out part-time resident tax forms; in this case, you’d fill out the tax form for the Feds and send it to a special IRS address, and the first full-residency year file taxes with the territory instead of the Feds - check with www.irs.gov for details, as they change somewhat from territory to territory.
For example, Guam is pretty simple, and the tax rate is more or less the same as the Fed. Saipan is much less…
O le mea a tamaali’i fa’asala, a o le mea a tufanua fa’alumaina.