I’m not going to vouch for the quality of this list, but if you look at the pictures in the link you will see that a po’ boy is not necessarily of the fried seafood variety.
It’s funny, as I’ve never actually had a seafood po’ boy the two times I’ve visited New Orleans. Both times I had non-shrimp and non-oyster versions of the sandwich (debris, ham & cheese, Italian sausage, IIRC). I’ve ironically only had shrimp po boys here in Chicago, funny enough.
I think the reason the seafood ones are the default for most people is because those are the most interesting and different sandwiches you’d get versus your local sub/hoagie shop. I like shrimp po boys fine, but I prefer my shrimp in non-sandwich form. It just doesn’t work as well for me on a sandwich. I may give the oysters a shot next time I’m in town, though.