Probably they were, then. There are a lot of people growing them either only as ornamentals or for the seed.
Near as I can tell, laws on growing Papaver somniferum (the opium poppy) and its ornamental cultivars vary by state. It seems doubtful that anyone would be prosecuted just for growing them for garden ornament, though there are anecdotes about grannies getting busted. Online sources vary widely on legality.
I don’t grow P. somniferum mainly because its flowering period is so short.
If you want to get a rise out of neighbors (and maybe the law), grow Texas Star hibiscus, which is much hardier than described in the linked article. Out of flower, the foliage is bound to draw attention.
Hehe, I had a friend who worked in a nursery who would grow his weed among Texas Star hibiscus plants.
California poppy (Eschscholzia californica) isn’t even in the same genus as opium poppy. Lots of flowers are called “poppies”. True poppies commonly grown in gardens, beside Papaver somniforum (opium poppy) are mostly cultivars of Papaver rhoeas, the annual Flanders Field poppy. I have both kinds in my flower border. Opium poppies are insanely showy for a couple weeks and then they melt into a pile of brown.
Also, I haven’t ingested dope since I was sixteen, when I didn’t like it but smoked anyway because I was trying to act normal. Except I had some gummies once when I was bedridden after a fall off my horse. They didn’t do much. It’s legal to grow here but of no interest to me.
Similar here, putting up with the smell of the plants alone is a major lifestyle commitment. A neighbor two houses down has a lot of plants, and when the wind is right we can smell it from our own yard and even inside the house. I wouldn’t want to grow simply for the fact that I wouldn’t want my house/clothes/furniture to constantly smell like someone’s damp golden retriever 24/7.