I have a B.S. in Chemical Engineering, and had to make the same choice as the OP.
I went with ChemE as an undergrad because I liked chemistry, and I thought an engineering degree would give me more flexibility for the future. I found out that that ChemE didn’t actually have a lot to do with pure chemistry. It’s more about unit processes (separations, mixing, chemical kinetics, mass and energy balances, thermodynamics, and mass, momentum, and heat transport). In my junior year, I briefly considered switching to a chemistry degree, but decided it would be almost as much work for a less prestigious and much less flexible degree (particularly as far as the Navy was concerned, as I was on a NROTC scholarship).
Overall, I found my ChemE degree program to be a real ass-kicker. Between reactor kinetics, transport, organic chemistry, physical chemistry, and partial differential equations, it’s not a cakewalk of a major.
In addition, I was actually pretty sick of the material by the time I finished my bachelor’s degree, so it was just as well that I went into the Navy as a nuclear submarine officer. (The degree did help me greatly in the Navy’s Nuclear Power School, however.) In any event, I never actually worked in the field as a practicing chemical engineer.
While in the Navy, the degree also allowed me to teach chemistry at the undergraduate level in a military prep school for several years. At the same time, I developed an interest in environmental engineering, which I got an M.S. degree in.
I’m now a practicing environmental engineer. I’ve taken a rather tortuous career path, but there you go.