There isn’t really an oscillating dipole that sits and resonates for some time. If there were, then your intuition would be correct that the system would emit radiation continuously at the transition frequency. The transition, though, is instantaneous, at least on the relevant time scale of h/(E2-E1). (And on any time scale we’ve ever probed. Atomic transitions are just a type of particle decay, and all particle decays we’ve probe are consistent with being instantaneous… You’re in state 1. Then bam!, you’re in state 2.)