Depends. As you stated, adding energy to something in orbit (and everything from earth has orbital energy) makes it fly “higher”. Considering that Earth is about 93 million miles from the sun, and the earth orbits the sun in 365 days, you can see that earths speed around the sun is pretty fast.
That in mind, it would take a heck of a rocket firing in opposition to earths orbit to slow it to a speed that would allow it to sink to venus’s orbit. Not only that, but it would take forever to get there. After all you wouldn’t want to slow it to a speed that was too slow (getting it there quickly), or you’d need another huge burst of power to speed it back up to venus’ orbital speed.
The solution, in simple terms, is to send it outward. By carefully calculating orbital patterns, you can send it to mars, have it slingshot around, then make minor burns near earth, then have it slip into orbit around venus. By going to mars, it sort of bleeds speed from the rocket, at which point you can keep the speed slow enough to get to venus without using huge amounts of fuel. (You send it around mars “backwards”, thereby obtaining the loss of energy without using fuel).
The upside is that if you miss, assuming you don’t bury the probe deep into venus, it takes very little energy to redo the trip and try again. The downside is that it’s complicated.
Basically, the same thing is done to get to the outer planets as well, but in reverse. Instead of coming in close to mars or another object (we use the earth itself more often than mars, building the probe’s energy by repeated slingshots, but mars is a better visual aid), and leaving by a distant orbit, you come in far away (diving for the planet), and leave when you’re really close to the atmosphere. This gives you a tremendous speed boost.
The last downside of these slingshots, besides time, is that they are hard on spacecraft. It takes a fraction of time to get an unmanned probe to mars compared to a human, mostly because doing the same to a human would mean turning that person into mush with the extra G’s.