In most modern compression codecs the data for each frame is more or less a list of changes from the previous frame. This means that to leap into the middle of the file the video system needs to “play” all frames up to that point so that it can derive the correct picture to begin modifying.
To stop this being a problem there’s usually a facility to embed complete keyframes along the length of the file that allow you to jump around. This comes with the disadvantage of increasing the length of the file, so it’s a step often left out when space is more important than convenience.
Whether a file without keyframes allows jumping like this depends on the kind of compression, codec version etc. Some won’t allow it at all while others will do odd things like building up the frame a bit at a time as things change.