Like EvilGhandi pointed out, many fishes can swim upside down. In most teleost (bony) fishes, the swim bladder is located above the digestive tract, but this may or may not place it above the center of gravity of the fish.
Only if the swim bladder is above the center of gravity of the fish will it have to ‘fight’ the effects of its swim bladder in order to swim upside down. Even then, most fishes can control the gases in their swim bladder and could release gases to make this easier.
There’s a whole bunch of other reasons fishes don’t typically swim upside-down, though. They do often have semicircular canals, which help them detect which direction the force of gravity operates in. Swimming upside-down for long periods of time may result in the same sort of discomfort a terrestrial animal would feel doing headstands for a long period of time. Fishes often show countershading which enables them to hide better when they are swimming rightside-up, but will make them more easily visible when upside-down. Fishes also often have a mouth position which makes feeding easier in the rightside-up position; for example, surface feeders may have a mouth on the dorsal side of the head while bottom feeders may have a mouth on the ventral side. Swimming upside-down would make using this advantage more difficult. Schooling fish might have trouble following and imitating the actions of the school if they are inverted relative to their schoolmates, in much the same way Calvin has trouble when inverted relative to Mrs. Wormwood and her class… or not. Finally, swimming upside-down is, like, soooo '90s. All the cool fish these days are learning how to hula-hoop.