There are some nuances here to be aware of.
First, if you use the ‘subwoofer out’ jack on your stereo, you’re getting a line-level signal that contains only low frequency information for the sub. When you’re in dolby digital 5.1 mode, your receiver will send the “LFE” channel to the subwoofer through the RCA jack. LFE stands for ‘low frequency effects’, and it’s a special channel in 5.1 for explosions, vibrations, and other low frequency goodness.
This is the prefered setup for watching movies in 5.1 surround.
If you want to use this output, you cannot feed this into the speaker wires in your wall and hook it up to the speaker terminals on the sub. You need to run a proper coax from the sub and plug it into the sub’s RCA jack.
If you want to run speaker-level stuff to the sub, then typically the proper way to do it is to run your left-right speakers to the sub, then run wires from the sub to your main speakers. The sub has a crossover in it that will filter out the high frequency signals and pass the low level stuff to the sub amp, and filter out the low frequency stuff and pass it to your speakers.
Throwing another wrinkle into the mix are the speaker settings on your receiver. On most modern receivers, there are setting for all four non-sub speakers. You can set them to be ‘small’ or ‘large’. If you set them to be small, the full frequency sounds of music and such will be send to every speaker. If you set them to ‘small’, the receiver will filter out the low frequency stuff and send it out the LFE channel to the sub, and send only the higher frequencies to the main speakers. This has several advantages. First, small speakers are lousy at recreating bass, so it’s better to let the sub do it. Second, it takes a load off the amplifier because bass frequencies eat the most amount of power. The sub has its own amp, so by setting all your speaker to ‘small’ and using the LFE output on the receiver, your system will probably sound better. But then you have to run the RCA out to the sub.
If you run your sub from the speaker outputs, you must set your speaker size to large, or your sub won’t get any bass frequencies at all. If you put the sub in parallel with your other speakers, rather than connecting your other speakers to the outputs of the sub, you’ll be sending full-range signals to all speaker, which is usually not optimal.
For most stereos, the short answer is that you should run a proper coax from the LFE output of your stereo to your sub, then connect your other speakers normally. Set your speaker sizes to ‘small’, and make sure you configure the receiver so that it knows there’s a sub on the LFE channel.
If your sub has a crossover control on it, and you feed in the LFE channel, just set the crossover up to the max. Or you can set it at 80hz, which I believe is the THX recommended setting for the subwoofer crossover - just in case your stereo isn’t doing it.