Basically, yes, index funds dump their shares of the departing stock and buy shares of the new stock. This affects both index funds and actively-managed funds that have investment policies saying they can only invest in S&P 500 stocks (a much smaller and shrinking group).
This pressure to buy stocks being included in the S&P 500 index leads to a well-studied phenomenon known as “the S&P 500 effect.” Standard & Poors announces index changes about a week in advance. From the time of the announcement that a stock is being added to the S&P 500 until it is actually added, the stock tends to go up in price about 3% compared to other stocks in the index. There is also some tendency for the stock to underperform after it is added to the index but the speed and certainty of this change is not as clear.
Because the S&P 500 is value-weighted, and because stocks added to the index tend to have very low market values compared to stocks already in the index, the effects of a single substitution aren’t that large. For example, Gartner is 0.05% of the S&P 500. If an index fund got completely on the wrong side of this substitution and bought Gartner for its portfolio only after the full average 3% price change occurred, it would underperform by 0.0015% compared to how it would have done if it bought Gartner the moment before the announcement. This is a pretty small effect.
But, most index funds are smarter than this. They don’t wait until the substitution is actually made to buy Gartner-they start buying on the announcement before the price has fully adjusted to the S&P 500 effect. Furthermore, index funds also invest in futures and options which are automatically adjusted to reflect new components in the underlying index. These derivatives aren’t affected by the S&P 500 effect at all. As a result, the 0.0015% can be thought of as a worst-case scenario. Actual funds mostly do better than this.
Finally, the effect as I’ve described it is really a chance for an index fund to do better than the index. Because the index change is announced in advance, the effect on Gartner’s stock price happens before it is added to the index. If the fund is smart enough to capture even part of the index effect from substitution, it will outperform its index by buying Gartner for less than the price at which Gartner is included in the index. This is means that the fund’s performance differ from the performance of the index, which is called tracking error, but since, in this case, the tracking error is making the fund investors more money, no one really complains. In fact, tracking error from substitutions and further tracking error from constantly arbitraging options and futures means that index funds tend to have raw performance (not including management fees) roughly equal to (and sometimes better than) the underlying index even though the fund also has to pay transaction costs to invest.
I don’t want to overstate the ability of index funds to outperform their index. Index funds bear management fees plus the S&P 500 effect isn’t purely on their side. Although index funds can generally benefit from the index effect when buying Gartner in our example, they will similarly lose when selling Dun & Bradstreet. However, even with this, because new stocks substituted into the index are usually bigger than the stocks they replace in these types of non-merger, non-bankruptcy substitutions, the likelihood is that the fund has more potential to make money on the positive S&P 500 effect than it loses on the smaller stock.