Think of it this way. Two people enter into a contract, a rich man and a poor woman. The rich man promises to marry her. This means she will live in a huge house, have servants, wear expensive clothes and jewelry, and never go hungry.
If he breaks the contract, he takes all that away. At a time when most women could only get low paying jobs, this is a serious thing.
If he breaks it off for a good reason, fine. But if he breaks it off without a good reason, then the perception is he was just using the woman. Make her a lot of promises, get what he wants - sex, or just lots of attention. Then dump her and move onto the next.
With two rich people, it’s a little different. If the man breaks up with her, maybe she’s “not good enough” for him. Maybe her family isn’t good enough for him. Other rich men decide to play it safe, and not get involved with her.
Also, upper class marriages were sometimes called off because a skeleton was found in the closet. So if a man calls off his marriage without giving any reason, people may start gossiping. There’s something wrong with her family, don’t marry into it.