Yes, I’d say actually worrying about dying in a house fire is absurd. Do the sensible things you can do to avoid it, mainly installing smoke alarms, and then stop worrying.
Do you think people should be checking every heating and electrical connection obsessively before they go to bed, and every time they get back to the house? Because that would be the equivalent to the level of caution you’re advocating about stranger rape.