It may be that the other family feels they don’t know you well enough yet. Or they could be nutty jerks. Who knows?
I recall one mom I barely knew by sight and did not know by name stopping my kids and I as we walked to school and asking from her car if the kids could have a sleep over. I said “no” and was unpopular with everyone, but I wasn’t even sure where this woman lived.
Then she did it again.
And again.
I finally invited her over for coffee and explained that I was fairly conservative with my kids and she needed to quit bringing up sleepovers until we knew each other better.
As you might imagine, this friendship did not go well. This woman had all kinds of boundary issues and strange behaviors.
Another thing I would consider at this point is developing some actual, trusted friendships with some other families.
You said you own your house. I assume you plan to stay there. Since it’s just the two of you it’s probably time to start developing a kid-friendly network if you don’t have one already.
School, scouting, and hobbies such as sports or music, etc., and even church membership (I can only go UU, myself) are an excellent ways to make connections with other families through group gatherings at events or each other houses for post-game/gymnastics/dance get-togethers. This will help both you and your daughter build up a community of like-minded folks you know and trust, and you will have some control over the kids your daughter gets to know and bond with.
Also, if you have not done so, do make an appearance at your daughter’s school when you can to see and be seen by other parents. You don’t have to be a ‘room mom’, but you can volunteer to help serve or clean up on ‘ice cream day’ or set up, host or clean up on 'field day. Get involved. Don’t show up just for ‘lunch with dad’ day.
Being an active parent will go a long way toward relieving the worries and stresses of the other parents and it will help you too by making you a ‘known quantity.’ You will find that other parents will be willing to help you out on that one day you can’t get to pick-up time on time, or forgot it was early release day, or missed adding money to your daughter’s lunch account, or whatever.
Really, you can’t have too many friends as a single parent. Parenting is hard enough with two parents as it is, so I can only imagine how hard it could be doing it all by oneself. Later on, in middle school and high school, as kids social lives become more complex, having build this network of solid, trusted friends will be a benefit to you and your daughter and you won’t be stuck with some random kid your daughter found and likes who has weird parents.