I have rooms built over crawlspaces and they’re toasty warm in the winter.
I’m not an expert, so hopefully better-informed people will answer. I’ve faced some of the same problems over the years and often did the correct thing last, which means I’ve made a lot of mistakes before getting to the right answer.
When a room is too cold in the winter, it’s usually because the heating requirements weren’t sized correctly for the room, or the heat is not getting to the room.
In a most modern homes, which probably have duct work, it could be that a duct is blocked or there are not enough ducts/vents to bring the correct amount of warm air into the room to heat it properly.
Call in a HVAC expert for an assessment. They can get you started in the right direction. They can give you a quote for fixing it, or give you advice about how to fix it if it’s something you can do yourself.
As to attics, of course you want to insulate the hell out of them, especially in cold northern areas, but I find they are not a source of cold air infiltration into a room.
Yes, attics are cold and yes, heat rises. That means you are heating your attic, which you don’t want to do, but it doesn’t mean that the attic is what is making your room cold. You should insulate the attic aggressively when you can, but that probably won’t keep an under-heated room warmer. It will slow heat loss, which assumes you are getting the right amount of heat to begin with.
As to insulating a crawlspace - please don’t do it until you check with an expert in your area. Depending on where you live and the climate, it can cause big problems, like mold. Since your crawl spaces are underground and lead from the basement, they are probably not a source of the cold.
If your windows are not leaky or drafty, you can put off new windows for a bit. Do some googling on the R-values of new windows and you’ll see that new windows may not bring you much increase in efficiency. I don’t want to discourage the improvements, and you should get them if you want them, but they likely will not solve the problem with a cold room unless, again, the windows are very drafty.
Start with HVAC. Go from there.
Anyway, good luck. Working in a cold room is unpleasant. It’s like working in your refrigerator.