OK, I can’t believe I’m spending precious brain energy thinking about this, but here’s my thoughts:
I’m assuming that the threat is flying humanoids as part of a larger army with lots of ground troops, so you’ll still want traditional high stone walls.
Then, basically, treat the roofs the same way castles treat the outer walls: put stone between you and the attacker, and make sure that every outside edge of stone can be fired on by a defender. The fighting areas on top of the curtain walls will be roofed with stone, and have walls on the front and rear with loopholes to shoot through. Towers on the ends of the walls would allow fire on the roof of the wall. The tower’s fighting platform is also roofed in stone, with two turrents sticking up that can cover the tower’s roof as well as each other’s roofs. The roofs of the great hall or whatever would all be slightly below the level of a tower in the corner that can fire down on the main roof.
For castles that aren’t already all connected (for instance, the classic double ring of curtain walls), you’d have to have covered ways connecting them so troops can advance or retreat without being exposed from above. You’d have to make sure the ways are still exposed to fire from the inner walls, so attackers can’t use them for shelter if the outer wall is abandoned.
If for some reason you need a doorway to the roof, you do it the same way as the gates on the ground: the doorway is set back (in this case, down), so the door is covered by fire from the walls on either side. And probably, behind the door, there’s a stairway down ending in another strong door, again surrounded by walls with holes so that defenders on the other side can do nasty things to anyone trying to force open the second door.
You would want some kind of sally-port in case the defenders have enough flyers to match up with the attackers, but that could just be a door into a courtyard, not necessarily to the roof.
Roofs would all have drainage arranged so that noxious liquids (burning oil, essence of rotting animal, etc.) would be directed to safe places. Ideally roofs would also all be steep and slippery, but it’s more important that they be covered by missle fire. Slanted roofs to help deflect medium-sized rocks might be useful. If the enemy has a captive roc or large dragon that can drop very large rocks on the castle, then you do what real-life castles did when the attackers managed to build a giant trebuchet: either counter-attack to take it out before it gets used, or surrender.
I don’t think spikes would be very useful in most cases: flyers can just hover and put their foot down carefully next to the spike. And, historically, metal was way too expensive to use that way. After all, if metal was that cheap, real castles would have covered the ground approaches with spikes and barbed wire. But historically scarce metal was best put towards more arrowheads. Wire might be useful to protect a small key area, but mostly I think it would get in the way of defender’s fire as much as anything else (though I’m assuming attackers can hover or fly slowly. If you know they’re limited to fairly high speed flight, then wires and barrage balloons start making some sense).
I do like the idea of noxious fumes, though the vision obscuring effects of smoke generally hurt the defenders more than the attackers.
Yes… or at least “mainly winged humanoids”… and he later mentioned that “powerful magic is rare”… so your comments on cost, weight and fighting ability are probably rather more germane than mine,
