Generally, I want the thinnest amount of clothes that can keep me the warmest. Specifically, I need a new jacket and a new pair of gloves.
What should I look for? Are there certain fabrics that are better at insulation than others?
With clothes, think in terms of layers. For instance, I’m currently wearing a tee shirt, a shirt, a jumper, a top, and a ski jacket. Each layer traps some warmth.
Gloves are a bit different, but, like a WW2 pilot, you can have silk under-gloves which will fit into other gloves.
My leather coat is not too heavy, and pretty much wind-proof. It has a removable liner. I’ve never found any gloves that actually keep my hands warm, though.
Mittens will keep your hands warmer than gloves with the obvious drawbacks of not being to use your fingers.
you aren’t in a real cold climate. a winter jacket with selected layers of clothes underneath should handle a wide temperature range. you can get a winter jacket with a zip out liner for a wide temperature range.
gloves with a knit or fur (real or fake) also trap much heat.
Stay away from cotton because it is very bad if it gets wet. Wet cotton will chill you down fast. If you might get wet the best way to stay warm is either wool or synthetics like fleece.
These days most people use fleece for jackets or down (which normally costs more than fleece) You can get fleece in various thicknesses , the thicker will keep you warmer but it costs more and weighs more.
But fleece isn’t synthetic . . . or is there a synthetic that’s also called fleece?
polar fleece is made from pete 1 plastics.
Gloves – leather with a Thinsulate lining. Fleece and wool are warm but the wind goes right through 'em. Corner glove stands usually have plenty of Thinsulate glove options.
For a coat your warmest thinnest option is probably a wool dress coat or wool pea coat. Always in style. Something like this:
http://oldnavy.gap.com/browse/product.do?cid=5286&vid=1&pid=803870
(the wind does NOT go through a wool coat because it is made of thick felted wool, not knitted wool like most gloves. Knitted wool has spaces. Felted wool is a solid fabric)
Alternatively, a windbreaker waterproof heavy nylon shell with a heavy fleece zip-out is very versatile. Basically like this:
http://www.llbean.com/llb/shop/65688?feat=504714-GN2
Insulation is a combination of trapped dead air and wind blocking. To that end, anything (wool, down, fleece, newspapers) that trap air will work. To a degree, insulation is directly related to bulk. You can reduce weight by using fabrics and materials that trap a lot of dead air without adding much bulk, and by having a good windproof layer on the outside.
It’s hard to beat the combination of inexpensive fleece with a windshell for light weight, low bulk, low cost. It’s hard to beat a down puffy jacket and a good Gore-Tex shell for bombproof warmth and protection.
For gloves, Thinsulite is a good choice since bulk is a problem for gloves. But as said above, mittens are much warmer in general.
Anything down filled (real down, not the big puffy feather filled) is light and warm. I survive my Canadian pre dawn winter morning dog walks wearing only a thin eddie bauer down filled
jacket with a windbreaker shell over top.
For gloves, thinsulate lined are nice.
What Telemark said…
yes, I meant synthetic fleece. There are many varieties you can buy - some are expensive which you can find at places like REI but you can also get basic stuff at Target which is decent.
Quoted for truth.
You should also think about the innermost layer. As Bijou Drains, says, cotton is probably the worst choice close to the body, since it get really clammy and uncomfortable with just a hint of moisture in it. And if you’re hiking or skiing, you will get a bit clammy on the inside because of perspiration. Good synthetics like polypropylene are good for high levels of physical activity, since they don’t absorb moisture and the sweat is transported outwards to the middle and outer layer. OTOH, wool has unbeatable properties if you’re planning on sweating just a little bit and generally will keep a middle-to-low activity level, since it feels dry and isolates just as well even when it has absorbed some 30% of its own weight in moisture. And when you sit down to dig out your Thermos bottle, you’ll really appreciate wearing wool underwear instead of synthetics. It feels almost as if some of the sheep’s body heat is still lingering in the fabric 
As a windshell in temperatures well below freezing, a good, well-impregnated cotton anorak is actually one of the best choices even if it’s got a lower coolness factor than the fancy-design synthetics. If it’s 10-20 C below freezing and I’ll be moving around, I prefer my old cotton anorak to my Gore-Tex jackets as an outer layer. Good, impregnated cotton breathes a lot better than modern membranes like Gore-Tex (ETA: or the Windstopper type membranes), and the breathing ability isn’t that much compromised by moisture freezing on the inside (have you ever had to turn your Gore jacket inside out and shake/brush out all the frost that has accumulated on the inside? I have, on several occasions). But cotton is clearly inferior to modern membranes if you get wet. And when you sit down, you’ll appreciate the better windshell properties of the membrane jackets.
There’s always a compromise between windproofing/waterproofing and breathability. You can’t maximise both. A really windproof outer layer can’t have top breathability, while an outer layer that breathes really well can’t give the same windproofing.
Now, this is for when you’re moving around. For really low activity levels, like walking to the store or sitting down for a break, a thick fleece middle layer (or two) and a down jacket/parka is probably the best choice, and you’ll look like the Michelin man 
Can I recommend something specific?
The Patagonia Nano Puff is an ultralightweight (8 oz) synthetic down jacket. It looks good (to me, anyway), insulates well, blocks wind, and is amazingly comfortable. It packs into its own chest pocket if you ever want to take it off and stash it in a backpack or something. It also repels light to moderate rain, but you’d want to pair it with a rain shell if it really gets pouring (it has a water-resistant coating, but it’s not an actual breathable waterproof membrane like GoreTex or eVent).
It’s become my favorite item of clothing – and I never had a favorite before it. Mainly, it’s just great at what it does: providing invisible warmth. I don’t feel any bulkier with it on, just warmer.
And if you buy it from REI, you can wear it all you want (including in inclement weather) and still return it if you don’t like it (you soulless demon, you).
As for gloves, Thinsulate and Gore Windstopper are both good fabrics, but it’s the puffy insulation inside them that really counts. Ski gloves are one option (and you can get them in actual waterproof Gore material if you don’t mind the decrease in dexterity), but as mentioned above mittens will ultimately keep you warmer because all your fingers contribute heat to each other and warm up their shared airspace. You could also slide a hand warmer into the glove.
Oh, and speaking of REI, they have a great writeup on this very topic, including a breakdown of various fabric types for each layer:
How to Dress in Layers
And lastly (missed edit window)…
Keep in mind that if you’re just walking around town, you don’t really need fancy gear. The gear is great if you need a combination of waterproofing, windproofing, warmth, light weight and breathability, but if you can exclude one or more of the criteria, stuff becomes a lot easier to find (and a lot cheaper).
For example, waterproofing may not be too important if you’re usually indoors or traveling by car or under an umbrella. Breathability is important mainly when you’re active and sweating; eventually all that built-up moisture will chill you, soak your layers, and chill you further. Leather, for example, is usually waterproof and keeps you warm, but it isn’t particularly breathable and it weights a lot. Down will offer unparalleled warmth-to-weight ratios, but it’s useless once it gets wet. Etc.
You don’t mention it in your OP, but if you really want to stay warm, the best thing you can do is cover your head. A lot of body heat escapes through your head, and unless you have very thick hair, that’s where a lot of your warmth is going-out your skull! 
Seriously, invest in a decent hat. Makes a ton of difference. I even look for lounge/sleep shirts that have hoods. When it gets really cold, I cover my head instead of cranking up the heat. Works great!
(snipped)
I’d add to this that covering your ass is important. A wise old man (old to me, at the time… so probably 40 
) who I worked with outdoors for 15-20 hour/day stretches kindly informed me that my coat was too short.
I believe his exact words were "1/3 of your body heat escapes from your head. That’s great. The other 2/3 come from your ass.).
While the math is obviously not accurate, getting a longer coat (and slapping on Carhart coveralls) get me generally warmish through that awful winter.
This will be on my mom’s tombstone:
“It’s WINTER, put on your HAT!!!”
You must be related to attorneys, where “Cover Your Ass” is their credo!