Tools → Options. Under the Preferences tab, click on E-mail Options → Advanced E-mail Options. There’s a check box about middle of the page that says “Show an envelope icon in the notification area” along with other notification options.
I was thinking of Outlook and missed that you had said Outlook Express. I have now searched through and been unable to find any such visible notification for Outlook Express, but I’m going to poke around a little longer. In the meantime, what version of OE do you have? And is it the same on both machines?
I have exactly the same version of OE, running on XP Service Pack 2, and it does the email notification without problems. I wonder if perhaps you have the “Hide inactive icons” property checked on the taskbar properties window, and this is making the icon invisible. That’s one possibility, the other is that it’s somehow permanently hidden.
To check that: Right-click the taskbar, click Properties, and then Customize under Notification Area. Scroll down to “You have new email” and set it to “Always Show”. Click your way back through the OK buttons.