I will have to confess that I have never associated a fez with leisure.
Aside from its presence in movies on top of Sydney Greenstreet or Middle Eastern men, the only thing I think of when I see a fez is “Oh, the Shriners are in town”. The association, at least for me, is not with leisure, but with successful members of the working-class – ie: your typical Shriner.
Just like bow ties?
I own two fezes, one for the Shrine and one for fun. They are in fact cool.
As stated upstream, a fez was a handy way to keep cigar smoke out of your hair. But I cannot prove that.
A friend made me a fez for my birthday.
I must dig out of storage the red fez I picked up in either Tangiers or Istanbul when I was in the navy many crescents ago. And the Arab headgear, like on the Arabian princes, and the French navy hat with the red pom pom, and the Brit navy cap that I got in trades.
A Fez is definitely cool headgear, and it says (or at least it should) to the observer that the wearer is a well-travelled individual, who has sampled many cultures, and wants to look like they spend their weekends in Middle Eastern Bazaars and Coffeehouses having an exotic and slightly adventurous time. At least IMHO. 
Conversely, a fez-wearing Muslim can do the bowing part of Muslim prayer without taking off his hat.
The Shriners wear fezzes as a part of their faux-Muslim ceremonial, um, stuff. Few Shriners are actual Muslims; most are Christian Protestants. (I’m a former Shriner, and I mean them no ill will.)
Costumes? Garb? Ensembles?
Oh yeah! And fancy titles!
I never realized this; is this why, when Grandpa Simpson inherited a bunch of money, he went out and bought a fez? (On the Simpsons)
Probably more of a nod to Akbar and Jeff, two fez-wearing characters from Matt Groenigs other cartoon Life In Hell.
Do they have an Exalted Ruler?
The man elected to head a Shrine Temple is the Potentate, and the other elected officials, as a group, are his Divan. Those were the terms in Murat Temple, anyway. Paul in Qatar has been to more Temples than I was, and he probably has a greater depth of that knowledge than I.
Murat Temple has a number of “units”, some for parade performances. There are clowns (Shriners are very big on clowns,) a motorcycle unit, a horse parade unit, an all-banjo band, a Turkish marching band, an old-time jazz band, a parade unit of de-powered IndyCars, Hillbillies, Mud Buggies, Flying Fezzes, a blood-drive unit, a Transportation Unit to take children to the Shriners Children’s Free Hospitals, and probably several more I can’t remember.