The origins of the fez, or "tarboosh" in Moroccan, is not clear. The design may have come from ancient Greece or the Balkans. In the 19th Century it gained wide acceptance when the Ottoman rulers moved to modernize traditional costumes. The brimless hat did not get in the way of a Muslim's daily prayers and was cleaner and less cumbersome than the turban. The name fez is believed to come from Fez, the city, which once produced the hat's red dye, made from crimson berries.
In some regions, the fez was historically considered the "hat of the oppressors" and wearing it was deemed politically incorrect. Morocco however has resisted the onslaught against the fez (it is noteworthy that the Ottomans never actually controlled Morocco) and the king of Morocco is the only Arab leader to wear it. Wearing a fez in Morocco is a nationalist statement and was seen as a protest against the French occupation. So, like many hats, the fez is a charged symbol with a long and complicated history.
In modern America, we associate the fez with the Shriners, Abu from Aladdin, or Akbar and Jeff of the "Life Is Hell" comic strip.