Hatters did, indeed, go mad. They inhaled fumes from the mercury that was part of the process of making felt hats. Not recognizing the violent twitching and derangement as symptoms of a brain disorder, people made fun of affected hat makers, often treating them as drunkards. In the U.S., the condition was called the "Danbury shakes." (Danbury, Connecticut, was a hat making center.) Mercury is no longer used in the felting process: hat making - and hat makers - are safe.