The pork pie hat was a staple of the British man-about-town style for many years. Pork pie hats are often associated with jazz, blues and ska musicians and fans. Charles Mingus wrote an elegy for jazz saxophone great Lester Young called “Goodbye Pork Pie Hat”. Many artists have performed this tune, including Jeff Beck, Rahsaan Roland Kirk, and Joni Mitchell. In Jamaica, the hat was popularized by the 1960s rude boy subculture, which traveled to the United Kingdom and influenced the mod and skinhead subcultures (although Jamaican and British pork pie hats are more similar to a very short-brimmed trilby rather than the US style). Jamaican ska artist Laurel Aitken performed the song "Give Me Back My Pork Pie Hat".

Robert Oppenheimer, the physicist known as the father of the atomic bomb frequently wore a pork pie hat. Singer Dean Martin was known to be partial to pork pie hats, and they became a trademark of the silent film comedian Buster Keaton who handmade his own.