I suggest you spend more time more frequently in the shower - maybe use some perfumed soap and as far as the hat goes... clean it with Saddle Soap and treat the outside with Mink Oil both of which have a pleasant fragrance. you can also wash your leathers with plain soap and warm water... but you might want to put it on a head form or wear it until it dries so it don't shrink.
I have a leather hat, different brand, that does the same thing and these methods have worked for me.
Also... get more than one hat and trade out often so you don't wear the same hat all the time and you will find the odor factor to be much reduced.
But, if you find you have to wear the same hat all the time... then you'll just have to put something pleasant under your nose when you take the hat off..