That reminds me of a great story. I had been sporting my Indy hat quite a bit for a several month period. One day, I was doing some housework, and for background noise I put in my Raiders DVD. My 3 year old daughter was toddling about the room, and paused to watch the show for a moment. It was the opening scene, where Indy whips the gun away from Baranga, and it shows his face for the first time. At that moment my daughter happily shrieked, "Daddy!!"
To answer your question, we've all (or most) had moments of self consciousness due to wearing hats and the inevitable comments, but like others have said, most of the comments I get are very positive, and frequently quite flattering. I've occasionally had "Indy" comments, or someone whistling the tune, but those are usually people I know, and I don't pay attention. I shrug and we both chuckle a bit.
Oddly, I find I am less self conscious wearing the hat with a jacket of some sort, usually in cooler weather, it just feels more natural.
You should see this jacket with an Indy hat, they look like they were made for eachother.
http://www.thefedoralounge.com/showthread.php?t=33546
Yes, that is me in the pic, btw