I'm 5'8" and 135 lbs. I ordered a small and it fits pretty well. I like it a lot. It doesn't feel or look baggy on me at all. They really are great shirts.
Cabela's is fairly generous with their shirt lengths. Go by you usual shirt size, or your chest size if you know it. They won't be "slim cut" but they shouldn't be baggy if the chest sizing is correct.