That's a great question, Spencer. In my opinion, there is no difference. These words have been used so often in advertising just about everything over the years that I really think that they've lost their meaning in terms of marketing a product.
Rather than calling something gourmet or premium or specialty, especially as it pertains to coffee, I think adjectives that are more pertinent are fresh, full-bodied, rich, and rare; but only if they are combined with statements about the bean's origin or growing conditions....such as Kenyan, Jamaican Blue, Kona and rich volcanic soils.
For me, slap a label on a bean's box or a menu board that says 100% Jamaican Blue or 100% Kona and you'll get my attention far faster than you will by calling the coffee gourmet.
I agree 100% with Matt. Gourmet, premium, and specialty mean nothing to me -- tell me what's in my coffee, and where it's from, and I can tell on my own whether it's something special or not.