Well, if you're a bounty hunter, you're outside the Imperial structure. Imperial officers kind of treat you with contempt (and keep right on doing so even after you've completed your class storyline), but you can treat them with contempt right back. If you're a Sith, the Empire belongs to you and, at least officially, all non-Force users are beneath you, though Grand Moffs still bully you while you're an apprentice, and you still have to worry at least slightly about being diplomatic until you're a Darth.

If you're an Imperial Agent, you're a citizen of the Empire, a member of a bureaucracy, and...as a non-Force user in the Empire the most you can ever aspire to, as far as relations with Sith go, is that you'll one day be able to bully apprentices and hope they either die or forget it happened before they rise high enough to pay you back.