How do I bring the Unusual Gem to an appraiser?
One of the things you can do is show it to Maul, the Nord who accosts you when you first enter Riften. He will direct you to Vex, in the Thieves Guild, who starts the next quest, No Stone Unturned.
You cannot access Vex without joining the Thieves Guild, so if you're interested in completing the quest, you'll have to join up.
Thanks, I had been wondering if there was a more legal path to take with the gems.
@KevinD well, ironically, none of the gems are marked owned, so taking the unusual gems themselves isn't considered stealing. You might have to do some B&E to get to some of them, though.
Nope. Depending on the path you take, that quest also has a sub-objective (non-"optional") of "Become a full member of the Thieves Guild." There's apparently no other way of pursuing that quest. (At least, not until TESVCS is released and some modder fixes it...)
I took it directly to Vex, but she wouldn't do business with "the likes of me". Was it because I killed the guys in the Ratway to get there? Or do I have to join the thieves guild? If so, how?
@Nilzor as I mention in my answer, you have to join the thieves guild first. Talk to Brynjolf in Riften and he'll start you on your way.