Guideline clarity
Most annotation mistakes trace back to an unclear guideline, not a careless annotator. If two reasonable people can read the same instruction and label the same item two different ways, the guideline is the problem, not the person applying it. A good guideline gives a clear rule for the common case and a written answer for the edge cases that come up often.
We write the guideline with you before labeling starts, and it gets updated the moment a new edge case shows up, not weeks later.
Inter-annotator agreement
This is the rate at which two or more annotators label the same item the same way. Low agreement usually means the guideline is ambiguous, not that the annotators are careless. Ask a vendor whether they measure this, and on what kind of task. Some tasks, like "is this a cat," have naturally high agreement, and some, like "is this comment sarcastic," don't, no matter how good the guideline is.
We track disagreement inside a batch and route it back into the guideline, so the same ambiguity doesn't keep costing time.
Edge cases
Every real dataset has items that don't fit the obvious categories cleanly: a photo half in shadow, a sentence that reads as sarcastic one way and sincere another, a speaker who switches languages mid-sentence. What matters isn't avoiding these, since they're unavoidable, but having a documented answer for them and applying it consistently once it exists.
Edge cases get written into the guideline as they come up, so the same call gets made the same way for the rest of the batch.
Review layers
A single annotator, working alone with no check, will make mistakes that a second pass would catch. Ask whether a vendor reviews every item, a sample, or nothing at all, and who does the reviewing: an annotator with more experience, or the same person who labeled it.
Every batch of ours goes through multi-level review before it reaches you.
How to check quality before you commit
Give a small, representative sample of your actual data, not a vendor's own showcase example, to two or three vendors with the same instructions. Compare the results against your own judgment of what correct looks like, not just against each other.
We do a small sample free, on your data, before you commit to anything larger.
Warning signs
Be cautious of a vendor who quotes a flat accuracy number with no explanation of how it's measured, can't describe their review process, or treats every edge case as a one-off instead of writing it into a guideline.