Our Approach
Multiple Independent Annotators
Each record is labeled by multiple independent annotators. When their responses diverge, we manually review and resolve the discrepancies.Validation Sample Size
The validation sample size is proportionate to each category’s representation in the dataset:- Typically 1-5% of overall data
- Ensures adequate validation points for every category
Golden Dataset Methodology
A key aspect of our methodology is having annotators independently solve the task rather than validate model outputs. This approach:- Prevents annotator bias
- Creates a “golden dataset” of correct answers
- Enables benchmarking of new model outputs across iterations without requiring fresh human validation each time
Why This Matters
This approach is particularly effective for accurately classifying permit descriptions, which often contain:- Industry-specific terminology
- Abbreviations
- Inconsistent formatting
Accuracy Results
Our case study on using specialist participants for data labeling shows how we achieved 98% accuracy in our classifications by incorporating a panel of experts from the construction industry.Learn more in our blog post on data labeling with construction industry specialists.