Two AIs Disagreed on 1 in 5 Test Enquiries. We Deleted Those.
Here is how we built the test for our lead-triage model. One AI wrote 440 invented estate-agency enquiries and labelled each one: buying, renting, selling, spam and so on. A second AI read every enquiry without seeing the first label, and labelled it again. Where the two disagreed, or the second wasn’t sure, we deleted the enquiry. That was 85 of the 440, nearly one in five. Then we tested our model on what was left and reported 89 %. ...