Testing, diagnosing, and localizing faults in AI systems — from deep learning programs and libraries to machine translation and coreference resolution.
Deep learning programs fail in silent, non-crashing ways — a wrong loss function, a learning rate that's too large, or a mismatched activation. DeepFD frames diagnosis as a three-step learning problem — diagnostic feature extraction, fault diagnosis, and fault localization — correctly diagnosing 52% of faulty programs (vs. 27% for prior state-of-the-art) and localizing 42% (vs. 23%).
Without reference translations, how do you know a translator is wrong? SemMT round-trips a sentence through a translation system and measures the semantic distance between input and output — capturing logical and numeric constraints via regex and DFA — when meaning drifts, a bug is exposed. SemMT improves accuracy by 21% and F-score by 23% over prior work.
Coreference systems link pronouns to the entities they refer to. Crest builds metamorphic test pairs from constituency and dependency relations that must preserve the same coreference — with no labeled test set required — and flags a bug whenever the predicted links become inconsistent across the pair. On 1,000 CoNLL-2012 sentences, Crest reaches 100% precision, versus 63–75% for prior baselines.