Trustworthy Classification for Complex Social Surveys: A Memory-Enhanced Hierarchical Framework with Calibrated Uncertainty

Abstract

Automated classification of complex social survey questionnaires is crucial for large-scale social science research but faces significant reliability challenges due to intricate hierarchical label structures, severe class imbalance, semantic ambiguity, and incomplete data coverage. Conventional classification methods often struggle with these combined complexities, yielding results that lack trustworthiness. We introduce HOCM, a framework designed for trustworthy classification in complex, real-world taxonomies. It features two synergistic components: (1) memory-enhanced contrastive learning, tailored to learn robust representations from noisy, imbalanced data by leveraging quality-aware category memory banks; and (2) hierarchical uncertainty calibration, which enforces taxonomic consistency while providing reliable confidence estimates and identifying inputs falling outside well-represented known categories. Our evaluation on a large-scale, real-world social survey dataset—a challenging exemplar of our target problem class—demonstrates that HOCM maintains strong accuracy on known classes while effectively identifying uncertain cases, significantly boosting accuracy on confident predictions. Furthermore, it adeptly detects low-resource/unknown categories. HOCM provides a more reliable automated classification tool, enabling efficient expert review and enhancing the trustworthiness of analysis in domains with complex, hierarchical data.

Contribution highlights

• We introduce HOCM, a framework designed for trustworthy classification in complex, real-world taxonomies.

Extracted from the public abstract; no claims have been added.

Code and data

Publication metadata

DOI

10.1609/aaai.v40i46.41291

Volume

40

Issue

46

Pages / article

39415-39422

Version relationship

No separate preprint-to-publication relationship is currently recorded.

BibTeX

@inproceedings{Wang2026TrustworthyClassification,
  title = {Trustworthy Classification for Complex Social Surveys: A Memory-Enhanced Hierarchical Framework with Calibrated Uncertainty},
  author = {Zeqiang Wang and Rebecca Oldroyd and Yuqi Wang and Jiageng Wu and Jie Yang and Wei Wang and Nishanth R Sastry and Jon Johnson and Suparna De},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume = {40},
  number = {46},
  pages = {39415--39422},
  year = {2026},
  doi = {10.1609/aaai.v40i46.41291},
  url = {https://ojs.aaai.org/index.php/AAAI/article/view/41291}
}