Hand-curated natural language (NL) systems use explicit linguistic knowledge to understand and generate text. This gives them an advantage over pure machine learning (ML) systems in terms of transparency and scrutability: Their knowledge can be inspected, debugged, and improved stably and incrementally over time without the need for retraining. However, maintaining this knowledge can be a challenge. Hand-curated NL systems use specialized forms of knowledge that frequently only a limited number of experts know how to interpret and debug. For a broad-coverage system, the bottleneck is often the time and attention of the few developers who can help it grow.

My thesis addresses this bottleneck by presenting two new techniques that enable non-expert users to debug and extend hand-curated NL systems. The first, Interactive Natural Language Debugging (INLD), concentrates on debugging individual sentences, asking the user simple questions in natural language to drill down on any underlying errors in the system's linguistic knowledge. The second, Generative Example-Driven Error Mining (GEDEM), concentrates on identifying likely errors at scale, asking the user for binary judgments that allow the system to flag potential errors for further review. Together, they reduce the need for expert knowledge to debug the system, instead relying on common linguistic intuition. In doing so, they expand the pool of users who can help maintain a hand-curated NL system.

I ground my discussion in the Companion cognitive architecture and CNLU, its broad-coverage semantic parser. I present implemented INLD and GEDEM systems for CNLU and evaluate them on two sets of examples, examining the reliability of user annotations and the usefulness of the systems' outputs. Lastly, I explore the possibility of automating these systems with large language models (LLMs).