Inside Our ProgramProgram Events
Events
-
Jul29
EVENT DETAILS
Artificial intelligence (AI) is increasingly important to decision-making across various domains, e.g., financial planning, disease diagnosis, etc. While applied methods excel at identifying specific phenomena and challenges within tested environments, theoretical methods provide rigorous guarantees across all possible scenarios, including those that may be rare or even adversarial. This worst-case guarantee is even more important in the modern AI workflow, where upstream AIs are provided as a service to downstream decision-making instead of being designed for an application.
This thesis evaluates and designs AIs out of the context of a particular application, through mechanism design and statistical decision theory that models rational decisions under uncertainty. At the core of the theoretical framework is proper scoring rules, the class of functions that evaluate a probabilistic prediction by the decision payoff it leads to. In mechanism design, proper scoring rules are mechanisms that elicit truthful predictions from a strategic agent who optimizes for expected score (McCarthy, 1956; Savage, 1971). In statistical decision theory and machine learning, proper scoring rules (a.k.a. proper losses) evaluate and reward predictions by their decision payoff (Gneiting and Raftery, 2007), which can be equivalently viewed as eliciting predictions from AIs or learning algorithms.
This thesis defense will be divided into two parts. Part I evaluates and improves the trustworthiness of predictions for downstream decision-making, from a statistical decision-theoretic perspective and an information elicitation perspective. Part II demonstrates the applications of the theoretical framework, including the design of a provably truthful text elicitation mechanism and a statistical decision-theoretic benchmark for human-AI interaction.
TIME Tuesday, July 29, 2025 at 1:00 PM - 3:00 PM
LOCATION Mudd Hall ( formerly Seeley G. Mudd Library) map it
CONTACT Wynante R Charles wynante.charles@northwestern.edu EMAIL
CALENDAR Department of Computer Science (CS)
-
Jul31
EVENT DETAILS
Data analysis processes are regularly employed to inform high-stakes decisions. However, typical workflows for implementing data analysis overlook the considerable subjectivity that is baked into the analysis process and how that might impact the results. This subjectivity reflects ontological uncertainty---implicit, qualitative uncertainty regarding how data should be analysed and modelled. While statisticians have proposed techniques such as multiverse analysis to surface implicit ontological uncertainty, analysts currently do not possess the tools to implement, evaluate and communicate the results of such analyses. This dissertation bridges that gap by providing a pipeline for systematically reasoning about ontological uncertainty that is implicit in data analysis. In the first study, I developed and evaluated multiverse, an R library, which lowers the barrier to implementing multiverse analysis. The library provides flexible and expressive syntax to allow analysts to declare any alternative data analysis step through local changes in code. The library is designed to integrate into both computational notebook and scripting data analysis workflows, and optimises execution by pruning redundant computations. . I evaluate how the multiverse R library supports programming multiverse analyses using (a) principles of cognitive ergonomics, and (b) case studies based on semi-structured interviews with researchers who have successfully implemented an end-to-end analysis using multiverse. I identified design trade-offs (e.g., increased flexibility versus learnability), and suggested future directions for supporting analysts in adopting multiverse analyses (e.g., how to evaluate a multiverse analysis?). In the second study, I address the issues of evaluation by first identifying principles for validating the composition of, and interpreting the uncertainty in, the results of a multiverse analysis. I designed Milliways, a novel interactive visualisation system, to support the principled validation and interpretation of multiverse analyses. Milliways provides interlinked panels presenting result distributions, individual analysis composition, multiverse code specification, and data summaries. In the third study, I compare the two different approaches for depicting ontological uncertainty---ensembles and p-boxes---by conducting experiments to investigate the impact of the visual representation on how the multiple uncertainty distributions are interpreted. Based on these results, I identified how the results of multiverse analyses should be visualised so that viewers adopt the desired (possibilistic) interpretation of ontological uncertainty. Together, these three studies outline a systematic approach for surfacing, reasoning about, and communicating ontological uncertainty that is often implicit in data analysis processes.
TIME Thursday, July 31, 2025 at 2:00 PM - 4:00 PM
LOCATION ITW, Ford Motor Company Engineering Design Center map it
CONTACT Wynante R Charles wynante.charles@northwestern.edu EMAIL
CALENDAR Department of Computer Science (CS)
-
Sep25
EVENT DETAILS
TBA
TIME Thursday, September 25, 2025 at 9:00 AM - 11:00 AM
CONTACT Wynante R Charles wynante.charles@northwestern.edu EMAIL
CALENDAR Department of Computer Science (CS)
