A New AI Framework to Reveal the Hidden Histories of Paintings

Northwestern Engineering’s Aggelos Katsaggelos is leading an international, interdisciplinary team developing a physics-informed AI framework to digitally isolate overlapping paint layers. Maria Kokkori, also at the McCormick School of Engineering, is a co-principal investigator on the project.

The project, “Hidden Paintings, Layered Histories: Reframing X-Ray Analysis,” is one of 19 projects receiving a combined $10 million from Schmidt Sciences through its Humanities and AI Virtual Institute (HAVI). The HAVI grants, announced in October,  support the application of new AI techniques to questions across art, history, archaeology, literature, music, and other humanities fields. Together with researchers at the Art Institute of Chicago and the Van Gogh Museum in Amsterdam, Katsaggelos and Kokkori plan to make their resulting tools open source and available for viewing through a public art exhibition. 

“We are grateful to Schmidt Sciences for supporting research that advances both AI and humanities scholarship,” said Katsaggelos, the Joseph Cummings Professor of Electrical and Computer Engineering and co-director of the Center for Scientific Studies in the Arts. “This funding will enable us to develop new physics-informed AI methods that help scholars reconstruct the hidden layers of paintings and better understand the choices artists made. Beyond cultural heritage, these methods could help solve complex scientific problems across a wide range of disciplines.”

For more than a century, X-rays have allowed art historians and conservators to look beneath the visible surface of paintings to discover abandoned compositions, changes made by artists, and earlier works concealed beneath layers of paint.

But everything the X-ray passes through, including the visible painting, hidden layers, canvas, stretcher, and other structural materials, is compressed into a single two-dimensional image. The new project aims to separate those overlapping layers using AI, providing a clearer picture of how paintings took shape and changed over time.

“Traditional X-rays allow us to know something is hidden without knowing exactly which layer belongs to what, how the layers relate, or the sequence in which they were created,” said Kokkori, an associate research professor of electrical and computer engineering at Northwestern Engineering and senior scientist at the Center for Scientific Studies in the Arts. “Our goal in this project is to move from simply detecting a hidden history to disentangling it.”

The team will ground its AI models in the material properties of artworks and the physics of how X-rays and light interact with pigments and layered materials.

“Our project directly advances this objective by developing a new physics-informed AI framework for stratigraphic source separation in paintings,” Katsaggelos said.  “The goal is to reconstruct those layers in ways that are consistent with both the measurements and the physics of the materials.”

At the center of the framework is a simulator that models how properties of a painting’s visible and concealed layers produce signals across different   imaging techniques. The simulator will both generate synthetic data for training the AI and provide physics-based constraints as the system attempts to reconstruct the hidden layers.

Katsaggelos said traditional analytical and computational approaches have been effective at solving inverse problems, but the complexity of this particular problem requires AI to capture the relationships between paint-layer properties and signals collected through a variety of scientific techniques and imaging methods.

The resulting framework will combine those different sources of information while also measuring uncertainty in its reconstructions. 

Van Gogh’s work will provide a test case for the research. The Art Institute and Van Gogh Museum hold extensive technical documentation and data related to his paintings, along with expertise in 19th-century painting materials.

Those resources could help researchers explore not only whether another composition exists beneath a painting, but also how the underlying composition relates to the finished work.

“Hidden layers matter not only for uncovering an unknown image but also for revealing artistic decision-making: reuse of a canvas, an abandoned idea, a change in direction, or connections between a painting and other works, drawings, or documents,” Kokkori said.

The researchers describe a painting as a “time-encoded object,” with its visible surface representing one moment and concealed layers preserving evidence of earlier decisions.

The project will also avoid presenting a single AI-generated reconstruction as definitive when the available evidence permits multiple interpretations. Because different hidden configurations can produce similar measurements, the system will instead generate a range of physically plausible reconstructions and indicate where evidence is strong and where uncertainty remains.

“If AI presents a single, highly polished reconstruction as fact, it risks turning a computational hypothesis into an apparent art-historical truth,” Kokkori said. “We want AI to make uncertainty visible, not conceal it.”

Cultural heritage presents challenges because researchers often lack definitive ground truth, while the relationships between paint layers and the signals captured by different analytical techniques can be highly complex.

“The challenge is to learn how the properties of different paint layers combine to produce the signals we measure,” Katsaggelos said. “Our AI models will capture these complex, nonlinear relationships across analytical techniques and use them to guide reconstruction of the hidden layers.”

The team’s framework is designed to combine information from the different  analytical methods while quantifying uncertainty in the properties it attributes to hidden layers. Additionally, this approach relies on the expertise of conservators, art historians, scientists, and other experts in evaluating what the computational results mean.

The collaboration brings together engineering, computer science, materials and heritage science, conservation, art history, and radiology. Katsaggelos, the principal investigator of the work, is joined by co-PI’s Kokkori and the Art Institute of Chicago’s Giovanni Verri, Gloria Groom, and Kim Muir.  Madeline Meier, a postdoctoral fellow at the Center for Scientific Studies in the Arts, and former Northwestern postdoctoral researchers Santiago Lopez and Manuel Ballester were pivotal in developing the proposal. The former co-director of the Center for Scientific Studies in the Arts, Francesca Casadio, also played a key role in developing the proposal.

For Kokkori, the project reflects a broader effort to integrate those fields rather than treating them as separate sources of expertise.

“Historical and material knowledge can inform computational model design, while computational methods enable humanists, scientists, and conservators to ask questions about artistic processes that were previously impossible to address at this scale,” she said.

This project aligns closely with the Center of Scientific Studies in the Arts’ broader vision: bringing together heritage science, materials and computer science, the humanities, and museums not as parallel areas of expertise but as an integrated approach to producing new knowledge.