Art History, IST faculty win NEH grant to use computer-aided image analysis

Elizabeth Mansfield, professor and head of the Department of Art History, and James Wang, professor of information sciences and technology, have received a Digital Humanities Advancement Grant (DHAG) from the National Endowment for the Humanities (NEH) for a project that will use computer-aided image analysis to examine the depiction of clouds in the paintings of John Constable, a 19^th-century European artist noted for his pictorial realism. The project, “Seeing Constable’s Clouds: An Application of Machine Learning to Art Historical Research,” will use computer vision and machine learning to reveal formal details and patterns that art historians may overlook or be unable to discern, due to their generalized sense of what a “Constable cloud” should look like. Specifically, Mansfield and Wang, who are co-principal investigators on the project, want to determine if a neural network can be trained to recognize the difference between a painted cloud by Constable, a “straight” photograph of a cloud from the repository of the National Oceanographic and Atmospheric Agency Photo Library and an artistic photograph of a cloud from the 1850s by Gustave Le Gray. “If a neural network can be trained to discriminate between these different kinds of representations of clouds—and not simply to discern imaging artifacts—this training might be applied to data sets composed entirely of images of paintings to see which are classed as more or less ‘photographic’—a measure of pictorial realism adopted by viewers in Constable’s lifetime,” explained Mansfield. “Such results might enable researchers to understand the visual cues that persuaded Constable’s 19^th-century viewers to see in his cloud studies—as viewers continue to see today—persuasive representations of natural phenomena that are virtually impossible to document exactly using brush and paint.” Computer vision and machine learning have been applied to art historical research since the early 2000s. Current applications of machine learning to art historical research include projects devoted to training computers to recognize specific objects or individuals depicted in various pictorial arts such as prints and paintings. Mansfield and Wang’s approach is to develop machine-learning algorithms to detect the most significant stylistic differences among cloud study paintings. The features that reveal the most distinctive differences among images with clouds will be determined by Convolutional Neural Network (CNN), a deep learning algorithm commonly applied to analyzing digital imagery, which can take in an input image, assign importance to various aspects/objects in the image and be able to differentiate one from the other.

Despite the recent rapid advances in machine learning and computer vision, it is often highly challenging to use computerized methods to analyze visual artworks because art is a representation of the real world.” – James Wang

"We are excited at the opportunity to help answer significant art historical questions while advancing technologies to face the challenges,” continued Wang. In addition to Mansfield and Wang, other Penn State collaborators on “Seeing Constable’s Clouds” include Catherine Adams, assistant curator of the Visual Resources Centre in the Department of Art History; Jia Li, professor of statistics and computer science; and John Russell, assistant professor and digital humanities librarian at the University Libraries and associate director of the Center for Humanities and Information. Training of a CNN is scheduled to begin in September 2020 and continue through the fall and winter. The intended final product for this phase is a CNN able not only to reliably distinguish images of clouds from other forms but also to distinguish images whose source is a photograph of a cloud from images whose source is a painting of a cloud. Digital Humanities Advancement Grants support innovative, experimental and/or computationally challenging projects at different stages throughout their lifecycles, from early start-up phases through implementation and sustainability. The NEH is an independent federal agency created in 1965, and one of the largest funders of humanities programs in the United States. The NEH awards grants to top-rated proposals examined by panels of independent, external reviewers. For more information, visit neh.gov.