New Turing Tests: Can Machines Write Limericks and KidLit?

Can machines be taught to produce sonnets, limericks, or children’s literature that rivals work written by people?

What about music? Is it possible for a computer to compose in the style of, say, a chorale? Or to improvise artfully along with a human performer?

“I’m curious about all that,” says Associate Dean for the Sciences Dan Rockmore, the William H. Neukom 1964 Distinguished Professor of Computational Science. With Michael Casey, the James Wright Professor of Music; Kyle Booten, a fellow at the Neukom Institute for Computational Science, and Allen Riddell, an assistant professor of information science at Indiana University, Rockmore has announced the third annual Turing Tests in the Creative Arts, named for mathematician Alan Turing, and sponsored by the Neukom Institute.   

There are four literary categories. For PoetiX, machines generate Shakespearean or Petrarchan sonnets. LyriX invites any kind of verse but limits submissions to 30 lines, with 80 characters per line. The machine must be able to produce an infinite number of poems, to be evaluated for artistry.

New this year is DigiKidLit, challenging a machine to take a noun phrase prompt or use some other mechanism to generate an original, short children’s story. For computers with a sense of humor, there’s another added category, LimeriX, which will be judged among human limericks.

In the past, says Rockmore, evaluators were not asked to comment on the merit of submissions—just to guess whether the literature sprang from algorithms or human authors. But he says he’s keen to incorporate feedback that he’s received from “the world of computational creativity”—artists who use computers in their creative work. “They say the question shouldn’t be, merely, is it like a human’s work, but rather, is it interesting?” says Rockmore. So, this year, in some categories, a qualitative evaluation has been included.

In music, the 2018 Turing Tests present a new challenge for computers to imitate a chosen style, including Charlie Parker solos, Bach chorales, electroacoustic music, and free composition. Finally, as in last year’s contests, this year’s test seeks the invention of systems able to improvise with a human musician.

Whatever the artistic medium, Rockmore believes the competition raises fundamental questions about the creative process.

“I’ve always wondered about the ability to create poetry, or a great novel, or a beautiful piece of music. Where does it come from? How do you do it?” he asks. “Why does some stuff work and other stuff doesn’t? My goal is not to put limerick writers or children’s book writers or composers out of business. It’s just that I am interested in this nexus between science and the arts.”

And he’s open to adding new challenges to the Turing Tests. So, next year, could an algorithm generate this annual news story announcing the contest in Dartmouth News? 

“I suppose it’s possible,” Rockmore says, grinning. But, he adds, narrative is tough for a machine to learn and master. “You can teach a robot author what a sentence is, and even to make a second sentence that is more or less about the same thing,” he explains. “The challenge is to ensure that there is a causal relationship between those things, and then to sustain that coherence as the work builds.”

OK, so computers may not excel at expository or creative writing—yet. But in the future, Rockmore sees potential for some literary collaboration with humans. “For example, you might devise an algorithm to write a sonnet, with the expectation that it mainly serves as starter material. Even if, as a whole, this first draft doesn’t work, it might have a few gems of expression that you keep while you replace the less successful ones, and then weave it all together,” he suggests. 

Of course, that kind of co-authorship is against the rules in the Turing competition, and in the past the judges have generally been able to tell which art came from algorithms and which came directly from human hands. (See last year’s winners here.)

“You may think these challenges are about showing how powerful machines can be,” Rockmore says. “But at the end of the day, what the results really reveal is just how interesting human beings are.”

Submissions for the 2018 Turing Tests in the Creative Arts are due May 15. Winners in each category receive $1,000.

Charlotte Albright can be reached at charlotte.e.albright@dartmouth.edu.