Princeton Undergrad Creates Google Deep Dream-Inspired Deepjazz A.I. Music Maker

Ji-Sung Kim spent the early morning hours of the first weekend of April wired on caffeine and coding on his computer as he developed deepjazz, a deep learning music generator. The 20-year-old Princeton computer science sophomore only had 36 hours to complete deepjazz during his first hackathon, HackPrinceton, held on April 1-3 at the university. After finishing a marathon of coding, he created a website for deepjazz, and posted the source code on GitHub.

But much to Kim’s surprise, the program shot off. Deepjazz is steadily trending on Python and GitHub — reaching as high as the top seventh program on GitHub overall. It was even featured on the front page of HackerNews and is still generating a lively discussion.

“I never expected my first side project to be this explosive in terms of popularity,” Kim tells Inverse. “It’s been pretty crazy and fun.”

Between eating, sleeping, and completing other coursework, Kim reasoned that it took him about 12 hours to develop the source code for deepjazz. However, he came up with the idea for an artificial intelligence music generator long before HackPrinceton. During a summer internship at the University of Chicago, he came across Google’s Deep Dream, a photo generator that interprets the patterns in an image and transforms them into other objects it knows. The result are wonky images that look like they came out a wild dream.

“The idea of using deep learning to interpret art was really interesting to me,” says Kim. “This kind of Deep Dream framework that Google presented and published was really fascinating because you are creating novel pieces of art from existing artworks.”

Deepjazz uses machine-learning to generate jazz music — “an A.I. built to make jazz” as its SoundCloud profile states. Kim, who hasn’t taken music theory classes but has played the clarinet for seven years, chose jazz music because of its unconventional melodies. An A.I. system’s music, “may have some unusual outputs, so I think jazz was especially well suited to the idea of generating music on the fly,” he says.

He repurposed an existing music generator optimized for jazz music that his friend Evan Chow developed called JazzML, using the code to get relevant data but transforming it into a binary matrix that is compatible with the two deep learning libraries Keras and Theano.

The deepjazz framework itself it a two-layer LSTM, which is a kind of artificial neural network architecture, Kim describes. After it learns an initial baseline seed sequence of musical notes (Kim used parts of Pat Metheny’s “And Then I Knew”), it assigns probabilities to notes and generates the next note based on those probabilities. For example, if you feed the program the scale A, B, C, there is a high probability that the next note deepjazz will generate is going to be D, Kim explains.

Historically, critics have said artificial intelligence music generators are plagued by producing songs that sound too robotic and sterile — lacking the color heard in music composed by humans. Jeffrey Bilmes, a former MIT student who wrote a thesis on computers that reproduce musical rhythms in 1993, told Inverse in November:

“When you’re learning to play music and you’re learning to play jazz, there’s a utility to intuitively understanding what it is about music that makes it human,” Bilmes said. “Humans are intuitive beings, and humans often aren’t able to describe how they are able to do human things. I felt at the time that maybe I was violating a sacred oath in defining these things [for computer programs].”

When Kim was doing research for deepjazz he came across many systems that generated music that sounded robotic.

“Music and art are things that we consider deeply human,” Kim says. “To make it sound more human and more lifelike it’s really kind of hard to categorize.” Kim suggests that generators could create songs that sound more human by programming them to sound less similar to the original track.

Other developers have contacted Kim and are interested in expanding deepjazz so more people can interact with it. Kim can see deepjazz one day evolving into an improvisation partner that can artificially generate a backtrack for a musician to riff from. Even further into the future, he can see applications that create new, similar sounding music to your favorite tracks or that can suggest new chords and progressions for musicians.

Kim admits that he is still far from being an expert in deep learning, but his experience developing deepjazz and training at internships and Princeton has given him some valuable insight into the field.

“A.I. is no longer a sci-fi dream anymore. It’s something that’s very real and it’s something that is approaching at a very fast pace,” Kim says. “Hopefully seeing that this college student, who’s not even an upperclassman yet, was able to make something within a hackathon is encouraging to other students who are struggling to get into computer science.”