DataRobot Trains Its AI to Predict Genres Based on Album Covers 

Scrolling through your music library, you may have noticed particular patterns in aesthetics among artists in particular genres. Enterprise AI firm DataRobot agrees: in a post on DataRobot’s blog, data scientist Rajiv Shah and data science intern Sara Cooper applied the company’s AI tools to see if they could effectively predict an album’s genre based on its cover art – an exercise they call an “enjoyable and unique way to explore visual AI.”

Using the Spotify API, the researchers collected track, album title, and cover art metadata on 277 tracks across three genres (indie, rock, and pop), supplementing that data with lyrics for those songs. Importing that data into DataRobot, the researchers set “genre” as the prediction target and let the AI take over. The software built “22 different types of models and blenders” that all attempted to predict genre based on some combination of cover art, track and album titles, and lyrical content. The most successful model combined cover art analysis with lyrical analysis, with cover art being far-and-away the most predictive factor.

The model performed very well on new albums: when the researchers tested new indie, pop, and rock albums, the model successfully predicted their genres with strong confidence. The team also used DataRobot’s “Activation Maps” feature to analyze the model’s predictions, allowing them to see what visual areas the AI considered relevant to its predictive abilities.

The Activation Maps with color overlays. Image courtesy of DataRobot.

“Looking at the indie albums, like the ones highlighted in orange, I notice they tend to feature more conceptual art,” the researchers wrote. “They have random objects, or abstract art, with only the occasional figure. It’s neat that the model caught the hands in two of them. Apparently indie artists like featuring the real music-makers. Pop albums, like the ones highlighted in red, have a singular figure highlighted. Most popular music originates from solo artists, and even famous bands like the Beatles eventually break up to pursue solo careers. Their fans will see them and quickly support the album as soon as they encounter it. For rock albums, the model mostly focuses on the groups of people, highlighted in blue.”

The researchers also took a deep dive into other visualizations of the AI’s processes and results, all of which are available in the full post here. Despite just being a fun exercise, they also thought of potential applications for the tool in the field.

“One application for this model could be scoring playlists for quicker album discovery,” they wrote. “One of my favorite Spotify features is the ‘Discover Weekly’ playlist, a list of around 30 songs similar to other songs you have previously ‘liked.’ Instead of going through all 30 songs, you could use the Spotify API to send the model the songs’ album covers, and then only listen to the songs of genres you prefer. Another potential application could be for marketing purposes. If someone listens to records with gothic cover art, suggest items that go with such an aesthetic, combat boots, ripped black jeans, dark lipstick, black hair dye, and similar attributes.”