PixelPlayer uses AI and machine learning to isolate musical sounds on videos

Just a heads up, if you buy something through our links, we may get a small share of the sale. It’s one of the ways we keep the lights on here. Click here for more.

A new tool from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) called PixelPlayer can recognize instruments being played in a video. In doing so, it can separate those instruments so you can hear certain ones louder than the others, according to Engadget.

CSAIL used a self-supervised deep learning technique to teach PixelPlayer using over 60 hours of videos. In total, three neural networks are at play in this system – one for video, one for audio, and a third for separating the sound. The last one “associates specific pixels with specific soundwaves,” and then separates the different sounds.

Take a look:

Cool, no? While PixelPlayer may not be for everyone, band and instrument enthusiasts can use the tool to isolate a favorite instrument and listen to that exclusively, and it could also offer beatmakers a convenient way to isolate certain instruments when creating new beats and music.

Still, as Engadget explains, PixelPlayer is far from perfect since it can only identify the sounds of 20 instruments and still has problems telling similar ones apart. Nonetheless:

With further development, it could become an effective audio editing tool, giving engineers a way to improve or restore the quality of old concert footage. It could also be used to train robots on how to identify various environmental sounds, such as those made by animals, vehicles and appliances.

This isn’t the first time machine learning and AI has been used in the music industry. Recently, Google’s research arm, Magenta, developed a deep neural network to generate sound called NSynth Super. Using the code, you can make your own digital instrument from scratch.

Can you see a future purpose for something like PixelPlayer? Leave your comments below. 

For other recent technology news, see:

• A YouTuber made a Fortnite parody of Bruno Mars’ Finesse music video and it is glorious
• Spotify wants to license your music directly and cut out the middleman
• Facebook’s new lip-syncing tool looks a lot like Musical.ly
• Apple quietly released a way to listen to Apple Music directly from the web

Bryan M. Wolfe

Bryan considers himself a well-rounded techie, having written articles for MakeUseOf, KnowTechie, AppAdvice, iDownload Blog. When he's not writing, he's being a single dad and rooting for his alma mater, Penn State, or cheering on the Patriots.