Better recommendations, happier users
As an Insight Data Science Fellow in New York City I immediately saw the business value in delivering better recommendations to users - I myself open up my Spotify Discover playlist ever week eager to see what music Spotify has found for me, I know that better recommendations bring value to the user.
The niche I found that had yet to be addressed in the publicly available literature ( inside word is that YouTube, Amazon, etc have been doing this for years ) was targeting users' sequential preferences, and with even more difficulty, targeting users' temporally variant sequential preferences. What does all of that mean? In the image above pretend that the album covers depict a song that user A listened to. Our job as the recommender system is to recommend to user A 3 songs that they might want to listen to where the grey box is. You and I both easily know that all three choices should be a Taylor Swift song but the truth is, this is a difficult thing to teach a computer.
My first challenge was finding data as good data makes a quality product. I spent several days searching for the right data, discarding some sets, such as the Netflix 1 million movies set, and treasuring others such as the #nowplaying-RS dataset.
While I knew there were several basic algorithms which could capture sequential patterns, such as LSTMs and CNNs, I knew that model iteration would be part of my process and so I built a framework to process data from arbitrary sources and feed this data to a given model for assessment. With this framework I was able to quickly iterate on datasets and models finding the right balance between accuracy and prediction speed as users have very little tolerance for lag.