Rachelle Palmer

Product Lead, Developer Experience

## Description of Application
A quick and easy example application project that creates a demo blog post engine. Very simple UI, ideal for beginners to Rust programming langauge.

##  Technology Stack
This code example utilizes the following technologies:

* MongoDB Atlas
* Rust
* Rocket.rs framework



Check out this quick and easy example application project that creates a demo blog post engine. It's ideal for beginners to the Rust programming language.

Beginner Coding Project: Build a Blog Engine With Rust and MongoDB

## Introduction
Build your first application with Java and Spring! This simple application demonstrates basic CRUD operations via a book app - you can add a book, edit a book, delete a book. Stores the data in MongoDB database. 


## Technology

* Java
* Spring Boot
* MongoDB




Build an application to track the books you've read with Spring Boot, Java, and MongoDB

Build a MongoDB Spring Boot Java Book Tracker for Beginners

## Creators
[Lucas De Oliveira](https://www.linkedin.com/in/lbdeoliveira/), [Chandrish Ambati](https://www.linkedin.com/in/avmchandrish/), and [Anish Mukherjee](https://www.linkedin.com/in/mukherjeeanish/) from University of San Francisco contributed this amazing project.

## Background to the Project
In 2018, Spotify organized an Association for Computing Machinery (ACM) [RecSys Challenge](https://www.recsyschallenge.com/2018/) where they posted [a dataset of one million playlists](https://www.aicrowd.com/challenges/spotify-million-playlist-dataset-challenge), challenging participants to recommend a list of 500 songs given a user-created playlist.

As both music lovers and data scientists, we were naturally drawn to this challenge. Right away, we agreed that combining song embeddings with some nearest-neighbors method for recommendation would likely produce very good results. Importantly, we were curious about how we could  solve  this recommendation task at scale with over [4 billion user-curated playlists](https://soundplate.com/how-many-playlists-are-there-on-spotify-and-other-spotify-stats/) on Spotify, where this number keeps growing. This realization raised serious questions about how to train a decent model since all that data would likely not fit in memory or a single server.

## What We Built
This project resulted in a scalable ETL pipeline utilizing
* Apache Spark
* MongoDB
* Amazon S3
* Databricks (PySpark)

These were used to train a deep learning Word2Vec model to build song and playlist embeddings for recommendation. We followed up with data visualizations we created on Tensorflow’s Embedding Projector.

## The Process
### Collecting Lyrics
The most tedious task of this project was collecting as many lyrics for the songs in the playlists as possible. We began by isolating the unique songs in the playlist files by their track URI; in total we had over 2 million unique songs. Then, we used the track name and artist name to look up the lyrics on the web. Initially, we used simple Python requests to pull in the lyrical information but this proved too slow for our purposes. We then used asyncio, which allowed us to make requests concurrently. This sped up the process significantly, reducing the downloading time of lyrics for 10k songs from 15 mins to under a minute. Ultimately, we were only able to collect lyrics for 138,000 songs.

### Pre-processing
The original dataset contains 1 million playlists spread across 1 thousand JSON files totaling about 33 GB of data. We used PySpark in Databricks to preprocess these separate JSON files into a single SparkSQL DataFrame and then joined this DataFrame with the lyrics we saved. 

While the aforementioned data collection and preprocessing steps are time-consuming, the model also needs to be re-trained and re-evaluated often, so it is critical to store data in a scalable database. In addition, we’d like to consider a database that is schemaless for future expansion in data sets and supports various data types.  Considering our needs, we concluded that MongoDB would be the optimal solution as a data and feature store.

Check out the [Preprocessing.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Preprocessing_Pipeline.ipynb) notebook to see how we preprocessed the data.

### Training Song Embeddings
For our analyses, we read our preprocessed data from MongoDB into a Spark DataFrame and grouped the records by playlist id (pid), aggregating all of the songs in a playlist into a list under the column song_list. 
Using the Word2Vec model in Spark MLlib we trained song embeddings by feeding lists of track IDs from a playlist into the model much like you would send a list of words from a sentence to train word embeddings. As shown below, we trained song embeddings in only 3 lines of PySpark code:
```
from pyspark.ml.feature import Word2Vec
word2Vec = Word2Vec(vectorSize=32, seed=42, inputCol="song_list").setMinCount(1)
word2Vec.sexMaxIter(10)
model = word2Vec.fit(df_play)
```

We then saved the song embeddings down to MongoDB for later use. Below is a snapshot of the song embeddings DataFrame that we saved:

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/song_embeddings_dataframe_c3fc7a455b.png)

Check out the [Song_Embeddings.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Song_Embeddings.ipynb) notebook to see how we train song embeddings.

### Training Playlists Embeddings
Finally, we extended our recommendation task beyond simple song recommendations to recommending entire playlists. Given an input playlist, we would return the k closest or most similar playlists. We took a “continuous bag of songs” approach to this problem by calculating playlist embeddings as the average of all song embeddings in that playlist.

This workflow started by reading back the song embeddings from MongoDB into a SparkSQL DataFrame. Then, we calculated a playlist embedding by taking the average of all song embeddings in that playlist and saved them in MongoDB.

Check out the [Playlist_Embeddings.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Playlist_Embeddings.ipynb) notebook to see how we did this.

### Training Lyrics Embeddings
Are you still reading? Whew!

We trained lyrics embeddings by loading in a song's lyrics, separating the words into lists, and feeding those words to a Word2Vec model to produce 32-dimensional vectors for each word. We then took the average embedding across all words as that song's lyrical embedding. Ultimately, our analytical goal here was to determine whether users create playlists based on common lyrical themes by seeing if the pairwise song embedding distance and the pairwise lyrical embedding distance between two songs were correlated. Unsurprisingly, it appears they are not.

Check out the [Lyrical_Embeddings.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Lyrical_Embeddings.ipynb) notebook to see our analysis.

## Notes on our Approach
You may be wondering why we used a language model (Word2Vec) to train these embeddings. Why not use a Pin2Vec or custom neural network model to predict implicit ratings? For practical reasons, we wanted to work exclusively in the Spark ecosystem and deal with the data in a distributed fashion. This was a constraint set on the project ahead of time and challenged us to think creatively.

However, we found Word2Vec an attractive candidate model for theoretical reasons as well. The Word2Vec model uses a word’s context to train static embeddings by training the input word’s embeddings to predict its surrounding words. In essence, the embedding of any word is determined by how it co-occurs with other words. This had a clear mapping to our own problem: by using a Word2Vec model the distance between song embeddings would reflect the songs’ co-occurrence throughout 1M playlists, making it a useful measure for a distance-based recommendation (nearest neighbors). It would effectively model how people grouped songs together, using user behavior as the determinant factor in similarity.

Additionally, the Word2Vec model accepts input in the form of a list of words. For each playlist we had a list of track IDs, which made working with the Word2Vec model not only conceptually but also practically appealing.

## Data Visualizations with Tensorflow and MongoDB
After all of that, we were finally ready to visualize our results and make some interactive recommendations. We decided to represent our embedding results visually using [Tensorflow’s Embedding Projector](https://projector.tensorflow.org) which maps the 32-dimensional song and playlist embeddings into an interactive visualization of a 3D embedding space. You have the choice of using PCA or tSNE for dimensionality reduction and cosine similarity or Euclidean distance for measuring distances between vectors.

Click [here](https://projector.tensorflow.org/?config=https://embedding-projector.s3.us-west-2.amazonaws.com/template_config_2M.json) for the song embeddings projector for the full 2 million songs, or [here](https://projector.tensorflow.org/?config=https://embedding-projector.s3.us-west-2.amazonaws.com/template_config.json) for a less crowded version with a random sample of 100k songs (shown below):

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/164569070_88be5d94_a291_495d_b33d_3094634d9205_afc69037e9.gif)

The neat thing about using Tensorflow’s projector is that it gives us a beautiful visualization tool and distance calculator all in one. Try searching on the right panel for a song and if the song is part of the original dataset, you will see the “most similar” songs appear under it.

## Using MongoDB for ML/AI
We were impressed by how easy it was to use MongoDB to reliably store and load our data. Because we were using distributed computing, it would have been infeasible to run our pipeline from start to finish any time we wanted to update our code or fine-tune the model. MongoDB allowed us to save our incremental results for later processing and modeling, which collectively saved us hours of waiting for code to re-run.

It worked well with all the tools we use everyday and the tooling we chose - we didn't have any areas of friction. 

We were shocked by how this method of training embeddings actually worked. While the 2 million song embedding projector is crowded visually, we see that the recommendations it produces are actually quite good at grouping songs together.

Consider the embedding recommendation for The Beatles’ “A Day In The Life”:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/165581471_420ad43d_47f9_4aee_bf0c_2de6ac301144_c656a07f0e.gif)

Or the recommendation for Jay Z’s “Heart of the City (Ain’t No Love)”:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/164567916_28f6ebd6_0473_4d7b_b60c_55af628016f7_ec16751581.png)

Fan of Taylor Swift? Here are the recommendations for “New Romantics”:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/164567889_9ccf8326_7bc8_4c8e_9bc6_f123c0acba3f_0fab2ea1db.png)

We were delighted to find naturally occurring clusters in the playlist embeddings. Most notably, we see a cluster containing mostly Christian rock, one with Christmas music, one for reggaeton, and one large cluster where genres span its length rather continuously and intuitively.

Note also that when we select a playlist, we have many recommended playlists with the same names. This in essence validates our song embeddings. Recall that playlist embeddings were created by taking the average embedding of all its songs; the name of the playlists did not factor in at all. The similar names only conceptually reinforce this fact.



## Next Steps?
We felt happy with the conclusion of this project but there is more that could be done here.

1. We could use these trained song embeddings in other downstream tasks and see how effective these are. Also, you could download the song embeddings we here: [Embeddings](https://embedding-projector.s3.us-west-2.amazonaws.com/emb_word2vec_2M.tsv) | [Meta Info](https://embedding-projector.s3.us-west-2.amazonaws.com/meta_word2vec_2M.tsv)
2. We could look at other methods of training these embeddings using some recurrent neural networks and enhanced implementation of this Word2Vec model.





Python code example application for Spotify playlist and song recommendations using spark and tensorflow

A Spotify Song and Playlist Recommendation Engine

JavaScript

Python

Java

Swift

TypeScript

Bash

Rust

Kotlin

Ruby

Spring

Node.js

Android

React

Unity

Spark

RaspberryPi

FastApi

Flask

Vercel

Docker

Kubernetes

Next.js

Serverless

React Native

Parquet

Netlify

Jetpack Compose

Mobile

Mongoose

Neurelo

Xamarin

Postman API

GitHub Actions

Jupyter

Kafka

Pandas

Quarkus

Django

Terraform

GraphQL

Azure

Google Cloud

.NET

Prisma

MongoDB

Data Visualization

Security

Vector Search

Shell

Cloud Manager

Charts

Change Streams

Indexes

Stream Processing

Time series

Schema

Aggregation Framework

Sharding

Atlas

Data API

Multi-Cloud

Data Federation

Online Archive

Triggers

Search

Compass

Connectors

Ops Manager

Introductory

Advanced

Intermediate

Student

Community

Champion

Partner

Full Application

Snippet

Lauren Schaefer

Node.js and MongoDB is a powerful pairing and in this code example project we show you how.

Connect to a MongoDB Database Using Node.js 3.3.2

Pavel Duchovny

Discover the Tour Planner: AI-powered travel planning using PHP, Laravel, MongoDB Vector Search & OpenAI. Personalized itineraries made easy!

Introducing the Tour Planner With MongoDB Vector Search

Blogue is a writing platform where all writers/non-writers are welcome. Its back end is Node.js and the front end is Next.js and React.js. Atlas is the storage.

Blogue

This is a theoretical Electronic Health Record system (EHR-S) in Peru, which uses a MongoDB cluster to store clinical information.

EHRS-Peru

Learn how to build a system that manages students' posts in universities. It allows students and clients to read posts in diverse categories.

Magazine Management

Rita Rodrigues

Let's analyse how hashtags on Twitter change over time. We'll use the Twitter API to fetch realtime trends data and save it to MongoDB for later analysis.

Trends Analyser

Looking for a code example that has microservices in Go with Docker, Kubernetes, and MongoDB? Look no further!

Cinema: Example Go Microservices Application

Michael Lynn

This article will help you build a command line executable tool that manipulates data in a MongoDB database.

Build a Command Line Tool With Swift and MongoDB

Aaron Bassett

In this quick start, we will create a CRUD (Create, Read, Update, Delete) app showing how you can integrate MongoDB with your Tornado projects.

Getting Started With MongoDB and Tornado

This project aims to develop a database for dog care providers and demonstrate how this data can be manipulated in MongoDB.

Example Application for Dog Care Providers (DCP)

Teo Wen Jie

Quick start to Reactive Java Spring Boot and Spring Data MongoDB with an example application which includes implementations ofReactiveMongoRepository and ReactiveMongoTemplate and multi-document ACID transactions

Reactive Java Spring Boot with MongoDB

Maxime Beugnet

Megha Arora

In this advanced MongoDB CSFLE Java template, you'll learn all the tips and tricks for a successful deployment of CSFLE with Spring Data MongoDB.

How to Implement Client-Side Field Level Encryption (CSFLE) in Java with Spring Data MongoDB

Jason Mimick

Learn how to get started with MongoDB Atlas and AWS CloudFormation.

Get Started with MongoDB Atlas and AWS CloudFormation

Take a shortcut to REST APIs with this Java/Spring Boot and MongoDB example application that embeds all you'll need to get going.

REST APIs with Java, Spring Boot, and MongoDB

Kaitlin Mahar

Curious about mobile and server-side swift? Use this tutorial and example code!

Building a Full Stack application with Swift

Get started with Python and MongoDB easily with example code in Github

Simple CRUD operations with Python and MongoDB

Dominic Frei

Persisting data is an important part of most games. Unity offers only a limited set of solutions, which means we have to look around for other options as well. In this tutorial series, we will explore the options given to us by Unity and third-party libraries.

Saving Data in Unity3D Using SQLite

This project downloads the substitution plan of my school and converts it into a user-friendly page.

myLeG

Shorten multiple URLs instantly while overcoming the barriers of user registration.

MergeURL - Python Example App

Find hostels and student apartments all over Kenya

Hostels Kenya Example App

A web-app that automates group formation for projects/assignments etc..

GroupUs

GoFifa - A comprehensive soccer stats tracker.

Go-FIFA

Final Space API is a public RESTful API based on the animated television show Final Space.

Final Space API

An environmental satellite to get information about your environment.

EnSat

Instantly find the street ball game you are looking for anytime and anywhere with Chember!

Chember

A web application to help users message and be messaged completely anonymously.

Anonytexts

Joe Karlsson

Learn all about developing IoT projects using JS and MongoDB by building an smart toilet for your cat! Click here for more!

An Introduction to IoT (Internet of Toilets)

Headshot of Adrienne Tacke outside, wearing a blue dress

Adrienne Tacke

Learn how to use MongoDB client-side field level encryption (CSFLE) with a C# application.

How to Use MongoDB Client-Side Field Level Encryption (CSFLE) with C#

Mohit Sharma

In this article, we will explore and learn how to build a splash screen with SplashScreen API, which was introduced in Android 12.

Code Examples

Featured Code Examples

Beginner Coding Project: Build a Blog Engine With Rust and MongoDB

Build a MongoDB Spring Boot Java Book Tracker for Beginners

A Spotify Song and Playlist Recommendation Engine

All Code Examples