Generating recommendations

all_recordings: recordings_df which is a dataframe is used to create a new RDD called all_recordings which contains all the recording_ids as contained in recordings_df. Number of elements in all_recordings are equal to number of rows in recordings_df = 22,335,149

Note: As 'rows' are to dataframes similarily 'elements' are to RDDs

all_recordings can be depicted as:

[ 1 ]

[ 2 ]

[ 3 ]

Note: recording_id are represented as [ x ] where x is an integer.

Model used to generate recommendations is listenbrainz-recommendation-model-55c5b675-a8d9-4489-aeb8-a499fb1ac7ab i.e. best model. Time taken to load best model is 0.15m

Sub-steps in generating recommendations for every user are as follows:

• get_user_id: A query(get_user_id) is processed to get user id from the user name provided for whom the recommendations shall be generated.

• user_playcounts:

  From the user id retrieved a user-playcounts dataframe is generated for every user, purpose of this dataframe is to get recording_ids of only those tracks that the user has ever listened to. The schema for user-playcounts dataframe is as follows:

  user_id	recording_id	count
  1	1	2

• user_recordings: The user-playcounts dataframe is used to create a new RDD called user_recordings which contains recording_ids of the tracks the user has ever listened to. user_recordings can be depicted as:

  [ 1 ]

  Since 'rob' has listened to one track of the three tracks ever listened by any user and the recording_id of that track is 1

• candidate_recordings: An RDD called candidate_recordngs is created by subtracting user_recordings from all_recordings, therefore candidate_recordings will contain recording_ids of only those tracks that the user has never listened to. This RDD would then be used to predict recommendations for a user ensuring only new tracks (tracks that are new to a user) are recommended to a user.

  candidate_recordings can be depicted as:

  all_recordings - user_recordings ~

  [ 2 ]

  [ 3 ]

  'rob' has never listened to tracks with recording ids as 2 and 3. Therefore, he will be recommedned tracks from amongst these two.

• predictAll : candidate_recordings is fed to Spark's inbuilt function 'predictAll' to predict tracks for users from the provided pool of recording_ids. The function returns recommended recording_ids. Let's say recording_id = 3 has been recommeded to 'rob'

• recommendations: The last step is to lookup for information (track_name, artist_name etc) corresponding to the recommended recording_ids which can be retrieved using recordings_df dataframe. The following track will be recommended to 'rob':

  ['Stand by me', 'b9da2ed1-6291-4b05-9e5e-b87551a8e75f', 'Ben E. King', '837555ba-012e-45f1-9a9c-9628da13ee54', 'Billboard Presents: Family Friendship Classics', '0fede8ca-7b38-455b-bde3-8a098c43031']

Following is the user wise time information:

Following is the user wise count information:

Time taken in generating recommendations for all the above listed users: 1.34h

Note: Total time lapsed in data collection, data preprocessing, model training and generating recommendations is 4.09h

Following are the recommendations generated for the above listed users:

Prev