Generating recommendations

Time taken to generate recommendations for each user:

Time taken in generating recommendations for all the above listed users: 0.12h

Note: Total time lapsed data processing and generating recommendations is 0.13h

Three dataframes namely, users_df, recordings_df, playcounts_df are loaded from HDFS in 0m

user_df can be depicted as:

recordings_df can be depicted as:

playcounts_df can be depicted as :

all_recordings: recordings_df (dataframe) is used to create a new dataframe called all_recordings which contains all the recording_ids as contained in recordings_df. Number of elements in all_recordings are equal to number of rows in recordings_df = 22,335,149

Note: As 'rows' are to dataframes similarily 'elements' are to RDDs

all_recordings can be depicted as:

Model used to generate recommendations is listenbrainz-recommendation-model-57c75248-d6e2-4a67-ae91-8017a4c9137b i.e. best model. Time taken to load best model is 0.16m

Sub-steps in generating recommendations for every user are as follows:

• get_user_id: A query(get_user_id) is processed to get user id from the user name provided for whom the recommendations shall be generated.

• user_playcounts: From the user id retrieved a user-playcounts dataframe is generated for every user, purpose of this dataframe is to get recording_ids of only those tracks that the user has ever listened to. The schema for user-playcounts dataframe is as follows:

  user_id	recording_id	count
  1	1	2

  Since 'rob' has listened to one track of the three tracks ever listened by any user and the recording_id of that track is 1

• candidate_recordings: A new dataframe called candidate_recordngs is created by subtracting user_recordings from all_recordings, therefore candidate_recordings will contain recording_ids of only those tracks that the user has never listened to.

  candidate_recordings can be depicted as:

  recording_id
  2
  3

  'rob' has never listened to tracks with recording ids as 2 and 3. Therefore, he will be recommedned tracks from amongst these two.

  The candiate_recordings dataframe is later converted to an RDD. This RDD would then be used to predict recommendations for a user ensuring only new tracks (tracks that are new to a user) are recommended to a user

• predictAll : candidate_recordings is fed to Spark's inbuilt function 'predictAll' to predict tracks for users from the provided pool of recording_ids. The function returns recommended recording_ids. Let's say recording_id = 3 has been recommeded to 'rob'

• recommendations: The last step is to lookup for information (track_name, artist_name etc) corresponding to the recommended recording_ids which can be retrieved using recordings_df dataframe. The following track will be recommended to 'rob':

  ['Stand by me', 'b9da2ed1-6291-4b05-9e5e-b87551a8e75f', 'Ben E. King', '837555ba-012e-45f1-9a9c-9628da13ee54', 'Billboard Presents: Family Friendship Classics', '0fede8ca-7b38-455b-bde3-8a098c43031']

Following are the recommendations generated for the above listed users:

Note: Number of rows in a dataframe or number of elements in an RDD (count information) is not included because it leades to unnecessary computation time.