MapReduce Use Case-Youtube Data Analysis

This blog is about analyzing the data of youtube.This total analysis is performed   in Hadoop MapReduce.
This youtube data is publicly available and the youtube data set is described below under the heading Data Set Description.
Using that dataset we will perform some Analysis and will draw out some insights like what are the top 10 rated videos in youtube, who uploaded the most number of videos.
By reading this blog you will understand how to handle data sets that does not have proper structure and how to sort the output of reducer.

DATA SET DESCRIPTION

Column 1: Video id of 11 characters.

Column 2: up loader of the video

Column 3: Interval between day of establishment of Youtube and the date of uploading of the video.

Column 4: Category of the video.

Column 5: Length of the video.

Column 6: Number of views for the video.

Column 7: Rating on the video.

Column 8: Number of ratings given for the video

Column 9: Number of comments done on the videos.

Column 10: Related video ids with the uploaded video.

You can download the data set from the below link.

DATA SET LINK

Youtube Data set

PROBLEM STATEMENT 1

Here we will find out what are the top 5 categories with maximum number of videos uploaded.

SOURCE CODE

Now from the mapper we want to get the video category as key and final int value ‘1’ as values which will be passed to the shuffle and sort phase and are further sent to the reducer phase where the aggregation of the values is performed.

MAPPER CODE

Explanation of the above Mapper code:
In line 1 we are taking a class by name Top5_categories,
In line 2 we are extending the Mapper default class having the arguments keyIn as LongWritable and ValueIn as Text and KeyOut as Text and ValueOut as IntWritable.

In line 3 we are declaring a private Text variable ‘category’ which will store the category of videos in youtube

In line 4 we are declaring a private final static IntWritable variable ‘one’ which will be constant for every value. MapReduce deals with Key and Value pairs.Here we can set the key as gender and value as age.

In line 5 we are overriding the map method which will run one time for every line.

In line 7 we are storing the line in a string variable ‘line’

In line 8 we are splitting the line by using tab “\t” delimiter and storing the values in a String Array so that all the columns in a row are stored in the string array.

In line 9 we are taking a condition if we have the string array of length greater than 6 which means if the line or row has at least 7 columns then it will enter into the if condition and execute the code to eliminate the ArrayIndexOutOfBoundsException. 

In line 10 we are storing the category which is in 4^th column.

In line 12 we are writing the key and value into the context which will be the output of the map method.

REDUCER CODE

While coming to the Reducer code

line 1 extends the default Reducer class with arguments KeyIn as Text and ValueIn as IntWritable which are same as the outputs of the mapper class and KeyOut as Text and ValueOut as IntWritbale which will be final outputs of our MapReduce program.

In line 2 we are overriding the Reduce method which will run each time for every key.

In line 3 we are declaring an integer variable sum which will store the sum of all the values for each key.

In line 4 a foreach loop is taken which will run each time for the values inside the “Iterable values” which are coming from the shuffle and sort phase after the mapper phase.

In line 5 we are storing and calculating the sum of the values.

In line 7 writes the respected key and the obtained sum as value to the context.

CONF CODE

This two configuration classes are included in the main class whereas to clarify the Output key type of mapper and the output value type of the Mapper.

You can download the whole source code from the below link.

SOURCE CODE LINK

GitHub link for Problem statement 1

HOW TO EXECUTE

hadoop jar top5.jar /youtubedata.txt /top5_out

Here ‘hadoop’ specifies we are running a Hadoop command and jar specifies which type of application we are running and top5.jar is the jar file which we have created  consisting of the above source code.

The path of the Input file in our case is root directory of hdfs denoted by /youtubedata.txt  and the output file location to store the output has been given as top5_out.

How to view output

hadoop fs -cat /tpo5_out/part-r-00000 | sort –n –k2 –r | head  –n5

Here ‘hadoop’ specifies that we are running a Hadoop command and dfs specifies that we are performing an operation related to Hadoop Distributed File Sysytem and ‘- cat’ is used to view the contents of a file and top5_out/part-r-00000 is the file where output is stored.

Part file containing the actual output is created by default by the TextInputFormat class of Hadoop.

Here sort –n –k2 –r | head –n5 brings you the top 5 categories with maximum number of videos uploaded.

Instead of writing a secondary sort after reducer we can simply use this command to get the required output.

Sort will sort the data, –n means sorting numerically, –k2 means second column,  –r is for recursive operation and head –n5 means bring the first 5 values after sorting.

Output

PROBLEM STATEMENT 2

In this problem statement we will find the top 10 rated videos in youtube.

SOURCE CODE

Now from the mapper we want to get the video id as key and rating as value which will be passed to the shuffle and sort phase and are further sent to the reducer phase where the aggregation of the values is performed.

MAPPER CODE

Explanation of the above Mapper code

In line 1 we are taking a class by name Video_rating

In line 2 we are extending the Mapper default class having the arguments keyIn as LongWritable and ValueIn as Text and KeyOut as Text and ValueOut as FloatWritable.

In line 4 we are declaring a private Text variable ‘video_name’ which will store the video name which is in encrypted format.

In line 5 we are declaring a private FloatWritable variable ‘rating’ which will store the rating of the video. MapReduce deals with Key and Value pairs.Here we can set the key as gender and value as age.

In line 6 we are overriding the map method which will run one time for every line.

In line 8 we are storing the line in a string variable ‘line’

In line 9 we are taking a condition if we have the string array length greater than 7 which means if the line or row has at least 7 columns then it will enter into the if condition and execute the code to eliminate the ArrayIndexOutOfBoundsException.

In line 10 we are splitting the line by using tab “\t” delimiter and storing the values in a String Array so that all the columns in a row are stored in the string array.

In line 11 we are storing the video name which is in the 1^st column.

In line 12 we are checking whether the data in that index is numeric data or not by using a regular expression which can be achieved by “matches function in java”,if it is numeric data then it will proceed and it should be a floating value as well.

In line 13 we are converting that numeric data into Float data by type casting.

In line 14 we are storing the rating of the video in ‘rating’ variable.

In line 17 we are writing the key and value into the context which will be the output of the map method.

REDUCER CODE

While coming to the Reducer code

line 1 extends the default Reducer class with arguments KeyIn as Text and ValueIn as IntWritable which are same as the outputs of the mapper class and KeyOut as Text and ValueOut as IntWritbale which will be final outputs of our MapReduce program.

In line 2 we are overriding the Reduce method which will run each time for every key.

In line 4 we are declaring an integer sum which will store the sum of all the ages of people into it.

In line 5 we are taking another variable as “l” which will be incremented every time as many values are there for that key.

In line 6 a foreach loop is taken which will run each time for the values inside the “Iterable values” which are coming from the shuffle and sort phase after the mapper phase.

In line 8 we are storing and calculating the sum of the values.

In line 10 we are performing the average of the obtained sum and writes the respected key and the obtained sum as value to the context.

CONF CODE

This two configuration classes are included in the main class whereas to clarify the Output key type of mapper and the output value type of the Mapper.

You can download the whole source code from the below link

SOURCE CODE LINK

GitHub link for Problem statement 2

HOW TO EXECUTE

hadoop jar video_rating.jar /youtubedata.txt /videorating_out

Explanation for the above command will be as same as given in problem statement 1.

How to view output

hadoop fs -cat /videorating_out/part-r-00000 | sort –n –k2 –r | head –n10

Explanation for the above command will be as same as given in problem statement 1.

Output:

We hope this blog will help you to get a grip on MapReduce programming. Refer the below blog to understand the analysis done on Titanic data set.
Titanic Data Analysis