Mapreduce Use Case – Sentiment Analysis on Twitter Data

This post is about performing Sentiment Analysis on Twitter data using Map Reduce. We will use the concept of distributed cache to implement Sentiment Analysis on Twitter data.

What does distributed cache do here?

By using distributed cache, we can perform map side joins. So, here we will join the dictionary dataset containing the sentiment values of each word. In order to perform Sentiment Analysis, we will be using a dictionary called AFINN.

AFINN is a dictionary, which consists of 2500 words rated from +5 to -5, depending on their meaning.

Here is the complete code to perform Sentiment Analysis using Map Reduce.

How did we achieve this?

Here are the three simple steps we have followed to perform Sentiment Analysis:

1. Implementing Distributed Caching

2. Writing a mapper class to calculate the sentiments

3. Writing a reducer class to display all the mapper output

4. Writing a Driver class for our mapreduce program
Implementing Distributed Caching

In Map Reduce, map-side joins are carried out by distributed cache. Distributed cache is applied when we have two datasets, where the smaller dataset size is limited to the cache memory of the cluster. Here, the dictionary is the smaller dataset, so we are using distributed cache. Here is the implementation of the distributed cache.

Initially, we have declared a HashMap by name AFINN_map to store the key and value i.e., the dictionary and the rating. You can download the dataset from here.

The data is tab separated, containing the word and its rating. So, we have read the file using the FSDataInputStream, read the file line by line using the readLine() method, we have split the line using the tab delimiter, and we have stored the word and its rating in the HashMap, which we have created with name AFINN_map.

The above line reads the cached file in the cluster. So, with that, our cached file was read and processed successfully and the distributed cache implementation is completed.

2. Writing a Map Method to Calculate the Sentiments

Now, the map method takes each record as input and the record is converted into a string, using the toString method. After this, we have created a jsonobject called jsonparser, which parses each record which is in JSON format.

Now we are extracting the tweet_id and the tweet_text which are required for sentiment analysis as shown below

Now, we are splitting the tweet_text into words, taking a for loop which repeats for all the words in the tweet_text, and we are performing an inner join to find the matches from the words in the AFINN dictionary if there is a match then take the rating of the word and add it. The same is as shown below.

Finally, in the context, we are writing the tweet_id as key and the combination of tweeted_text and the sentiment_sum as value.

3. Writing a Reducer Class to Display all the Mappers Output

In the reducer class, we are just passing the input of the mapper as its output.

4. Driver class

In the Driver class, we need to provide the path for the cached dataset, using the below line.

On top of this, we need to provide the input(tweets_folder) path and the output folder path as arguments.

How to run our mapreduce program? 

In order to run this program, we need to build a jar file of the above project, using the normal Hadoop command to run the program. You can refer to the below screenshot for this.

hadoop jar twitter.jar /user/flume/tweets /user/flume/tweets/output

The output will be created in the part file in the directory /user/flume/tweets/part-r-00000 and the result is displayed as shown in the below screenshot.

We hope this post has been helpful in understanding how to perform Sentiment Analysis using Map Reduce. In the case of any queries, feel free to comment below and we will get back to you at the earliest.

Keep visiting our site www.acadgild.com for more updates on Big Data and other technologies.