Determining Popular Hashtags in Twitter Using Pig

In our previous post we had seen how to perform Sentiment Analysis on Twitter data using Pig. In this post we will be discussing how to find out the most popular hashtags in the tweets.

You can refer to this post to know how to get tweets from Twitter using Flume.

We have collected and stored the tweets in HDFS. The tweets are located in the following location of the HDFS: /user/flume/tweets/

The data from Twitter is in ‘Json’ format, so a Pig JsonLoader is required to load the data into Pig. You need to download the required jars for the JsonLoader from the below link:

Pig JsonLoader Jar Files

Next, register the downloaded jars in Pig by using the following commands:

You can refer to the below screen shot for the same.

Note: You need to provide the path of the jar file accordingly.

After registering the required jars, we can now write a Pig script to perform Sentiment Analysis.

Below is a sample tweets collected for this purpose:

The tweets are in nested Json format and consists of map data types. We need to load the tweets using JsonLoader which supports maps, so we are using elephant bird JsonLoader to load the tweets.

Below is the first Pig statement that is required to load the tweets into Pig:

When we dump the above relation, we can see that all the tweets have been loaded successfully.

Now, let’s extract the id and the hashtag from the above tweets and the Pig statement for doing this is as shown below

In the tweet, the hashtag is present in the map object entities, which can be seen in the below image.

Since the hashtags are inside the map entities, we have extracted the entities as map[ ] data type. The schema of the relation extract_details can be viewed using the below command:

Now, from the entities, we have to extract the hashtags which is again a map. So we will extract the hashtags as map[ ] data type as well.

The extracted hashtags can be viewed by dumping the above relation.

In the above image, we can see that the hashtags object has been extracted successfully. The extracted hashtags is also a map data type, which can see by describing the relation. You can refer to the below image for the same.

Now, from the extracted hashtags, we need to extract text which contains the actual hashtag. This can be done using the following command:

Here, we have extracted the text which starts with # and named it with an alias name text.

We can see the extracted hashtags text in the below screen shot:

In the above image, we can see the hashtag’s text and the tweet_id from which it has originated.

Now, we will group the relation by hashtag’s text by using the below relation:

We have successfully grouped by hashtag’s text. We can see the schema by describing the relation.

The next thing to do is, count the number of times the hashtag is repeated by the user. This can be achieved using the below relation:

Now we have the hashtags and its count in a relation as shown in the below screen shot.

We have now successfully performed hashtag count on Twitter data using Pig!

Hope this post has help you learn how to count hashtags. Keep visiting our blog for more updates on Big Data and other technologies.