In this post, we will be running Pig scripts and Hive queries in both YARN as well as the TEZ engine. We will be analysing how the performance varies and which will be faster whether YARN or TEZ.

Pig Script on Tez and YARN

Now let us write a Pig script for a dictionary called AFINN, in which 2477 words are rated from -5 to +5 based on the words meaning. In the Pig script we will be counting how many positive words (0-5) and negative words (-5 to -1) are there.

The Pig script for calculating the number of negative and positive words in the dictionary looks like as shown below:

Now, let’s save the output of the script in HDFS /AFINN/yarn/ directory for YARN output and /AFINN/tez/ for the output from Tez. Let’s assign the name the file containing the above Pig script as dictionary.pig.

Pig on YARN

Let’s run the above code using YARN engine and note down the time.

We can see that YARN took 24 seconds and 88 milliseconds to complete this job. Now, let us run the same script using TEZ engine.

Pig on TEZ

The command for running Pig using Tez engine is as follows:

We can see that Tez completed the job in just 15 seconds and 295 milliseconds.

HIVE ON YARN and TEZ

Here we will create a hive table and load a dictionary dataset which we have into the table and we will run a hive query for calculating the number of positive and negative words are there in the dictionary.

Creation of hive table and loading the dataset is as shown below:

HIVE ON YARN

Let’s run the query for counting the number of positive and negative words in the dictionary on YARN engine.

You can see that Hive on YARN took 18.282 seconds.

HIVE ON TEZ

Now, let’s run the same query on Tez engine.

To make a Hive query run on Tez engine, we need to set the Hive engine explicitly by using the below command:

We can see that Hive on Tez took 10.177 seconds to run the same query.

We can check whether the job is running in YARN or TEZ engine by checking it in the Resource manager’s web UI.

localhost:8088

In the above screen shot, we can see the job application_id and its Application type. Application type gives the engine on which the script had run. In the above screen shot, we have the application id’s and their engines for Hive, on which we ran the earlier query.

By this we can say that Tez engine is faster than YARN engine.

Hope this post has provided you a clear picture about running Pig scripts and Hive queries on both YARN and Tez engine and analyzing their performances.