Learn. Do. Earn.

Category: Spark

Introduction to Spark 2.X

Getting-started-with-Spark-2
In this post, we will be discussing the new features of Spark 2.0.0 and its installation in Hadoop 2.7. We highly recommend our readers to go through the below posts on Spark, to get a clear idea of what Spark is and the reasons behind its popularity. Beginner’s Guide...
Read More
Leave a Comment

Social Media Analysis Using Apache Flink

Social media Analysis using Flink
In this post, we will be looking at a case study to calculate the average number of friends based on their age, on a social media website using Apache Flink in Scala. In our previous post, we had a brief introduction to Flink. Hence, we request you to go...
Read More
Leave a Comment

Beginner’s Guide for Apache Flink

Introduction to Apache Flink
In this post, we will be discussing Apache Flink, its installation in a single node cluster and how it is a contender for the present Big Data frameworks. Let’s begin with the basics. What is Apache Flink? Apache Flink is an open-source platform for distributed stream and batch data...
Read More
Leave a Comment

Streaming Twitter Data using Spark

Spark-Use-Case
In this post, we will be discussing how to stream Twitter data using Spark Streaming. Let’s begin with what Spark Streaming is. Before going to spark streaming, we recommend our users to get some idea on Spark core and RDD’s. Spark RDD’s in Scala part-1 Spark RDD’s in Scala...
Read More
Leave a Comment

Spark Use Case – Popular Movie Analysis

Spark-Use-Case
In this blog, we will work on a case study to find the list of most popular movies. We will perform various transformations and actions to display a list of movies with maximum occurrence in the given data set. Let’s  start our discussion with the data definition by considering...
Read More
Leave a Comment

Spark Use Case – Weather Data Analysis

Spark-Use-Case
In this post, we will work on a case study to find the minimum temperature observed in a given weather station in a particular year. Let’s begin by considering a sample of four records. Data Definition: Column 1: Weather Station Column 2: Date(year/Month/Day) Column 3: Observation Type Column 4:...
Read More
Leave a Comment

Spark Use Case – Social Media Analysis

Spark-Use-Case
In this post, we will work on a case study to calculate the average number of friends based on their age, on a social media website. Let’s begin by considering a sample of four records. Column 1: User ID Column 2: User Name Column 3: Age of the User...
Read More
Leave a Comment

Implementing Custom Input Format in Spark

Custom input format in spark
In this post, we will be discussing how to implement custom input format in Spark. In Spark, we will implement the custom input format by using Hadoop custom input format. You can refer to our previous post to get an idea of how custom input format has been implemented...
Read More
Leave a Comment
© Copyright 2016.  ACADGILD.