As is the case with scripts in other programming languages such as SQL, Unix Shell, etc., Pig scripts are used to execute a set of Apache Pig commands collectively. This helps in reducing the time and effort invested in writing and executing each command manually while doing the Pig programming. This blog (Pig Programming: Create Your First Apache Pig script) is a step by step guide to help you create your first Apache Pig script.
Pig Programming: Create Your First Apache Pig Script
An Apache Pig script works in two modes:
Local Mode: In ‘local mode’, you can execute the pig script in local file system. In this case you don’t need to store the data in Hadoop HDFS file system, instead you can work with the data stored in local file system itself.
HDFS Mode: In ‘HDFS mode’, the data needs to be stored in HDFS file system and you can process the data with the help of pig script.
Pig Script in HDFS Mode:
Step1: Writing a script
Open an editor (e.g. gedit) in your Cloudera Demo VM environment:
Command: gedit sample.pig
This command will create a ‘sample.pig’ file inside the home directory of cloudera user.
Let’s write few PIG commands in the sample script.
Let us say our task is to read data from a data file and to display the required contents on the as output.
The sample data file contains following data:
Shabbir Khan 9314573259 Bangalore Engineer
Manish Sharma 8882148796 Gurgaon Lecturer
Mahesh Kumar 8521548932 Noida Business
Sampath Reddy 8547987412 Hyderabad Engineer
Mohan Reddy 9256458798 Hyderabad Professor
Save the text file with the name ‘information.txt’
The sample data file contains five columns FirstName, LastName, MobileNo, City, and Profession separated by tab key. Our task is to read the content of this file in to HDFS and display First Name, Mobile Number and Profession of these contacts.
To process this data using Pig, this file should be present in Apache Hadoop HDFS.
Use the following command:
Command: hadoop dfs –copyFromLocal information.txt hdfs:/
Edit the Pig script (sample.pig) to include following commands:
A = LOAD ‘/information.txt’ using PigStorage (‘\t’) as (FName: chararray, LName: chararray, MobileNo: chararray, City: chararray, Profession: chararray);
B = FOREACH A generate FName, MobileNo, Profession;
Save and close the file.
The first command loads the file ‘information.txt’ into variable A with indirect schema (FName, LName, MobileNo, City, Profession).
The second command loads the required data from variable A to variable B.
The third line displays the content of variable B on the terminal/console.
Step 2: Execute the Pig Script
To execute the pig script in HDFS mode, run the following command:
Command: pig sample.pig
Review the result.
Congratulations on executing your first Pig script successfully!