Big Data Hadoop on the Go…!!

Posted by

Hola amigos! In our last post, we discussed few techniques which would make installation of software easy on Linux platforms such as Ubuntu.  As promised before, today we are going to gift you’ll with some easy steps or procedures to install Big Data Hadoop v2.7.3 in Linux platforms. Big data versions such as 2.7.2, 2.7.3 and v2.7.4 can be installed with higher java prerequisites in ubuntu workstations. But, I would personally like you’ll to suggest preferring Hadoop v2.7.4 instead of v2.7.2 due to configuration conflicts with platform services. The plugins favorable with v2.7.3 are Apache Sqoop v1.4.6 and Apache Hive v2.3.0. The steps for installing Hadoop v2.7.4 are listed below with a few examples.

First of all, you need to install the latest Java JDK(8) from the core site. Also, you can install the latest java by following the steps given HERE.

                          $wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz

  • Start an open SSH(Secure Shell) connection for secure authentication between the host and the client

                     $sudo apt-get install openssh-server

  • $sudo ufw allow 22
  • $sudo /etc/init.d/apache2 restart
  • Add a new user : $sudo adduser hadoop
  • $sudo visudo

Add this line below #root line — ‘hadoop ALL=(ALL:ALL)’

  • $update-alternatives –config java
  • $su – hadoop
  • $ssh-keygen -t rsa
  • $cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  • $chmod 0600 ~/.ssh/authorized_keys
  • $ssh localhost
  • $cd ..
  • $sudo mv (your_username)/hadoop-2.7.3.tar.gz hadoop/
  • $cd hadoop/tar xzvf hadoop-2.7.3.tar.gz
  • Upon extraction we get all the subfiles of .tar file placed in the destination /home/hadoop or /usr/local
  • $mv hadoop-2.7.3 hadoop
  • $sudo nano ~/.bashrc
  • Set up the hadoop’s environment variables and destination path such as :
#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
#HADOOP VARIABLES END
  • $source ~/.bashrc
  • $sudo nano /home/hadoop/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Adding the above statement in the hadoop-env.sh file ensures that the value of JAVA_HOME variable will be available to Hadoop whenever it is started up.

  • $ sudo nano /home/hadoop/hadoop/etc/hadoop/hdfs-site.xml

(Copy contents of hdfs-site.xml of hadoop stuffs to opened hdfs-site.xml file)

Copy the contents given below in the <configuration> tags and paste it in the hdfs-site.xml

<configuration>
 <property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
</configuration>
  • $ sudo nano /home/hadoop/hadoop/etc/hadoop/core-site.xml 

(Copy contents of core-site.xml.xml of hadoop stuffs to opened file)

Copy the contents given below in the <configuration> tags and paste it in the core-site.xml

<configuration>
 <property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
 </property>

 <property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
 </property>
</configuration>
  • $ mv /home/hadoop/hadoop/etc/hadoop/mapred-site.xml.template /home/hadoop/hadoop/etc/hadoop/mapred-site.xml
  • $ sudo nano /home/hadoop/hadoop/etc/hadoop/mapred-site.xml

(Copy contents of mapred-site.xml of hadoop stuffs to opened file)

Copy the contents given below in the <configuration> tags and paste it in the mapred-site.xml

<configuration>
 <property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
 </property>
</configuration>
  • $sudo nano /home/hadoop/hadoop/etc/hadoop/yarn-site.xml

(Copy contents of yarn-site.xml of hadoop stuffs to opened file)

Copy the contents given below in the <configuration> tags and paste it in the yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

  • $ hdfs namenode -format
  • $ cd $HADOOP_HOME/sbin/
  • $ start-dfs.sh
  • $ jps
  • $ stop-dfs.sh

jnn

  • Check Hadoop’s version on your PC

$ Hadoop version

hadoop1.png

DOWNLOAD THE FULL STEPS MENTIONED ABOVE IN PDF FORMAT BY CLICKING ON THE SERVER LINKS BELOW:

I hope you like and appreciate the information we share through our posts! Thank you for a wonderful response and constant support. Your valuable feedbacks and suggestions will be appreciated.  In our next post, we’ll be sharing some easy steps to install Big data extension plugins such as Apache Sqoop, Apache Hive, and other plugins.  Stay Tuned!

 

Advertisements

One comment

  1. Hmm it looks like your website ate my first comment (it was super long) so I guess I’ll just sum
    it up what I had written and say, I’m thoroughly enjoying your blog.
    I as well am an aspiring blog blogger but I’m still new to everything.
    Do you have any recommendations for novice blog writers? I’d definitely appreciate it.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s