How to Install and Configure Apache Hadoop on a Single Node in CentOS 7

Step 4: Format Hadoop Namenode

17. Once hadoop single node cluster has been setup it’s time to initialize HDFS file system by formatting the /opt/volume/namenode storage directory with the following command:

$ hdfs namenode -format
Format Hadoop Namenode
Format Hadoop Namenode
Hadoop Namenode Formatting Process
Hadoop Namenode Formatting Process

Step 5: Start and Test Hadoop Cluster

18. The Hadoop commands are located in $HADOOP_HOME/sbin directory. In order to start Hadoop services run the below commands on your console:

$ start-dfs.sh
$ start-yarn.sh

Check the services status with the following command.

$ jps
Start and Test Hadoop Cluster
Start and Test Hadoop Cluster

Alternatively, you can view a list of all open sockets for Apache Hadoop on your system using the ss command.

$ ss -tul
$ ss -tuln # Numerical output
Check Apache Hadoop Sockets
Check Apache Hadoop Sockets

19. To test hadoop file system cluster create a random directory in the HDFS file system and copy a file from local file system to HDFS storage (insert data to HDFS).

$ hdfs dfs -mkdir /my_storage
$ hdfs dfs -put LICENSE.txt /my_storage
Check Hadoop Filesystem Cluster
Check Hadoop Filesystem Cluster

To view a file content or list a directory inside HDFS file system issue the below commands:

$ hdfs dfs -cat /my_storage/LICENSE.txt
$ hdfs dfs -ls /my_storage/
List Hadoop Filesystem Content
List Hadoop Filesystem Content
Check Hadoop Filesystem Directory
Check Hadoop Filesystem Directory

To retrieve data from HDFS to our local file system use the below command:

$ hdfs dfs -get /my_storage/ ./
Copy Hadoop Filesystem Data to Local System
Copy Hadoop Filesystem Data to Local System

Get the full list of HDFS command options by issuing:

$ hdfs dfs -help

Step 6: Browse Hadoop Services

20. In order to access Hadoop services from a remote browser visit the following links (replace the IP Address of FQDN accordingly). Also, make sure the below ports are open on your system firewall.

For Hadoop Overview of NameNode service.

http://192.168.1.41:50070 
Access Hadoop Services
Access Hadoop Services

For Hadoop file system browsing (Directory Browse).

http://192.168.1.41:50070/explorer.html
Hadoop Filesystem Directory Browsing
Hadoop Filesystem Directory Browsing

For Cluster and Apps Information (ResourceManager).

http://192.168.1.41:8088 
Hadoop Cluster Applications
Hadoop Cluster Applications

For NodeManager Information.

http://192.168.1.41:8042 
Hadoop NodeManager
Hadoop NodeManager

Step 7: Manage Hadoop Services

21. To stop all hadoop instances run the below commands:

$ stop-yarn.sh
$ stop-dfs.sh
Stop Hadoop Services
Stop Hadoop Services

22. In order to enable Hadoop daemons system-wide, login with root user, open /etc/rc.local file for editing and add the below lines:

$ su - root
# vi /etc/rc.local

Add these excerpt to rc.local file.

su - hadoop -c "/opt/hadoop/sbin/start-dfs.sh"
su - hadoop -c "/opt/hadoop/sbin/start-yarn.sh"
exit 0
Enable Hadoop Services at System-Boot
Enable Hadoop Services at System-Boot

Then, add executable permissions for rc.local file and enable, start and check service status by issuing the below commands:

$ chmod +x /etc/rc.d/rc.local
$ systemctl enable rc-local
$ systemctl start rc-local
$ systemctl status rc-local
Enable and Check Hadoop Services
Enable and Check Hadoop Services

That’s it! Next time you reboot your machine the Hadoop services will be automatically started for you! All you need to do is to fire-up a Hadoop compatible application and you’re ready to go!

For additional information please consult official Apache Hadoop documentation webpage and Hadoop Wiki page.

If you liked this article, then do subscribe to email alerts for Linux tutorials. If you have any questions or doubts? do ask for help in the comments section.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

Support Us

We are thankful for your never ending support.

46 thoughts on “How to Install and Configure Apache Hadoop on a Single Node in CentOS 7”

  1. Hi everybody

    I install the Hadoop but now i try to upload file a get error Permission denied: user=dr.who, access=WRITE, inode=”/”:hadoop:supergroup:drwxr-xr-x

    Reply
  2. Following this guide while doing this step – ssh-copy-id master.

    I am getting errors:

    /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys
    ##################################################################################
    NOTE: You have accessed a system owned by FIWARE Lab. You must have authorisation
    before using it, and your use will be strictly limited to that indicated in the
    authorisation.
    Unauthorised access to this system or improper use of the same is prohibited and
    is against the FIWARE Terms & Conditions Policy and the legislation in force. The
    use of this system may be monitored.
    #################################################################################

    Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

    Please help me out.

    Reply
  3. I am not able to start yarn, can please help me out here:

    [[email protected] ~]$ start-yarn.sh
    
    starting yarn daemons
    starting resourcemanager, logging to /opt/hadoop/logs/yarn-hadoop-resourcemanager-master.out
    [email protected]'s password: [email protected]'s password: 
    192.168.25.129: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-master.out
    
    
    [email protected]'s password: master.hadoop.com: Permission denied, please try again.
    
    [email protected]'s password: master.hadoop.com: Permission denied, please try again.
    
    master.hadoop.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
    
    Reply

Got something to say? Join the discussion.

Have a question or suggestion? Please leave a comment to start the discussion. Please keep in mind that all comments are moderated and your email address will NOT be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.