Q/A (Hadoop)

Q - What if Namenode fails in Hadoop?

A - The single point of failure in Hadoop 1x is Namenode. If Namenode gets fail, the whole Hadoop cluster will not work. Actually, there will not any data loss only the cluster work will be shut down, because Namenode is only the point of contact to all Datanodes and if the Namenode fails, all communication will stop.


Q - What is Data Locality?

A - If we bring the data from slave to master, it will cost network congestion + input output channel congestion , and at the same time master node will take a lot of time to process this huge amount of data. We can send this process to data, means we can send the logic to all slaves which contains data and perform processing in the slave itself, result will be sent to name node, will take less time.


Q - An a folder, 100 files are there. Each file size is 1 mb, if block size is 64 mb, total how many blocks will be created?

A - 100 blocks will be created.


Q - Explain what is heartbeat in HDFS?

A - Heartbeat is referred to a signal used between a data node and Name node, and between task tracker and job tracker, if the Name node or job tracker does not respond to the signal, then it is considered there is some issues with data node or task tracker.


Q - What happens when a data node fails?

A - When a data node fails...

  • Jobtracker and Namenode detect the failure.
  • On the failed node all tasks are re-scheduled.
  • Namenode replicates the user's data to another node.




Q - Explain what are the basic parameters of a Mapper?

A - The basic parameters of a Mapper are

  • LongWritable and Text
  • Text and IntWritable



Q - What Are Hadoop Daemons?

A - Daemons are the processes that run in the background. There are four primary daemons: Namenode, Datanode, Resource Manager (runs on master node for YARN), Node Manager (runs on slave node for YARN).


Q - Why divide the file into blocks?

A - Let’s assume that we don’t divide, now it’s very difficult to store a 100 TB file on a single machine. Even if we store, then each read and write operation on that whole file is going to take very high seek time. But if we have multiple blocks of size 128 MB then its become easy to perform various read and write operations on it compared to doing it on a whole file at once. So we divide the file to have faster data access i.e. reduce seek time.


Q - Why replicate the blocks in data nodes while storing?

A - Let’s assume we don’t replicate and only one yellow block is present on Datanode D1. Now if the data node D1 crashes we will lose the block and which will make the overall data inconsistent and faulty. So we replicate the blocks to achieve fault-tolerance.

Comments

  1. UnknownJune 25, 2021 at 7:30 PM

    Please add more interview questions as it is helping me a lot in my interview.

    ReplyDelete

Post a Comment

If you have any doubts, Please let me know

Popular posts from this blog

Hadoop

Image
What is Hadoop ? Hadoop is a framework that allows us to store and process large data sets in parallel and distributed fashion. It was developed by Doug Cutting in 2005, was working in yahoo. It efficiently store and process large datasets ranging in size from gigabytes to petabytes of data. Instead of using one large computer to store and process the data, Hadoop allows clustering multiple computers to analyze massive datasets in parallel more quickly. Hadoop is used for Batch processing where, f irst data is collected, entered and then processed. Later, it produces batch result. It uses master-slave architecture where one master and multiple slave works in a cluster.. Hadoop uses commodity hardware. So, it is cost effective. It is scalable as per requirement. Hadoop deals with flat files in any format. Hadoop follows RPC communication model. By default, Communication happens in every 3 seconds where slave communicates to master. There are three components of Hadoop: HDFS  - Hado...
Read more

Rack Awareness Algorithm

Image
Block -   Block is a small chunk of data. It contains minimum amount of data that can be read or write. HDFS stores each file in terms of blocks. Block size in Hadoop 1x is 64 MB. Block size in Hadoop 2x is 128 MB. Files are split into 64 MB or 128 MB blocks depending on Hadoop version and then stored into the Hadoop file system. Why HDFS block size are large in size Reason for having HDFS blocks in large size is to reduce the cost of seek time. In general, the seek time is 10 ms and disk transfer rate is 100 MB/S. To make the seek time 1% of the disk transfer rate, the block size should be 100 MB. The default size HDFS block is 64 MB. Rack  - Rack is a collection of machine which are connected using same network switch. If the network goes down, all the machine in a network will go down. Rack Awareness algorithm came into the picture to overcome this problem. In Rack Awareness, NameNode chooses the DataNode which is closer to the same rack or nearby rack. NameNode maintains...
Read more

Big Data

Image
What is Big Data ? Big Data refers to large amount of data sets that can not be efficiently managed by the common database management system. Extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interaction. Data increasing day by day 1 ZB = 1000 EB 1 EB = 1000 PB 1 PB = 1000 TB 1 TB = 1000 GB 1 GB = 1000 MB 1 MB = 1000 KB 1 KB = 1000 BYTE Note - Around 2.5 quintillion bytes of data is generated every day in all over the world, calculated by IBM. Some Facts Every sec - 822 tweets on Twitter Every min - 510 Comments on Facebook Every min - 293000 statuses on Facebook Every min - 136000 photos uploads on Facebook Every hour - Walmart handles 1 million customer transaction Challenges In collecting data. In storing data. In processing data. In sharing and Securing data. In creating and utilizing meaningful insights from their data. Where we can loss data In Power Failure - If electricity i...
Read more