Big Data Tutorial

Advantages of Rack Awareness So , now you will be thinking why do we need a Rack Awareness algorithm? The reasons are: To improve the network performance:   The communication between nodes residing on different racks is directed via switch. In general, you will find  greater network bandwidth  between machines in the same rack than the machines residing in different rack. So, the Rack Awareness helps you to have reduce write traffic in between different racks and thus providing a better write performance. Also, you will be gaining increased read performance because you are using the bandwidth of multiple racks. To prevent loss of data:   We don’t have to worry about the data even if an entire rack fails because of the switch failure or power failure. And if you think about it, it will make sense, as it is said that  never put all your eggs in the same basket. # Rack AwarenessAlgorithm

Block -   Block is a small chunk of data. It contains minimum amount of data that can be read or write. HDFS stores each file in terms of blocks. Block size in Hadoop 1x is 64 MB. Block size in Hadoop 2x is 128 MB. Files are split into 64 MB or 128 MB blocks depending on Hadoop version and then stored into the Hadoop file system. Why HDFS block size are large in size Reason for having HDFS blocks in large size is to reduce the cost of seek time. In general, the seek time is 10 ms and disk transfer rate is 100 MB/S. To make the seek time 1% of the disk transfer rate, the block size should be 100 MB. The default size HDFS block is 64 MB. Rack  - Rack is a collection of machine which are connected using same network switch. If the network goes down, all the machine in a network will go down. Rack Awareness algorithm came into the picture to overcome this problem. In Rack Awareness, NameNode chooses the DataNode which is closer to the same rack or nearby rack. NameNode maintains...