Q/A (Big Data) 2

Q - Hive Stores metadata in which type of database?

A - MySQL


Q - Why to analyze data?

A - (i) Exponential growth in a machine data over the last years.

     (ii) Growing no of machine and usage of IOT devices.


Q - From where, large data is generated?

  • Sensor data
  • Machine data
  • Business data


Q - What is communication model in hadoop?

  • Hadoop follows RPC communication.
  • Communication happens in every 3 sec (by default).
  • Slaves communicate to master.


Q - What are important configuration files in hadoop?

  • Hdfs-site.xml – Block replication
  • Core-site.xml – I/O settings , setting of Hadoop cluster
  • Hadoop-env.sh – Environment setup
  • Master
  • Slave
  • Mapred-site.sh – For setting a map reduce


Q - Big Data as an opportunity?
  • Cost Reduction - Cost effective storage system for huge data sets.
  • Next generation products - Automated car, Health care.
  • Faster and better decision making - Provides ways to analyze information quickly and make decisions.
  • Improved services or products - Evaluation of customer needs and satisfaction.

Q - What is different types of file format in Hadoop or Hive?
  • Text/CSV
  • SequenceFile
  • Avro
  • Parquet
  • RCFile (Record Columnar File)
  • ORC (Optimized Row Columnar)

Q - What is difference between Avro and parkout?
Avro - Avro is a row-based storage format. This format includes in each file, the definition of the scheme of your data in JSON format, improving interoperability and allowing the evolution of the scheme. Avro also allows block compression in addition to its divisibility, making it a good choice for most cases when using Hadoop.

Parquet - Parquet is a column-based binary storage format that can store nested data structures. This format is very efficient in terms of disk input/output operations when the necessary columns to be used are specified. This format is very optimized for use with Cloudera Impala.


Q - What is checkpoint in hadoop?
- If any changes happen in name node,It records in edit logs and in a regular interval, it merges with fs_image (this process is called checkpoint).

Comments

Post a Comment

If you have any doubts, Please let me know

Popular posts from this blog

Hadoop

Image
What is Hadoop ? Hadoop is a framework that allows us to store and process large data sets in parallel and distributed fashion. It was developed by Doug Cutting in 2005, was working in yahoo. It efficiently store and process large datasets ranging in size from gigabytes to petabytes of data. Instead of using one large computer to store and process the data, Hadoop allows clustering multiple computers to analyze massive datasets in parallel more quickly. Hadoop is used for Batch processing where, f irst data is collected, entered and then processed. Later, it produces batch result. It uses master-slave architecture where one master and multiple slave works in a cluster.. Hadoop uses commodity hardware. So, it is cost effective. It is scalable as per requirement. Hadoop deals with flat files in any format. Hadoop follows RPC communication model. By default, Communication happens in every 3 seconds where slave communicates to master. There are three components of Hadoop: HDFS  - Hado...
Read more

Rack Awareness Algorithm

Image
Block -   Block is a small chunk of data. It contains minimum amount of data that can be read or write. HDFS stores each file in terms of blocks. Block size in Hadoop 1x is 64 MB. Block size in Hadoop 2x is 128 MB. Files are split into 64 MB or 128 MB blocks depending on Hadoop version and then stored into the Hadoop file system. Why HDFS block size are large in size Reason for having HDFS blocks in large size is to reduce the cost of seek time. In general, the seek time is 10 ms and disk transfer rate is 100 MB/S. To make the seek time 1% of the disk transfer rate, the block size should be 100 MB. The default size HDFS block is 64 MB. Rack  - Rack is a collection of machine which are connected using same network switch. If the network goes down, all the machine in a network will go down. Rack Awareness algorithm came into the picture to overcome this problem. In Rack Awareness, NameNode chooses the DataNode which is closer to the same rack or nearby rack. NameNode maintains...
Read more

Big Data

Image
What is Big Data ? Big Data refers to large amount of data sets that can not be efficiently managed by the common database management system. Extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interaction. Data increasing day by day 1 ZB = 1000 EB 1 EB = 1000 PB 1 PB = 1000 TB 1 TB = 1000 GB 1 GB = 1000 MB 1 MB = 1000 KB 1 KB = 1000 BYTE Note - Around 2.5 quintillion bytes of data is generated every day in all over the world, calculated by IBM. Some Facts Every sec - 822 tweets on Twitter Every min - 510 Comments on Facebook Every min - 293000 statuses on Facebook Every min - 136000 photos uploads on Facebook Every hour - Walmart handles 1 million customer transaction Challenges In collecting data. In storing data. In processing data. In sharing and Securing data. In creating and utilizing meaningful insights from their data. Where we can loss data In Power Failure - If electricity i...
Read more