Explore topic-wise InterviewSolutions in .

Spark is a MapReduce improvement in Hadoop. The difference between Spark and MapReduce is that Spark processes and retains DATA in memory for later STEPS, whereas MapReduce processes data on the disc. As a RESULT, Spark's data processing speed is up to 100 times quicker than MapReduce for lesser workloads. Spark also constructs a Directed Acyclic Graph (DAG) to schedule tasks and orchestrate nodes throughout the Hadoop cluster, as opposed to MapReduce's two-stage EXECUTION procedure. 

Apache Spark is an open-source distributed processing solution for big data workloads. For rapid queries against any SIZE of data, it uses in-memory caching and efficient query EXECUTION. Simply put, Spark is a general-purpose data processing ENGINE that is quick and SCALABLE.

The replication factor is the number of times the Hadoop FRAMEWORK replicates each Data BLOCK. Fault tolerance is provided by replicating the block. The replication factor is SET to 3 by default, however, it can be modified to 2 (less than 3) or raised to meet your NEEDS (more than 3.)

It is a utility or feature included with a Hadoop distribution that allows developers or programmers to construct Map-Reduce programs in many programming languages such as PYTHON, C++, Ruby, Pearl, and others. We can USE any language that can READ from standard INPUT (STDIN), such as keyboard input, and write using standard output (STDOUT).

XML CONFIGURATION FILES AVAILABLE in HADOOP are:

• Core-site
• Mapred-site
• Yarn-site
• HDFS-site

A snowflake schema is a logical ARRANGEMENT of tables in a multidimensional database that matches the snowflake shape (in the ER diagram).  A Snowflake Schema is an enlarged STAR Schema with additional dimensions. After the DIMENSION tables have been NORMALIZED, the data is separated into new tables. 

Snowflaking has the potential to improve the performance of certain queries. The schema is organized so that each fact is surrounded by its related dimensions, and those dimensions are linked to other dimensions, forming a snowflake pattern.

In a DATA warehouse, a star schema can include one fact table and a number of associated dimension TABLES in the center. It's called a star schema because its structure resembles that of a star. The simplest sort of Data Warehouse schema is the Star Schema data model. It is also known as the Star JOIN Schema, and it is designed for MASSIVE data sets.

Hadoop Distributed Cache is a Hadoop MAPREDUCE Framework TECHNIQUE that provides a service for copying read-only files, archives, or jar files to worker nodes before any job tasks are executed on that node. To MINIMIZE network bandwidth, files are usually COPIED only once per job. Distributed Cache is a program that distributes read-only data/text files, archives, jars, and other files.

Classification and Optimization-based Scheduling for HETEROGENEOUS Hadoop Systems (COSHH), as the name IMPLIES, enables scheduling at both the cluster and application levels to have a DIRECT positive impact on TASK completion time.

These are the main METHODS of reducer:

• setup(): This COMMAND is used to specify parameters such as the size of input data and the distributed cache.
• cleaning(): is a function for DELETING temporary files.
• reduce(): it's CALLED once per key with the CORRESPONDING reduced task.

Indexing is a technique for improving database performance by reducing the number of DISC ACCESSES necessary when a query is run. It's a DATA structure strategy for FINDING and accessing data in a database rapidly.

The following STEPS occur when the block scanner detects a corrupt data block:

• First and FOREMOST, when the Block Scanner detects a CORRUPTED data block, DataNode notifies NameNode.
• NameNode begins the process of constructing a new replica from a corrupted block replica.
• The replication factor is compared to the replication COUNT of the right replicas. The faulty data block will not be REMOVED if a match is detected.

The NAMENODE and the DATANODE COMMUNICATE VIA these messages:

• BLOCK reports
• Heartbeats

The heartbeat is a communication link that runs between the NAMENODE and the Datanode. It's the signal that the Datanode SENDS to the Namenode at REGULAR INTERVALS. If a Datanode in HDFS fails to send a heartbeat to Namenode after 10 MINUTES, Namenode assumes the Datanode is unavailable.

MapReduce is a programming MODEL and software framework for processing large volumes of data. MAP and Reduce are the two phases of MapReduce. The map TURNS a set of data into another set of data by breaking down individual elements into TUPLES (key/value pairs). Second, there's the reduction job, which takes the RESULT of a map as an input and condenses the data tuples into a smaller set. The reduction work is always executed after the map job, as the name MapReduce suggests.

Hadoop has the following COMPONENTS:

• Hadoop Common: A collection of Hadoop tools and libraries.
• Hadoop HDFS: Hadoop's storage unit is the Hadoop Distributed File SYSTEM (HDFS). HDFS stores data in a distributed fashion. HDFS is made up of two parts: a name node and a data node. While there is only one name node, numerous data nodes are possible.
• Hadoop MapReduce: Hadoop's PROCESSING unit is MapReduce. The processing is done on the slave nodes in the MapReduce technique, and the final result is delivered to the master node.
• Hadoop YARN: Hadoop's YARN is an acronym for Yet ANOTHER Resource Negotiator. It is Hadoop's resource management unit, and it is included in Hadoop version 2 as a component. It's in charge of managing cluster resources to avoid overloading a single machine.

• Block: In HDFS, a "block" refers to the smallest amount of data that may be read or written. 
• Block Scanner: Block Scanner keeps track of the LIST of blocks on a DataNode and checks them for checksum problems. To save disc bandwidth on the data node, Block SCANNERS USE a THROTTLING TECHNIQUE.

In an HDFS cluster, there is only one NAMENODE. This node keeps TRACK of DataNode metadata. Because there is only one NameNode in an HDFS cluster, it is the single POINT of failure. The system may become inaccessible if NameNode crashes. In a high-availability system, a PASSIVE NameNode backs up the PRIMARY one and takes over if the primary one fails.

The HDFS system is BUILT on the foundation of NameNode. It keeps track of where the DATA file is KEPT by STORING the DIRECTORY tree of the files in a single file system. 

HDFS is an acronym for Hadoop Distributed FILE System. It is a distributed file system that RUNS on commodity hardware and can handle massive DATA COLLECTIONS.

SQL, Amazon WEB Services, Hadoop, and Python are all required skills for data ENGINEERS. Other TOOLS critical for data engineers are PostgreSQL, MongoDB, Apache Spark, Apache Kafka, Amazon REDSHIFT, Snowflake, and Amazon Athena.

Hadoop has the following features:

• It is open-source and easy to use.
• Hadoop is extremely scalable. A significant volume of data is split across several devices in a cluster and processed in parallel. According to the needs of the hour, the number of these devices or nodes can be increased or decreased.
• Data in Hadoop is copied across multiple DATANODES in a Hadoop cluster, ensuring data availability even if one of your SYSTEMS fails.
• Hadoop is built in such a WAY that it can efficiently handle any TYPE of dataset, including structured (MySQL Data), semi-structured (XML, JSON), and unstructured (Images and Videos). This means it can analyze any type of data regardless of its form, making it extremely flexible.
• Hadoop provides faster data processing. More Features.

• Data science is a broad TOPIC of research. It focuses on extracting data from extremely HUGE datasets (sometimes it is known as "big data"). Data scientists can operate in a variety of fields, including industry, government, and applied sciences. All data scientists have the same goal: to analyze data and derive insights from it that are relevant to their field of work.
• A data engineer's job is to develop or integrate many COMPONENTS of complex systems, taking into account the information needed, the COMPANY's goals, and the end requirements. This necessitates the creation of extremely complicated data pipelines. These data pipelines, like oil pipelines, take raw, UNSTRUCTURED data from a variety of sources. They then channel them into a single database (or larger structure) for storage.

There are TWO design schemas available in data MODELING:

• Star SCHEMA
• Snowflake Schema

Data Modeling is the act of CREATING a visual representation of an entire information system or parts of it in order to express linkages between data points and structures. The purpose is to show the MANY types of data that are used and stored in the system, as well as the relationships between them, how the data can be classified and arranged, and its formats and features. Data can be modeled according to the needs and requirements at various DEGREES of abstraction. The process begins with stakeholders and end-users providing information about business requirements. These business rules are then converted into data structures, which are used to create a CONCRETE database design.

The APPLICATION of DATA COLLECTING and analysis is the emphasis of data ENGINEERING. The information gathered from numerous sources is merely raw information. Data engineering helps in the transformation of unusable data into useful information. It is the process of transforming, cleansing, profiling, and aggregating huge data sets in a NUTSHELL.