Explore topic-wise InterviewSolutions in .

• Heterogeneity: The design of applications should allow the users to access services and run applications over a heterogeneous collection of computers and networks taking into consideration Hardware devices, OS, networks, Programming languages.
• Transparency: Distributed system Designers must hide the complexity of the system as MUCH as they can. Some Terms of transparency are location, access, migration, RELOCATION, and so on.
• Openness: It is a characteristic that determines whether the system can be extended and reimplemented in VARIOUS ways.
• Security: Distributed system Designers must take care of confidentiality, integrity, and availability.
• Scalability: A system is said to be scalable if it can handle the addition of users and RESOURCES without SUFFERING a noticeable loss of performance.

Recommended Resource:
Spark Interview

In general Apache Flume architecture is composed of the following components:

1. Flume SOURCE
2. Flume Channel
3. Flume Sink
4. Flume Agent
5. Flume Event

1. Flume Source: Flume Source is available on various networking PLATFORMS like Facebook or Instagram. It is a Data generator that collects data from the generator, and then the data is transferred to a Flume Channel in the form of a Flume.
2. Flume Channel: The data from the flume source is sent to an Intermediate Store which buffers the events till they GET transferred into Sink. The Intermediate Store is called Flume Channel. Channel is an intermediate source. It is a bridge between Source and a Sink Flume channel. It supports both the Memory channel and File channel. The file channel is non-volatile which means once the data is entered into the channel, the data will never be lost unless you delete it. In contrast, in the Memory channel, events are stored in memory, so it’s volatile, meaning data may be lost, but Memory Channel is very fast in nature.
3. Flume sink: Data repositories like HDFS, have Flume Sink. Which takes Flume events from the Flume Channel and stores them into the Destination specified like HDFS. It is done in such a way where it should deliver the events to the Store or another agent. Various sinks like Hive Sink, THRIFT Sink, etc are supported by the Flume.
4. Flume Agent: A Java process that works on Source, Channel, Sink combination is called Flume Agent. One or more than one agent is possible in Flume. Connected Flume agents which are distributed in nature can also be collectively called Flume.
5. Flume Event: An Event is the unit of data transported in Flume. The general representation of the Data Object in Flume is called Event. The event is made up of a payload of a byte array with OPTIONAL headers.

APACHE Flume is a tool/service/data INGESTION mechanism for assembling, aggregating, and carrying huge amounts of streaming data such as record files, EVENTS from various references to a centralized data store.

Flume is a very stable, DISTRIBUTED, and configurable tool. It is generally designed to copy streaming data (LOG data) from various web servers to HDFS.

There are BASICALLY TWO file FORMATS sqoop allos to IMPORT DATA they are:

• Delimited Text File format
• Sequence File Format

By DEFAULT,the is table DATA in APACHE HIVE is stored in: Hdfs://namenode_server/user/hive/warehouse

If the SOURCE data GETS updated in a very short interval of time, the synchronization of data in HDFS that is imported by Sqoop is done with the HELP of incremental parameters.

We should USE incremental import along with the append choice even when the table is refreshed continuously with new rows. PRINCIPALLY where values of a few columns are examined, and if it encounters any revised value for those columns, only a new row will be inserted. Similar to incremental import, the origin has a date column examined for all the records that have been modified after the last import, depending on the previous revised column in the beginning. The values would be modernized.

Apache HIVE organizes tables into PARTITIONS. Partitioning is the manner in which a table is split into RELATED components depending on the values of APPROPRIATE columns like date, city, and department.

Every table in the hive can have one or more than one partition keys to recognize a distinct partition. With the help of partitions, it is effortless to do queries on SLICES of the data.

No, as of now MULTILINE COMMENTS are not SUPPORTED in HIVE, only single-line comments are supported.

Metastore is USED to store the metadata information; it’s also possible to use RDBMS and the open-source ORM layer, converting OBJECT Representation into a relational schema. It’s the central repository of Apache Hive metadata. It stores metadata for Hive tables (similar to their schema and LOCATION) and partitions in a relational database. It gives the client access to this information by USING metastore service API. Disk storage for the Hive metadata is SEPARATE from HDFS storage.

The APPLICATIONS that are SUPPORTED by APACHE HIVE are,        

• Java
• PHP
•  Python
• C++
• Ruby

The data is stored in MEMORY by APACHE Spark for faster processing and development of machine learning models, which may need a lot of Machine Learning algorithms for multiple repetitions and various CONCEPTUAL steps to create an optimized model. In the case of Graph algorithms, it moves within all the nodes and edges to make a graph. These LOW latency WORKLOADS, which need many iterations, enhance the performance.

Resilient Distributed DATASETS is the basic data structure of Apache Spark. It is INSTALLED in the Spark Core. They are immutable and fault-tolerant. RDDs are generated by transforming already PRESENT RDDs or storing an outer dataset from well-built storage like HDFS or HBase.

Since they have distributed collections of objects, they can be operated in parallel. Resilient Distributed Datasets are DIVIDED into parts such that they can be executed on various nodes of a CLUSTER.

The primary PARAMETERS of a mapper are text, LONGWRITABLE, text, and IntWritable. The initial TWO represent INPUT parameters, and the other two signify intermediate output parameters.

• Both the Jobtracker and the name node DETECT the failure on which blocks were the DataNode FAILED.
• On the failed node all the tasks are rescheduled by locating other DataNodes with copies of these blocks
• User’s DATA will be replicated to another node from namenode to maintain the CONFIGURED replication factor.

Distributed Cache is a significant feature provided by the MapReduce Framework, practiced when you want to share the FILES across all nodes in a Hadoop cluster. These files can be jar files or simple properties files.

Hadoop's MapReduce framework allows the facility to cache SMALL to moderate read-only files such as text files, ZIP files, jar files, ETC., and distribute them to all the Datanodes(worker-nodes) MapReduce jobs are running. All Datanode gets a COPY of the file(local-copy), which is sent by Distributed Cache. 

• The client application is USED to submit the jobs to the Jobtracker.
• The JobTracker associates with the NameNode to determine the data location.
• With the help of AVAILABLE slots and the near the data, JobTracker LOCATES TASKTRACKER nodes.
• It submits the work on the selected TaskTracker Nodes.
• When a TASK fails, JobTracker notifies and decides the further steps.
• JobTracker monitors the TaskTracker nodes.

The dfsadmin tools are a specific set of tools designed to HELP you root out information about your Hadoop Distributed FILE system (HDFS). As a BONUS, you can use them to perform some administration operations on HDFS as well.

The command used to find the status of the BLOCK  is:  HDFS fsck <PATH> -files –blocks
And the command used to find File-system HEALTH is: hdfs fsck/ -files –blocks –locations > dfs-fsck.log

The two types of metadata that NameNode server stores are in DISK and RAM.
Metadata is linked to two files which are:

• EditLogs: It contains all the latest CHANGES in the file SYSTEM regarding the last FsImage.
• FsImage: It contains the whole state of the namespace of the file system from the origination of the NameNode.

Once the file is deleted from HDFS, the NameNode will immediately store this in the EditLog.
All the file systems and metadata which are present in the Namenode’s Ram are read by the Secondary NameNode continuously and later get recorded into the file system or hard disk. EditLogs is combined with FsImage in the NameNode. Periodically, Secondary NameNode downloads the EditLogs from the NameNode, and then it is implemented to FsImage. The new FsImage is then copied back into the NameNode and used only after the NameNode has started the SUBSEQUENT time.

Hadoop provides an option where a particular set of lousy input records can be skipped when processing map inputs. APPLICATIONS can manage this feature through the SkipBadRecords class.
This feature can be USED when map tasks fail deterministically on a particular input. This USUALLY happens due to faults in the map FUNCTION. The user would have to fix these ISSUES. 

By default, the replication factor is 3. There are no two copies that will be on the same data NODE. Usually, the first two copies will be on the same rack, and the third copy will be off the shelf. It is advised to set the replication factor to at LEAST three so that one copy is always safe, EVEN if something happens to the rack.
We can set the default replication factor of the file system as WELL as of each file and directory exclusively. For files that are not essential, we can lower the replication factor, and critical files should have a high replication factor.

By default, the size of the HDFS data block is 128 MB. The ideas for the large size of blocks are:

• To reduce the expense of seek: Because of the large size blocks, the time consumed to shift the data from the disk can be longer than the usual time taken to commence the block. As a result, the multiple blocks are transferred at the disk TRANSFER rate.
• If there are small blocks, the NUMBER of blocks will be too many in Hadoop HDFS and too much metadata to STORE. Managing such a VAST number of blocks and metadata will create overhead and head to traffic in a network.