Explore topic-wise InterviewSolutions in Current Affairs.

Steps for data preparation are:

• Gather data: The data preparation process starts with obtaining the correct data. This can originate from a current data catalogue or can be appended ad-hoc.
• Discover and assess data: After assembling the data, it is essential for each dataset to be identified. This step is about learning to understand the data and knowing what MUST be done before the data becomes valuable in a distinct context. Discovery is a big task but can be done with the help of data visualization tools that assist users and help them browse their data.
• Clean and verify data:
  Even though cleaning and verifying data takes a lot of time, it is the most important step, because this step not only eliminates the incorrect data but also fills the rifts. Significant tasks here include:
  • ELIMINATING alien data and outliers.
  • Filling in missing values.
  • Adjusting data to a regulated pattern.
  • Masking private or sensitive data entries.
    After cleaning the data, the mistakes that we came across during the data development process have to be examined and approved. Generally, an error in the system will become apparent during this step and need to be fixed before proceeding.
• Transform and enrich data:
  Transforming data modernizes the arrangement or value entries to reach a well-defined result or make the data more quickly recognized by broader viewers. Improving data refers to adding and CONNECTING data with other SIMILAR INFORMATION to provide deeper insights.
• Store data:
  Lastly, the data can be collected or channeled into a third-party application like a business intelligence tool-making technique for processing and analysis.

Important Resources

Big Data Tools

Big Data Engineer

Applications of Big Data

Big Data Projects

Highest Paying Jobs

Data preparation is the method of cleansing and modifying raw data before processing and analyzing it. It is a crucial step before processing and usually requires reformatting data, making improvements to data, and CONSOLIDATING data sets to enrich data.

Data preparation is an unending task for data specialists or business users. But, it is essential to convert data into context to get insights and then, can eliminate the biased results FOUND due to poor data quality.

For instance, the data construction PROCESS typically includes standardizing data formats, enhancing source data, and/or ELIMINATING outliers.

An open-ended question and there are many ways to achieve this.

• Programming:  Coding/ Programming is the most tried out method to transform unstructured data into a structured form. Programming is advantageous to accomplish because we get independence with it, which you can use to change the STRUCTURE of the data in any form possible. Several programming languages, such as Python, JAVA, etc., can be used.
• Data/Business Tools: Many BI (Business Intelligence) tools support the drag and drop functionality for converting unstructured data into structured data. One cautious thing before using BI tools is that most of these tools are paid, and we have to be financially capable to support these tools. For people who lack both experience and skills needed for option 1, this is the way to GO.

Pros of Big DATA are:

• Increased productivity: Recently, it was found that 59.9% of businesses use big data tools like Hadoop and Spark to develop their sales. Current big data tools enable analysts to examine instantly, which enhances their productivity. Also, the insights inferred from the analysis of big data can be used by organizations to increase productivity in different forms throughout the company.
• Reduce costs: Big data analytics help businesses reduce their costs. In most companies, big data tools had served them to enhance operational performance and decrease costs, and in few other companies had started using big data to decrease EXPENSES. Interestingly, very few companies selected cost reduction as their primary goal for big data analytics, suggesting that this is merely a very welcome SIDE benefit for many.
• Improved customer service: Improving customer service has always been one of the primary goals for big data analytics projects, and it has been a success for many companies with the help of this. Various customer contact points like Social media, customer relationship management systems, etc., transfer a lot of information about their customers. And this analysis and data is used to improve the services for the customers
• Fraud detection: The primary purpose of using Big data analytics is in the financial services industry for detecting frauds. The advantage of big data analytics systems is that it depends on machine learning, because of which they are great at recognizing patterns and irregularities. As a result, these techniques can give banks and credit card companies the capacity to detect stolen credit cards or deceitful purchases, usually before the cardholder knows that something is wrong.
• More significant innovation: A few companies have started investing in analytics with the sole purpose to bring new things and disturb their markets. The REASON behind this is if they can see the future of the market with the help of insights before their competitors, they can come out strong from that situation with a few new goods and services and capture the market quickly.

On the other hand, implementing big data analytics is not as easy as we think; there are a few difficulties too when it comes to implementing it. 

Cons of Big Data are:

• Need for talent: The number one big data challenge that we have been facing for the past three years is the skill set required for it. A lot of companies also face difficulty when designing a data lake. Hiring or training staff will only increase the cost considerably, and also, imbibing big data skills takes a lot of time.
• Cybersecurity risks: Storing, especially sensitive big data will make those businesses a prime target for cyberattackers. Security is one of the top big data challenges, and cybersecurity breaches are the single greatest data threat that enterprises encounter.
• Hardware needs: Another critical concern for businesses is the IT base necessary to help big data analytics drives. Storage SPACE for storing the data, networking bandwidth for transferring it to and from analytics systems, and calculating resources to achieve those analytics are costly to buy and keep.
• Data quality: The disadvantage in working with big data was the requirement to address data quality problems. Before companies can use big data for analytics purposes, data scientists and analysts need to ensure that the data they are working with is accurate, appropriate, and in the proper format for analysis. This slows the process, but if companies don't take care of data quality issues, they may find that the insights produced by their analytics are useless or even harmful if performed.

• Persistent znodes:  The default znode in ZooKeeper is the Persistent Znode. It permanently stays in the zookeeper server until any other clients leave it apart.
• Ephemeral znodes: These are the temporary znodes. It is smashed WHENEVER the creator CLIENT logs out of the ZooKeeper server. For example, ASSUME client1 created eznode1. Once client1 logs out of the ZooKeeper server, the eznode1 gets destroyed.
• Sequential znodes: Sequential znode is assigned a 10-digit NUMBER in numerical order at the end of its name. Assume client1 produced a sznode1. In the ZooKeeper server, the sznode1 will be named like this:
  sznode0000000001.
  If client1 generates another sequential znode, it will bear the following number in a sequence. So the subsequent sequential znode is <znode name>0000000002.

It is a Tool which is used for copying a very large AMOUNT of data to and from HADOOP file systems in parallel. It uses MapReduce to affect its distribution, ERROR handling, recovery, and reporting. It expands a LIST of files and directories into input to map tasks, each of which will copy a PARTITION of the files specified in the source list.   

Outliers are the data points that are very far from the group, which is not a part of any group or cluster. This may affect the behavior of the model, they may PREDICT wrong RESULTS, or their accuracy will be very low. Therefore Outliers must be handled CAREFULLY as they may also contain some helpful information. The presence of these outliers may lead to MISLEADING a Big Data model or a MACHINE Learning Model. The results of this may be, 

• Poor Results
• Lower accuracy
• Longer Training Time

Hadoop can provide an option wherein a particular set of lousy input records could be skipped while processing map inputs. SkipBadRecords class in Hadoop OFFERS an optional mode of execution in which the bad records can be detected and neglected in multiple ATTEMPTS. This may happen due to the presence of some bugs in the map FUNCTION. The USER has to manually fix it, which may sometimes be possible because the bug may be in third-party libraries. With the help of this feature, only a small amount of data is lost, which may be acceptable because we are dealing with a large amount of data.

The chief configuration parameters that the user of the MapReduce FRAMEWORK needs to mention is:

• JOB’s INPUT Location
• Job’s Output Location
• The Input format
• The Output format
• The Class INCLUDING the Map function
• The Class including the REDUCE function
• JAR file, which includes the mapper, the Reducer, and the driver classes.

• Heterogeneity: The design of applications should allow the users to access services and run applications over a heterogeneous collection of computers and networks, considering hardware DEVICES, OS, Network, PROGRAMMING languages.
• Transparency: Distributed system Designers must hide the complexity of the system as much as they can. Some Terms of transparency are location, access, migration, relocation, and so on.
• Openness: It is a CHARACTERISTIC that determines whether the system can be extended and reimplemented in various ways.
• Security: Distributed system Designers must take care of confidentiality, INTEGRITY, and availability.
• Scalability: A system is said to be scalable if it can manage the increase of users and resources without UNDERGOING a striking loss of performance.

Missing values in Big Data generally refer to the values which aren’t present in a particular column, in the worst case they may lead to erroneous data and may provide incorrect results. There are several techniques used to deal with the missing values they are - 

• Mean or Median Imputation:
  When data is missing at irregular intervals, we can use list-wise or pair-wise deletion of the missing observations. Still, there can be multiple reasons why this may not be the most workable option:
  • There may not be enough notes with non-missing data to produce a reliable analysis
  • In predictive analytics, missing data can prevent the forecasts for those observations which have missing data
  • External factors may REQUIRE SPECIFIC observations to be part of the analysis
    In such cases, we impute values for missing data. A simple technique is to use the mean or median of the non-missing observations. This can be useful in cases where the number of missing observations is low. However, for many missing values, using mean or median can result in loss of variation in data, and it is better to use imputations.
• Multivariate Imputation by Chained Equations (MICE):
  MICE believes that the missing data are Missing at Random (MAR). It imputes data on a variable-by-variable basis by specifying an imputation model per variable. MICE uses predictive mean matching (PMM) for continuous variables, logistic regressions for binary variables, bayesian polytomous regressions for factor variables, and PROPORTIONAL odds model for ordered variables to impute missing data.
  To set up the data for MICE, it is essential to note that the algorithm uses all the variables in the data for predictions. In this case, variables that may not be useful for predictions, like the ID variable, should be removed before implementing this algorithm.
  
  Data$ID <- NULL
  Secondly, as MENTIONED above, the algorithm treats different variables differently. So, all categorical variables should be treated as factor variables before implementing MICE.
  
  Data$year <- as.factor(Data$year)
  Data$gender <- as.factor(Data$gender)
  Then you can implement the algorithm using the MICE library in R
  library(mice)
  init = mice(Data, maxit=0)
  method = init$method
  predMat = init$predictorMatrix
  set.seed(101)
  imputed = mice(Data, method=method, predictorMatrix=predMat, m=5)
  You can also ignore some variables as predictors or skip a variable from being imputed using the MICE library in R. Additionally, the library also allows you to set a method of imputation discussed above depending upon the nature of the variable.
• Random Forest:
  Random forest is a non-parametric imputation method appropriate to multiple changeable types that work well with both data missing at random and not missing at irregular. Random forest uses several decision trees to discover missing values and outputs OOB imputation error estimates.
  One warning is that random forest works best with large datasets, and using random forest on small datasets encompasses the RISK of overfitting. The extent of overfitting leading to false imputations will depend upon how closely the distribution for predictor variables for non-missing data resembles the distribution of predictor variables for missing data. For example, suppose the distribution of race/ethnicity for non-missing data is similar to the distribution of race/ethnicity for missing data. In that case, overfitting is not likely to throw off results. However, if the two distributions differ, the accuracy of imputations will suffer.

-compress-codec parameter is GENERALLY used to get the output file of a sqoop import in FORMATS other than .GZ.

The following commands will help you RESTART NameNode and all the daemons:

You can STOP the NameNode with ./sbin /Hadoop-daemon.sh stop NameNode COMMAND and then start the NameNode USING ./sbin/Hadoop-daemon.sh start NameNode command.You can stop all the daemons with the ./sbin/stop-all.sh command and then start the daemons using the ./sbin/start-all.sh command.

During processing, Big data may contain a large amount of data that is not REQUIRED at a particular time, So we may be required to select only some specific features that we are interested in. The PROCESS of extracting only the needed features from the Big data is called FEATURE selection.

Feature selection Methods are -

• Filters Method: In this method of variable ranking, we only consider the importance and usefulness of a feature.
• Wrappers Method: In this method, ‘INDUCTION algorithm’ is used, Which can be used to produce a classifier.
• Embedded Method: This method is a combination of EFFICIENCIES of both Filters and wrappers methods.

In general partitioning in Hive is a logical division of tables into related columns such as date, city, and department based on the values of partitioned columns. Then these partitions are subdivided into buckets so that they provide extra structure to the data that may be used for more EFFICIENT querying.

Now let’s experience data partitioning in Hive with an instance. Consider a table named Table1. The table contains client details like id, NAME, DEPT, and year of joining. ASSUME we need to retrieve the details of all the clients who JOINED in 2014.

Then, the query examines the whole table for the necessary data. But if we partition the client data by the year and save it in a different file, this will decrease the query processing time. 

The command used for copying data from the LOCAL system to HDFS is:
HADOOP FS –copyFromLocal [source][destination]

• Robust: It is extremely robust and easy to use. In addition, it has COMMUNITY support and contribution.
• Full Load: Loading a table in Sqoop can be done in one command. Multiple tables can also be loaded in the same process.
• INCREMENTAL Load: Incremental load functionality is also supported. Whenever the table is updated, with the help of Sqoop, it can be loaded in parts too.
• Parallel import/export: Importing and exporting of data is done by the YARN FRAMEWORK. It also provides fault tolerance too.
• Import results of SQL query: It allows us to import the OUTPUT from the SQL query into the Hadoop Distributed File SYSTEM.

By default, the replication factor is 3. There are no TWO copies that will be on the same data node. Usually, the first two copies will be on the same rack, and the third copy will be off the shelf. It is advised to set the replication factor to at least three so that ONE copy is always safe, even if SOMETHING happens to the rack.

We can set the default replication factor of the file system and each file and directory exclusively. We can lower the replication factor for files that are not essential, and critical files should have a HIGH replication factor.

Hadoop’s most remarkable TECHNIQUE for ADDRESSING big data challenges is its capability to divide and conquer with Zookeeper. After the problem has been divided, the conquering relies on employing distributed and parallel processing methods ACROSS the Hadoop cluster.
The interactive tools cannot provide the insights or timeliness needed to make business judgments for big data problems. In those cases, you need to build distributed applications to solve those big data problems. Zookeeper is Hadoop’s way of coordinating all the elements of these distributed applications.
Zookeeper as technology is simple, but its features are powerful. ARGUABLY, it would be difficult, if not impossible, to create resilient, fault-tolerant distributed Hadoop applications without it.

Benefits of using a Zookeeper are:

• Simple distributed coordination process: The coordination process among all nodes in Zookeeper is straightforward.
• Synchronization: Mutual exclusion and co-operation among server processes.
• Ordered Messages: Zookeeper tracks with a number by denoting its order with the stamping of each update; with the help of all this, messages are ordered here.
• Serialization: Encode the data according to specific rules. Ensure your application runs consistently.
• Reliability: The zookeeper is very reliable. In CASE of an update, it keeps all the data until forwarded.
• Atomicity: Data transfer either succeeds or fails, but no transaction is partial.

Overfitting is generally a modeling error referring to a model that is TIGHTLY fitted to the data, i.e. When a modeling function is closely fitted to a limited data set. Due to Overfitting, the predictivity of such models gets reduced. This effect leads to a decrease in generalization ability failing to generalize when applied OUTSIDE the sample data.

There are several Methods to avoid Overfitting; some of them are:

• Cross-validation: A cross-validation method refers to DIVIDING the data into multiple small TEST data sets, which can be used to tune the model.  
• Early stopping: After a certain NUMBER of iterations, the generalizing capacity of the model weakens; in order to avoid that, a method called early stopping is used in order to avoid Overfitting before the model crosses that point.                                                          
• Regularization: this method is used to penalize all the parameters except intercept so that the model generalizes the data instead of Overfitting.

Distributed Cache is a significant feature provided by the MapReduce Framework, practiced when you want to share the files across all nodes in a Hadoop CLUSTER. These files can be jar files or simple properties files. Hadoop's MapReduce framework allows the facility to cache small to moderate read-only files such as TEXT files, zip files, jar files, etc., and DISTRIBUTE them to all the Datanodes(worker-nodes) MapReduce jobs are running. All DATANODE gets a copy of the file(local-copy), which Distributed Cache sends. 

The core methods of a REDUCER are:

• setup(): setup is a method CALLED just to configure different parameters for the reducer.
• reduce(): reduce is the primary operation of the reducer. The specific function of this method includes DEFINING the TASK that has to be worked on for a DISTINCT set of values that share a key.
• cleanup(): cleanup is used to clean or delete any temporary files or data after performing reduce() task.

MapReduce is a programming model created for distributed computation on big data sets in parallel. A MapReduce model has a map FUNCTION that performs filtering and sorting and a reduced function, which serves as a summary operation.  

MapReduce is an important PART of the Apache Hadoop open-source ecosystem, and it’s extensively used for querying and selecting data in the Hadoop Distributed File System (HDFS). A variety of queries may be done DEPENDING on the broad spectrum of MapReduce algorithms possible for creating data selections. In addition, MapReduce is fit for iterative computation involving LARGE quantities of data requiring parallel processing. This is because it represents a data flow rather than a procedure.  

The more enhanced data we produce and accumulate, the HIGHER the need to process all that data to make it usable. MapReduce’s iterative, parallel processing programming model is a good tool for creating a sense of big data.

Hadoop MAPREDUCE is a software framework for processing enormous data sets. It is the main component for data processing in the Hadoop framework. It divides the INPUT data into several parts and runs a program on every data component parallel. The word MapReduce refers to two separate and different tasks. The first is the map operation, which takes a set of data and transforms it into a DIVERSE COLLECTION of data, where INDIVIDUAL elements are divided into tuples. The reduce operation consolidates those data tuples based on the key and subsequently modifies the value of the key.

Big Data processing methods analyze big data sets at a massive scale. Offline BATCH data processing is typically FULL power and full scale, tackling arbitrary BI SCENARIOS. In contrast, real-time stream processing is conducted on the most recent slice of data for data profiling to pick outliers, impostor transaction exposures, safety monitoring, etc. However, the most challenging task is to do fast or real-time ad-hoc analytics on a big comprehensive data set. It substantially means you need to scan tons of data within seconds. This is only PROBABLE when data is processed with HIGH parallelism.

Different techniques of Big Data Processing are:

• Batch Processing of Big Data
• Big Data Stream Processing 
• Real-Time Big Data Processing
• Map Reduce