Explore topic-wise InterviewSolutions in .

Check point:

• When a graph FAILS in the middle of the process, a RECOVERY point is created, known as Check point
• The rest of the process will be continued after the check point
• Data from the check point is fetched and continue to execute after CORRECTION.

Phase:

• If a graph is created with PHASES, each phase is assigned to some part of memory one after another. 
• All the phases will run one by one
• The intermediate FILE will be deleted

Check point:

Phase:

• A graphical / program hand is known as deadlock.
• The progression of a program would be stopped when a dead lock OCCURS.
• Data flow pattern LIKELY CAUSES a deadlock
• If a graph flows diverge and CONVERGE in a single phase, it is potential for a deadlock
• A component might wait for the records to arrive on one flow during the flow converge, even though the unread data accumulates on OTHERS.
• In GDE version 1.8, the occurrence of a dead lock is very rare.

EME:

• EME stands for Enterprise Metadata Environment 
• It is a repository to AbInitio. It holds transformations, database configuration files, metadata and target INFORMATION

GDE: 

• GDE – Graphical Development Environment
• It is an end user environment. Graphs are DEVELOPED in this environment
• It PROVIDES GUI for editing and executing AbInitio PROGRAMS 

Co-operative System:

• Co-operative system is the server of AbInitio.
• It is installed on a specific OS platform known as Native OS. 
• All GENERATED graphs in GDE are later deployed and executed in co-operative system.

EME:

GDE: 

Co-operative System:

AbInitio supports 3 parallelisms. They are

Data PARALLELISM : Same data is PARALLELLY worked in a single application

COMPONENT Parallelism : Different data is worked parallelly in a single application

PIPELINE Parallelism : Data is passed from one component to another component. Data is worked on both of the components.

AbInitio supports 3 parallelisms. They are

Data Parallelism : Same data is parallelly worked in a single application

Component Parallelism : Different data is worked parallelly in a single application

Pipeline Parallelism : Data is passed from one component to another component. Data is worked on both of the components.

Duplicate records can be avoided by USING the following:

• Using Dedup sort
• Performing AGGREGATION
• Utilizing the ROLLUP component

Duplicate records can be avoided by using the following:

• MAX CORE is the space CONSUMED by a component that is used for calculations
• Each component has different MAX COREs
• Component performances will be INFLUENCED by the MAX CORE’s CONTRIBUTION
• The PROCESS may slow down / fasten if a wrong MAX CORE is set

• This function is similar to the function NVL() in Oracle database
• It performs the FIRST values which are not null among other values AVAILABLE in the function and ASSIGNS to the variable

Example: A set of variables, say v1,v2,v3,v4,v5,v6 are assigned with NULL.
Another variable NUM is assigned with value 340 (num=340)
num = first_defined(NULL, v1,v2,v3,v4,v5,v6,NUM) 
The result of num is 340

Example: A set of variables, say v1,v2,v3,v4,v5,v6 are assigned with NULL.
Another variable num is assigned with value 340 (num=340)
num = first_defined(NULL, v1,v2,v3,v4,v5,v6,NUM) 
The result of num is 340

• A decimal STRIP takes the decimal values out of the data.
• It trims any leading zeros
• The result is a VALID decimal number

Ex: 
decimal_strip("-0184o") := "-184" 
decimal_strip("oxyas97abc") := "97" 
decimal_strip("+$78ab=-*&^*&%cdw") := "78" 
decimal_strip("Honda") "0"

Ex: 
decimal_strip("-0184o") := "-184" 
decimal_strip("oxyas97abc") := "97" 
decimal_strip("+$78ab=-*&^*&%cdw") := "78" 
decimal_strip("Honda") "0"

To make a graph behave dynamically, PDL is used

• - Suppose there is a need to have a dynamic field that is to be added to a predefined DML while executing the graph
• - Then a graph level parameter can be DEFINED 
• - Utilize this parameter while embedding the DML in output port.

For Example : define a parameter named myfield with a VALUE “string(“ | “”) NAME;”

• Use ${mystring} at the time of embedding the dml in out port.
• Use $substitution as an interpretation option

To make a graph behave dynamically, PDL is used

For Example : define a parameter named myfield with a value “string(“ | “”) name;”

Following is the ORDER of EVALUATION:

• Host setup script will be executed first
• All Common parameters, that is, INCLUDED , are evaluated
• All Sandbox parameters are evaluated
• The project script – project-start.ksh is executed
• All FORM parameters are evaluated
• GRAPH parameters are evaluated
• The Start Script of graph is executed

Following is the order of evaluation:

Use decimal cast with the size in the transform() function, when the size of the STRING and decimal is same.

Ex: If the source field is defined as string(8).

- The DESTINATION is defined as decimal(8) 

- Let us ASSUME the field NAME is salary.

- The function is out.field :: (decimal(8)) in salary

- If the size of the destination field is lesser that the input then string_substring() function can be used

Ex : Say the destination field is decimal(5) then use…

- out.field :: (decimal(5))string_lrtrim(string_substring(in.field,1,5))

- The ‘ lrtrim ‘ function is used to remove leading and trailing spaces in the string

Use decimal cast with the size in the transform() function, when the size of the string and decimal is same.

Ex: If the source field is defined as string(8).

- The destination is defined as decimal(8) 

- Let us assume the field name is salary.

- The function is out.field :: (decimal(8)) in salary

- If the size of the destination field is lesser that the input then string_substring() function can be used

Ex : Say the destination field is decimal(5) then use…

- out.field :: (decimal(5))string_lrtrim(string_substring(in.field,1,5))

- The ‘ lrtrim ‘ function is used to remove leading and trailing spaces in the string

The following are the ways to IMPROVE the PERFORMANCE of a graph :

• Make sure that a limited number of components are used in a particular phase
• Implement the usage of optimum value of max core values for the purpose of sorting and joining components.
• Utilize the minimum number of SORT components
• Utilize the minimum number of sorted join components and replace them by in-memory join / hash join, if needed and possible
• Restrict only the needed fields in sort, REFORMAT, join components
• Utilize phasing or flow BUFFERS when merged or sorted joins
• Use sorted join, when two inputs are huge, otherwise use hash join

The following are the ways to improve the performance of a graph :

Partitioning by Key / Hash Partition :

• The partitioning technique that is used when the KEYS are diverse
• Large data skew can EXIST when the key is present in large volume
• It is apt for parallel data processing

Round Robin Partition : 

• This partition technique uniformly distributed the data on every DESTINATION data partitions
• When number of records is DIVISIBLE by number of partitions, then the skew is zero.

For example: a pack of 52 cards is distributed among 4 players in a round-robin fashion.

Partitioning by Key / Hash Partition :

Round Robin Partition : 

For example: a pack of 52 cards is distributed among 4 players in a round-robin fashion.

• Open Add Default Rules dialog box.
• Select Match Names – to match the names that generates a set of rules to copy input fields to out fields with same name
• Use WILDCARD(. *) RULE : This rule generates only one rule to copy input fields to output fields with the same name
• If not DISPLAYED – display the Transform Editor GRID
• CLICK the Business Rule tab . Select Edit?Add Default Rules
• Nothing is needed to write in the reformat .xfr file in case of reformat, if there is no need to use any real transform other than reducing the set of fields.

• ROLLUP component ALLOWS the USERS to group the records on certain field values.
• It is a multi stage function and contains
• Initialize 2. Rollup 3. Finalize functions which are mandatory
• To counts of a particular group Rollup needs a temporary variable
• The initialize function is invoked first for each group
• Rollup is called for each of the records in the group.
• The FINALLY function calls only once at the end of last rollup call.

• A limit is an integer parameter which represents a number of reject EVENTS
• Ramp parameter contain a real number representing a rate of reject events of CERTAIN processed records
• The formula is - No. of bad records allowed = limit + no. of records x ramp
• A ramp is a percentage VALUE from 0 to 1.
• These two provides the threshold value of bad records.

Look-up

• A lookup file represents a set of serial files / FLAT files
• A lookup is a specific DATA set that is keyed.
• The key is used for mapping values based on the data available in a particular file
• The data set can be static or DYNAMIC. 
• Hash-joins can be replaced by reformatting and any of the input in lookup to join should contain less number of records with a slim LENGTH of records
• Abinitio has certain functions for retrieval of values using the key for the lookup.

Look-up

• LOCAL lookup FILE has records which can be placed in main memory
• They use transform FUNCTION for retrieving records much faster than retrieving from the DISK.

To run a graph infinitely:

• The .ksh graph FILE should be CALLED by the end SCRIPT in the graph.
• If the graph name is abc.mp then the graph should call the abc.ksh file.

To run a graph infinitely:

The following is the process to add default RULES in TRANSFORMER

• Double CLICK on the transform PARAMETER in the parameter tab page in component properties
• Click on EDIT menu in Transform editor
• Select Add Default Rules from the dropdown list box.
• It shows Match Names and Wildcard options. Select either of them.

The following is the process to add default rules in transformer

• Abinitio supports SERIAL and parallel LAYOUTS.
• A graph layout supports both serial and parallel layouts at a time.
• The parallel layout depends on the DEGREE of the data parallelism
• A multi-file system is a 4-way parallel system
• A component in a graph system can RUN 4-way parallel system.

• Aggregation and Rollup, both are USED to summarize the data.
• Rollup is much better and convenient to use.
• Rollup can PERFORM some additional functionality, like input filtering and output filtering of records.
• AGGREGATE does not display the INTERMEDIATE results in main memory, where as Rollup can.
• Analyzing a particular SUMMARIZATION is much simpler compared to Aggregations.

The syntax for m_dump in Abinitio is used to view the DATA in multifile from unix prompt. The command for m_dump includes

• m_dump a.dml a.dat: This command will print the data as it MANIFESTED from GDE when we view data in formatted text
• m_dump a.dml a.dat>b.dat: The output is re-directed in b.dat and will ACT as a serial file.b.dat that can be referred when it is REQUIRED.

The syntax for m_dump in Abinitio is used to view the data in multifile from unix prompt. The command for m_dump includes

Roll-up component ENABLES the users to group the records on CERTAIN FIELD VALUES. It is a multiple stage function and consists initialize 2 and Rollup 3.

Roll-up component enables the users to group the records on certain field values. It is a multiple stage function and consists initialize 2 and Rollup 3.

Air command used in Abinitio includes

• air object Is<EME path for the object-/Projects/edf/..&GT; : It is used to see the listings of OBJECTS in a directory INSIDE the project
• air object RM<EME path for the object-/Projects/edf/..> : It is used to remove an object from the repository
• air object versions-verbose<EME path for the object-/Projects/edf/..> : It gives the version history of the object.

Other air command for Abinitio include air object cat, air object modify, air lock show user, etc.

Air command used in Abinitio includes

Other air command for Abinitio include air object cat, air object modify, air lock show user, etc.

De-partition is done in ORDER to read data from multiple flow or operations and are used to re-join data RECORDS from different flows. There are several de-partition components AVAILABLE which INCLUDES Gather, Merge, Interleave, and Concatenation.

De-partition is done in order to read data from multiple flow or operations and are used to re-join data records from different flows. There are several de-partition components available which includes Gather, Merge, Interleave, and Concatenation.

A SANDBOX is REFERRED for the COLLECTION of GRAPHS and related files that are saved in a single directory tree and behaves as a group for the purposes of NAVIGATION, version control, and migration.

A SANDBOX is referred for the collection of graphs and related files that are saved in a single directory tree and behaves as a group for the purposes of navigation, version control, and migration.

In Abinitio, partition is the process of dividing data sets into multiple sets for further processing. DIFFERENT types of partition COMPONENT includes

• Partition by Round-Robin: Distributing data evenly, in block size chunks, across the output partitions
• Partition by Range: You can divide data evenly among nodes, BASED on a set of PARTITIONING ranges and key
• Partition by Percentage: Distribution data, so the output is proportional to fractions of 100
• Partition by Load balance: Dynamic load balancing
• Partition by Expression: Data dividing according to a DML expression
• Partition by Key: Data grouping by a key

In Abinitio, partition is the process of dividing data sets into multiple sets for further processing. Different types of partition component includes

• Dedup component: It is used to REMOVE duplicate records
• REPLICATE component: It combines the DATA records from the inputs into one flow and writes a copy of that flow to each of its output ports

The Sort Component in Abinitio re-orders the data. It comprises of TWO PARAMETERS “Key” and “Max-core”.

• Key: It is ONE of the parameters for sort component which DETERMINES the COLLATION order
• Max-core: This parameter controls how often the sort component dumps data from memory to disk

The Sort Component in Abinitio re-orders the data. It comprises of two parameters “Key” and “Max-core”.

Different types of parallelism USED in ABINITIO includes

• Component parallelism: A graph with MULTIPLE processes executing simultaneously on SEPARATE DATA uses parallelism
• Data parallelism: A graph that works with data divided into segments and operates on each segments respectively, uses data parallelism.
• Pipeline parallelism: A graph that deals with multiple components executing simultaneously on the same data uses pipeline parallelism. Each component in the pipeline read continuously from the upstream components, processes data and writes to downstream components. Both components can operate in parallel.

Different types of parallelism used in Abinitio includes

LOOKUP file defines one or more serial file (Flat Files); it is a PHYSICAL file where the DATA for the Look-up is STORED. While Look-up is the component of ABINITIO graph, where we can save data and retrieve it by using a key parameter.

Lookup file defines one or more serial file (Flat Files); it is a physical file where the data for the Look-up is stored. While Look-up is the component of abinitio graph, where we can save data and retrieve it by using a key parameter.

To execute graph infinitely, the graph END script should CALL the .KSH FILE of the graph. Therefore, if the graph name is abc.mp then in the end script of the graph it should call to abc.ksh. This will run the graph for infinitely.

To execute graph infinitely, the graph end script should call the .ksh file of the graph. Therefore, if the graph name is abc.mp then in the end script of the graph it should call to abc.ksh. This will run the graph for infinitely.

The .dbc EXTENSION PROVIDES the GDE with the information to connect with the database are 

• Name and version number of the data-base to which you want to connect
• Name of the COMPUTER on which the data-base instance or server to which you want to connect runs, or on which the database REMOTE access software is installed
• Name of the server, database instance or provider to which you want to link

The .dbc extension provides the GDE with the information to connect with the database are 

The file extensions used in Abinitio are 

• .mp: It stores Ab initio graph or graph component
• .mpc: CUSTOM component or program
• .mdc: DATASET or custom data-set component
• .dml: Data manipulation language file or record TYPE definition
• .xfr: Transform FUNCTION file
• .DAT: Data file (multifile or serial file)

The file extensions used in Abinitio are 

To connect with Ab initio Server, there are several WAYS like

• SET AB_AIR_ROOT
• Login to EME WEB interface- http://serverhost:[serverport]/abinitio
• Through GDE, you can connect to EME data-store
• Through air-command

To connect with Ab initio Server, there are several ways like

Abinition is LOGICALLY divided into two segments

• DATA Integration PORTION
• User Interface ( Access to the meta-data information)

Abinition is logically divided into two segments

In Ab INITIO, dependency analysis is a PROCESS through which the EME EXAMINES a project entirely and traces how DATA is transferred and transformed- from component-to-component, field-by-field, WITHIN and between graphs.

In Ab initio, dependency analysis is a process through which the EME examines a project entirely and traces how data is transferred and transformed- from component-to-component, field-by-field, within and between graphs.

The Abinitio co-operating system PROVIDE features like

• Manage and RUN Abinitio graph and control the ETL processes
• Provide Ab INITIO extensions to the operating system
• ETL processes monitoring and DEBUGGING
• Meta-data management and interaction with the EME

The Abinitio co-operating system provide features like

Architecture of Abinitio includes

• GDE (GRAPHICAL DEVELOPMENT ENVIRONMENT)
• Co-operating System
• Enterprise meta-environment (EME)
• Conduct-IT

Architecture of Abinitio includes

“Abinitio” is a latin WORD meaning “from the BEGINNING.” Abinitio is a TOOL used to extract, transform and load data. It is ALSO used for data analysis, data manipulation, batch processing, and graphical user INTERFACE based parallel processing.

“Abinitio” is a latin word meaning “from the beginning.” Abinitio is a tool used to extract, transform and load data. It is also used for data analysis, data manipulation, batch processing, and graphical user interface based parallel processing.