CDC implementation using mapping variable:

Scnerio :- Say we need to implement Change Data Capture for CUSTOMER data load. We need to load any new
customer or changed customers data to a flat file
We have Update_TS as a column that changes for every new or updated customer
Create a mapping variable as shown in below image.
$$M_DATA_END_TIME as Date/Time

Step 2)
In source qualifier override give below filter condition

3) in expression t/f
SETMAXVARIABLE($M_DATA_END_TIME,UPDATE_TS)

This expression will make sure that, latest value from the the column UPDATE_TS is stored into the
repository after the successful completion of the session run.
Map remining columns to target and mapping is complete
Once you look into the session log file you can see the mapping variable value is retrieved from the
repository and used in the source SQL, just like shown in the image below.



Mapping variable is used in CDC logic
New Value of the mapping variable is stored in repository ,only after successful run of the session Since the last
success point - variable value, is stored in repository, we cannot go back and reprocess an already processed data
set Mapping variable takes the starting value from either
a) Parameter file
b) pre session variable assignment
c) value in repository
d) Initial value e)default value
We can override Mapping varible using parameter file parameter file is defined at workflw or session level as
below

The parameter file will have the name of the mapping variable and the overriding value for variable
eg
[s_Load_CUST_DIM]
$$M_LST_RUN_TS=01-01-2013 00:00:00
Note : Remove mapping variable value from the parameter file to use the value saved in the repository.
The parameter file will look like
[s_Load_CUST_DIM]
$$M_LST_RUN_TS=
Advantages :-
env when there is a need to override the repository persisted mapping
variable value.

releases

Another way is



Step 2:-







What are complex mapping you have created:-

Most complex logic we use is denormalization. We dont have any Denormalizer transformation inInformatica. So
we will have to use an aggregator followed by an expression. Apart from this, we usemost of the complex in
expression transformation involving lot of nested IIF and Decodestatements...another one is the union
transformation and joiner.

how to get the first 100 rows from the flat file into the target?

SOLN1:
task ----->(link) session (workflow manager)double click on link and type $$source sucsess rows(parameter in
session variables) = 100it should automatically stops session.





RECOVERY MECHINISM

Recovery allows user to restart a failed session, and complete it as if session had run wihout a pause

Recovery are of 2 types
1) Workflow recovery
2) Session recovery


Workflow Recovery:- Allows u to continue procesing workflow and wfl tasks frompoint of interruption
Workflow state of operation is maintained which includes state of tasks and wfl variable values

There are 3 types of recovery configuration
a.Workflow configuration for recovery :- Either configure wfl for recovery, or configure wfl to Suspend
on task error
Below is how u enable Automatica recovery for workflow, if wfl stops aborts or teminates



IF the HA option is unchecked , then we will have to manually tell power center to recover the
wfl

SUSPEND:- We can configure wfl to suspend on error.We can recover suspended wfl after fixing
the task error in case task has failed
If the workflow is not able to recover automatically from failure with in the maximum allowed
number of attempts, it goes to 'suspended' state.

We can set up the workflow to suspend on error as shown in below image





SESSION AND TASK CONFIGURATION FOR RECOVERY

Each session or task in a workflow has its own recovery strategy.
When the Integration Service recovers a workflow, it recovers tasks based on the recovery
strategy of each task or session specified.

Three different options are available.

Restart task :
Fail task and continue workflow
Resume from the last checkpoint




you can see the available command task recovery options as shown in below image.









When you run a session enabled for recovery, the Integration Service creates and stores session status information
in recoverytables and cache files. If the session fails and you recover the session, the Integration Service uses this
status information todetermine where the previous session failed and where to continue the session

Recovery Tables :-
When you enable recovery for a session that writes to a relational target, the Integration Service creates two
tables at the targetdatabase, PM_RECOVERY and PM_TGT_RUN_ID,PM_REC_STATE in
($PMRootDir/storage) and save the workflow run stats during the process
T hen the Integration Service runs a failed session in recovery mode, it uses the information in therecovery tables
to determine where the previous session failed, and then continues the session from that point
These tables are used to recover data..so the last checkpoint is where the last row failed to be loaded to
target table


PM_RECOVERY
This table records the target load information, It has
.Repository id
Workflow ID
Workflow run ID

Folder ID
Session instance ID
Target instance ID. The Integration Service creates one row per target instance
Partition ID. The Integration Service creates one row per partition

Row count. Stores the last committed number of rows
Check point.

PM_TGT_RUN_ID
PM_TGT_RUN_ID is for internal use. It stores the latest unique number the Integration Service uses to identify the
target instance.This table has a single field, LAST_TGT_RUN_ID.

The Integration Service creates cache files for both relational and non-relational targets. The session will fail if
either the recoverytables or the pmgmd cache files cannot be created due to insufficient privileges.
DO not change or edit the cache files

Recovery Types
Internally, recovery can be divided into two types:
..incremental recovery
..full load recovery.

In incremental load recovery, the Integration Service uses the ROW_COUNT information in the PM_RECOVERY
table to determine thenumber of rows committed to the target.
When the session is recovered, previously loaded rows will be skipped. Thus, incrementalrecovery ensures that
there are no duplicates loaded to the target.
The Integration Service performs incremental recovery for relational, normal load targets.

In full load recovery, data is loaded from the beginning during the recovery run. The Integration Service performs
full load recoveryfor relational bulk load, flat file (including regular, MQSeries, and external loader), XML, and SAP
BW targets. Recovery for nonrelationaltargets is done using the pmgmd file created at the $PMCacheDir cache
directory.

Recovery and Truncation
During the recovery session run, the Integration Service ignores the Truncate Target Table option for relational
normal load targets,since the load will continue from the last commit point.
For relational bulk load targets the Integration Service allows the TruncateTarget Table option in both normal and
recovery run.
At the end of a successful session run the Integration Service deletes all recovery information in the target
database and in$PMCacheDir.
The Integration Service always resets the recovery information when it runs the session normally, regardless
ofwhether the previous session run succeeded or failed.

For session to run in recovery mode

You should select a Recovery Strategy (under Edit Session >Properties > General Options)
The number of partitions is not changed between the normal and recovery run.
The recovery table created at the target database is not dropped or changed by the user.
The OS platform is not changed between normal and recovery run.



RECOVERING WORKFLOW FROM FAILURE

Recovering Automatically
If you have High Availability (HA) licence and the workflow is configured to recover automatically as described
above, Integration service automatically attempts to recover the workflow based on the recovery strategy set of
each session or task in the workflow. If the workflow is not able to recover automatically from failure with in the
maximum allowed number of attempts, it goes to 'suspended' state, which can be then manually recovered.


Recovering Manually