Handling Flatfiles using SAP Data Services

Flat files and XML-files

Flat files, Excel and XML-files and their respective File Formats can be
created by the DS Developer. Folders should be maintained in Dataservices
Server to maintain Input and Output files.These folders can be created by DS
Server Admin upon requirement and request from the project manager/lead.
The input folder should be given read only, copy, move, delete permissions
to the Dataservices server user because the input data files shouldnt be
modified. The ouput folder should be given owner permissions to the
Dataservices server user. The input and output files should be stored under
respective Solution Area Specific Folders. Standard way to access these files
is to assigning folder path to the substitution parameter/global varialble and
using the variable in the File format editor. Always there could be multiple
files existing in the input folder to read, so as to read them all at once the file
name can be mentioned like *.* or *.formatname or
Domainname_*(here * means datetime of the input file). The name of the
file can be maintained ina global variable as well if the job has logic to
retrieve file names for each load. It is standard way to maintain Substitution
parameters/Global variables in all environments to prevent file access errors.
Input Files
Solution
Area

Substitution
Parameter

Path for Dev

MDM

$$MDMIn

/F65/sapdata1/ds/MDM/data-in/

Finance

$$FinanceIn

/F65/sapdata1/ds/Finance/data-in/

Common

$$CommonIn

/F65/sapdata1/ds/Common/data-in/

Solution
Area

Substitution
Parameter

Path for Dev

MDM

$$MDMOut

/F65/sapdata1/ds/ MDM/data-out/

Finance

$$FinanceOut

/F65/sapdata1/ds/ Finance/data-out/

Common

$$CommonOut

/F65/sapdata1/ds/ Common/data-out/

Output Files

Folder requirement should be managed by DS Manager/Lead.

This is a temporary storage area and files stored in IN and OUT folders should
be either replaced with subsequent runs or to be deleted in DS jobs in
subsequent runs.

1) Loading data from a flatfile to a Database table:Below are the steps to create File format, DB connection using Datastore and
loading data.
I) Creating DataStore for SQL database.
1. Logon to the SAP Business Object Data Designer.
2. In Local Object Library click on Datastore Tab.

3. In Local Object Library click on Datastore Tab.

Window for creating new datastore will open as shown below.

Give all the details as shown below.
Here database name "RAHUL_MORE" that i am using is already created
before creating the datastore.

4. Click "OK" & the Datastore will be created & can be seen in the Local
Object Library as shown below.

Now we will create File Format to process the source file.

II) Creating File Format
1. Click on the "Formats" tab in the Local Object Library in Data Service
Designer.

2. Select "Flat Files", right click & click on "New". Flat File Editor will open
as shown below.

We are using following file as the source.

3. Give all the details in the File Format Editor as follows.

When you select the File Name it will ask to overwrite the schema. I have
selected "Yes" option in this option.
As a result of this you can see the altered schema on top of the RHS.
Here the root directory path and File name can be maintained using
Substitution parameter/Global variable.

4. After giving all the details press "Save & Close". File Format will be created
& can be seen in the Local Object Library.

Now after creating the Datastore & File Format we will design a job
in the Data Service Designer.
III) Creating a job in Data Service Designer.
1.

Create a project.
Go to the "Project Area" in the Data Service Designer & click the icon

for creating the project from menu bar.

Window for creating the new project will open. Give project name &

click "Create"

New project will be created & can be seen in the Project Area.
2. Create a Batch Job.

Right click on the project & click on "New Batch job".

Give appropriate name to the job.

3. Add a Dataflow into the job.

Select the job, drag dataflow from palette into it & name it.

4.Build the scenario in the dataflow.

Double click on the dataflow.

Drag a file format created by us earlier into the dataflow.

Drag a Query into the dataflow & connect it to the file format.

5. Mapping
Open the query & do mapping.

Select all the fields on LHS, right click & select "Map to Output".

Now you can see the fields mapped to RHS as shown below.

To make "ID" as the key field , Right click on the "ID" field & mark it as
a Primary Key.

6.Insert Target Table.

We can either import the already created table or we can use template
table which afterwards will actually be created in the database.
We are using template table here.

Drag a template table from palette into the dataflow.

Give name to the table & select appropriate datastore.

Click on "OK".
Template table can be seen in the dataflow.
Connect Template table to Query.

7.Save & Validate the Job.

Click "Validate All" icon from the tool bar.

Following message is displayed if no error found in the job.

"Validate : No Errors Found (BODI 1220017)"

8.Execute the Job.

Select the job, right click & press "Execute".

9.Check the output in the table.

After execution table will be created in the database & you can check the
output.
For checking the output you can click on the Magnifying Glass icon of the
table in the DS Designer.

2) Loading data from a XML file to a Database table:Generally XML data is maintained in 2 different files.
XSD/DTD
XML
XML Data file
DTD or XSD(XML Schema definition) file consists of structure or column
names and datatypes.
XML file consists of data for the structure in XSD.
These files should be combined to extract data from .xml file according to
the structure in .XSD.
Steps to create Format and extracting XML data as follows.
Go to local object library in data services designer. Click Formats -> XML
Schemas

Right click on XML schemas and click on new

A new window pops up. Give the desired format name under the tab Format
name

We have to browse the xml schema file. Files in .XSD format have to be used. For this we can use any
tools available to convert a .xml file into .xsd file.

The content of xml file used in this example is

After converting to .XSD file the contents becomes as given below

Import the xml schema.

If the file imported is not proper (format issue),then an error will be thrown
while importing.
For E.g.:

Rectify the error and import the proper xml schema.

After importing successfully give the corresponding Root element name from
the drop down.

Click Ok.
The imported schema will be found under XML Schemas in local object
library.

Or

Create a File Format for DTD to use as Source

Create a project

Create a Batch Job.

Take Work Flow and Data Flow.

Drag the DTD File Format into workspace and make XML File Source.

Double click on Source and Browse the XML file.

Take two XML_Pipeline Transformations.

Go to XML_Pipeline and Drag the required columns into Schema Out

Go to another XML_Pipeline and Drag the required columns into Schema Out.

Take Query Transformation and join to the two XML_Pipeline Transformations.

Go to Qyery Transformation and Drag the columns to schema out and write
condition in Where.

In Query Transformation make Nested Schema to Unnest by selecting the column

and Right clicking.

Take the Template Table and Join to the Query Transform.

Validate and Execute the Job.

After Executing the Job ,Check the Data in the Target.