Data Integration for LMI

Data Integration for LMI platform

Previous to applying data analysis and representing it through visualizations, data must be obtained (extracted), integrated, cleansed and stored in an appropriate data structure and during these initial phase of development, many challenges encountered include:

• Read multiples types of excel files, webs service, etc. and brings all of these data in a single repository into one homogeneous format,
• Make easy maintenance solution and traceability relation,
• Ensure good time/cost perspective of the same resource,
• Process needs to be run repeatedly (monthly time),
• Apply complex rules and transformation on data.

Majority of these needs are meet with ETL tools (e.g. Extraction, Transform and Load tools).

For our LMI platform, EmployID Consultancy Network offers a robust and comprehensive ETL process orchestrated by specific software for data integration (Talend for Data Integration). The overall process is represented by the following figure:

Figure 1: LMI Data Integration System

Talend is an open source platform for ETL and Data elaboration. Talend OS is a visual suite (based on Java & Perl) to develop ETL processes with the advantage of having built in integration with Analytics and Machine Learning/IIA software such R-Project and Python.

Given the need for both structured and unstructured data (for possible future use), the ideal ETL tools is to access and load data to and from data sources including RSS (Rich Site Summary) feeds, API calls, RDBMS (Relational Database Management System) and unstructured data stores such as Hadoop.

The process consists of the following stages:

• Talend Data Integration Module extract data from different Datasources.
• Data is processed, combined, etc.
• Data is loaded into Data Warehouse (full data) and Data marts (data segment).
• Once ETL is finished, the data is available for LMI Tool. Talend publish different interfaces to consume data (a JavaScript code running in the browser to gather automatically all data from SORS web service).

For more details, consult our Technical Report “Data Integration for LMI platform in ZRSZ”

Sustainability for LMI platform

In ensuring the long-term sustainability of the LMI Platform, the Information Technology Assistant will provide a type of Help Desk support to all persons involved in the IT Department via contact email or phone according to service agreement. The Help Desk Service allows organization manage incidences and requests by implementing and automating ticketing processes so our offering service comply with the best practice