PIM connector - Key concept

The censhare PIM connector is a configurable interface that provides export and import functionality for product data. The connector allows multiple transformations so that users can export data into different formats and/or data structures. Likewise, the import can be configured so that users can import differently structured data into censhare.

Target groups

Solution developers

Purpose

The PIM connector is a configurable interface that provides export and import functionality for product data. The connector allows multiple transformations so that users can export data into different formats and/or data structures. Likewise, the import can be configured so that users can import differently structured data into censhare.

Context

The PIM connector requires the installation of a Pentaho server and the configuration of the export and import actions and pipelines in the censhare Admin Client.

Prerequisites

Pentaho PDI CE is required for the installation. Depending on the data you want to import/export, your censhare system must be configured accordingly. For example, to exchange product data, you need a censhare PIM application.

Introduction

One of the main purposes of a PIM system is to store all kinds of product data like technical specs, marketing-related, etc. in one system. censhare PIM provides a standard data model and asset structures. However, in order to consolidate data from different sources and allocate them to customers, suppliers or vendors, different source and target data models are often required.

The schema transformations and data mappings from censhare to the target format and vice versa are carried out by an ETL (Extract, Transform, Load) processor. 

The PIM connector transforms, splits and formats data in a bi-directional interface. It creates an output format and data model that can be exchanged with other systems. Likewise, imported data from other systems are transformed and stored according to the censhare data model. The PIM connector can be configured for both directions (export and import) independently. You can create custom transformations for export and import.

This process runs in a third-party software. Pentaho PDI CE is an open-source ETL solution. Therefore, it is delivered with the respective server action, export and import transformations.

Currently, the PIM connector supports the export/import of Product data and Salesforce data. The respective data schema and transformations are part of the standard configuration. censhare also provides preconfigured schemas for the export into and import from Microsoft Excel files.

Process diagram

The following diagram shows the export process of product data to Excel using Pentaho. The processor pipeline can be different depending on the exchange format you want to generate and the used ETL solution. See also the following Processor pipeline section.

Processor pipeline

As shown in the diagram above, the PIM connector processes the data to be exported or imported in a processor pipeline. The pipeline consists of several process stages and is configurable in the censhare Admin Client. You can add as many stages as needed to process the data. The processor pipeline creates the exchange format that is processed in the connected ETL solution (Pentaho). In the export pipeline, censhare first generates a JSON representation of the collected data records. These can either be processed directly in Java (not represented in the graphic) or rendered as XML which is then transformed using XSLT.

The process pipeline can be configured according to the direction (export/import) and the format expected by the ETL tool. If you do not want to transform the data, you can as well export the raw XML or JSON data from censhare. For these cases, the processor pipeline has to be configured accordingly. The setup of the pipeline depends on the data model and format the connected ETL solution expects from censhare (export pipeline) or, respectively, delivers to censhare (import pipeline).

Example: Export pipeline for Pentaho server

• Collect records - This stage is necessary if you want to export data from censhare. censhare creates a JSON representation of the data records.

• JSON to XML - In this stage, the JSON representation of the data records is transformed into raw XML.

• XSLT - In this stage, you can use any XSLT transformation. It creates the exchange format that can be parsed from the connected ETL tool. The Pentaho export/import transformations are part of the censhare standard configuration. If you want to connect to other ETL software, you have to create a custom transformation according to the specifications of the third-party software.

Example: Import pipeline from Pentaho server

• XSLT - This is the only stage necessary in the import pipeline. It creates the raw XML data which can be imported into censhare. The transformation is part of the censhare standard configuration.

Components and resources

The PIM connector consists of the two censhare modules "Schema-based export" and "Schema-based import". For each, you can add custom configurations. For the setup, a Transformation asset with the resource usage "Connector transformation" is required. In the modules, a schema, a processor pipeline and a transformation must be configured. The schema describes the data structure in a JSON file.

In the export direction, the processor pipeline collects the data from the selected assets (products) according to the schema and creates an XML representation. censhare then passes this XML file to the Pentaho server, where the data are transformed and output. For this step, users have to select an Export transformation in censhare when calling the server action. A dialog provides a list of available transformations. For more information, see Create custom schemas and transformations for the PIM connector. The Pentaho export asset contains the transformation and job files for the Pentaho server (Kettle transformation files and Kettle job files). Optionally, you can provide test data in order to test the output without production data from your system.

In the import direction, censhare passes the selected Pentaho import format and the import file to the Pentaho server. There, the data are transformed to an XML representation that is read by censhare. censhare writes the data into the database. Optionally, you can provide test data in order to test the import without production data.

For each transformation, you can configure multiple export respective import formats. The Dialog assets for the export and import server actions create a dialog where users can select the format they want to export or import.

The Server actions (censhare Web), dialogs and transformations are stored in Module assets. The censhare standard configuration contains default configurations to connect with Pentaho and export/import to and from Microsoft Excel. Based on these, you can create custom configurations for your own purposes.

The following table shows the Module assets and available default configurations with their meaning. For more information, see Create custom schemas and transformations for the PIM connector.