Table input

Allows data to be extracted from structured, semi-structured, and unstructured data sources for further processing, transformation, and loading into target systems. It provides flexibility and customization options to meet specific data extraction requirements.

The ‘Add Filter’ feature allows users to apply filters on the data at the source, enabling them to extract only the relevant data based on specified conditions. This feature will consist of three input fields: select column, operator, and input field value. Users also can add multiple conditions or group conditions for more advanced filtering.

The CDC feature in table input enhances data integration by capturing and replicating real-time changes from source to target systems. This addition aims to enhance data accuracy, reduce processing load and latency, and improve the overall value proposition of our ETL tool.

Key benefits include near real-time data capture for the latest information, increased efficiency by processing only changed data, quicker insights for informed decision-making, and scalability to accommodate growing business needs.

To use Table input:

Step 1: Open the table input source and select the connection from the options.

If you are selecting Table mode.

Click on “Get Tables” and choose the desired table.

Note: Enable a filter to use the CDC feature; otherwise, choose “Save.”

Step 2: After enabling the filter option, specify the condition and validate it.

To set multiple conditions, select “+ Condition,” pick the group condition value, and validate it. If the user has added extra conditions, they can delete them.

Step 3: Enable the CDC to provide the conditions for the CDC. If the user chooses “Maximum,” the attribute’s maximum value will be displayed, and the value tab will be disabled.

If the user selects “Customized,” the value tab will be enabled, allowing the user to enter a custom value.

After applying the necessary condition, click “Apply” and “Save.”

Preview

If users are selecting Query mode.

Write the query in the query box and validate it. Click “Save.”Note: The CDC aims to identify and capture the changes over time in the data in a database. The format for the CDC query is ‘$${varName}’ for the column name.