Lumenore data loader

The Lumenore Data Loader is a function for sending processed data to a specific table in your database. After your data has been cleaned or transformed in your pipeline, this component loads it into the right place.

It also gives you extra features like:

• Automatically handling table structure (schema) if needed
• Saving rows that fail to load into a separate “rejected” table, so you don’t lose anything
• Option to clear the table before loading new data, so it’s always fresh

This makes it easy to keep your database updated with your pipeline’s latest, high-quality data.

Steps to use Lumenore Data Loader:

Step 1: Open the “Lumenore Data Loader” and connect to the graph.

Step 2: Click “Select connection” and select the connection where you want to load the data from the drop-down menu.

Step 3: Choose the destination table for loading data. Click “Fetch tables” to see available options in the selected connection.

Step 4: Specify a “Rejected tables” where rejected or failed rows will be saved during the load process. This helps with error tracking.

Step 5: When “Create Table if Not “ is enabled, the system will automatically create the target table if it doesn’t already exist in the selected database.

Step 6: When the “Truncate” is enabled, it clears (deletes all records from) the destination table before loading the new data. This ensures that only fresh data exists post-load

Step 7: “Specify Database Fields” allows manual mapping of fields between the stream and the database.

Step 8: Click “Save” and “Run”

For example:

• Selected the connection named “etl_connection.”
• Now, pick the desired table.
• The name of the rejected table is also created.
• Enabled “Create table if not exists.”
• Enabled “”
• Specified the data fields and mapped
• Save and run.