Generic bulk loader

The Generic Bulk Loader is a powerful function for loading large amounts of data into different types of databases. It’s called “generic” because it’s not limited to one specific system—depending on the connection you set, you can use it with many databases, like MySQL, PostgreSQL, Snowflake, and more.

Unlike Vertica Bulk Loader, which only work with Vertica, this one is universal, making it great when you’re working with multiple platforms or want more flexibility in your data pipelines.

It’s useful when you need fast, reliable data loading without being locked into one database type.

Steps to use Generic Bulk Loader:

Step 1: Drag and drop the “Generic Bulk Loader” component and connect it to the source.

Step 2: Click “Select connection” and select the connection where you want to load the data from the drop-down menu.

Step 3: Choose the destination table for loading data. Click “Fetch tables” to see available options in the selected connection.

Step 4: Specify a “Rejected tables” where rejected or failed rows will be saved during the load process. This helps with error tracking.

Step 5: When the “Truncate” is enabled, it clears (deletes all records from) the destination table before loading the new data. This ensures that only fresh data exists post-load

Step 6: “Specify Database Fields” allows manual mapping of fields between the stream and the database.

Step 7: Click “Save” and “Run”

Note: Not all connections are permissible for target components. Each target component is compatible only with specific allowed connections.

For example:

• Choose the “etl_connection” from the available options.
• Clicked “Fetch Tables” to retrieve the tables and select the desired table.
• The name of the rejected table is also created.
• Enabled “Truncate.””
• Enabled “Specify database fields” and mapped them.