Lumenore Data Magnet (ETL tool)

Estimated reading: 5 minutes 2081 views

The Data Magnet is a user interface (UI) based module that facilitates data integration by enabling users to extract data from various source systems, transform it, and deliver it in a presentable way in the data analytics platform. The available features of the Data Magnet that support real-time data integration are as follows:

Data Extraction: This involves extracting data from homogeneous or heterogeneous sources.
Data Transformation: This stage involves cleaning and transforming data into a proper storage format/structure for querying and analysis.
Data Loading: This stage describes data insertion into the final target database, such as an operational data store, a data mart, a data lake, or a data warehouse.

How to Access Lumenore Data Magnet?

Step 1: Access Lumenore by logging in with your credentials and then navigating to the Data Magnet tab.

Step 2: Upon opening Data Magnet, the user will see:

Total Projects:
Shows the total number of projects (in this case, 5).
Includes a button to “Create new Project”.
Introductory Video Card:
A call to action to watch a video to understand how to use Data Magnet.
Search & Filter Section
- Search bar: Allows users to search for projects by name.
- Sort by dropdown: Allows sorting of projects (likely by name, created on, etc.).
- Tabs (Recents / All): Switch between recently accessed projects and all projects.
Project List Table
This section displays a list of individual projects with the following columns:
- Project name: Name of the project (e.g., “PMG”, “Test Demo”)
- Created on: Timestamp showing when the project was created
- Users: Name of the users of the project
- Actions: A button/link to “Open” the project for data transformation tasks
Additional Notes
- The interface is clean, modern, and user-friendly, typical of collaborative SaaS platforms.
- This page is the entry point into deeper data ETL and preparation flows.

Let’s begin with managing the project.

Project management

Step 3: Here, you can view all the projects created in the organization.

Columns and Their Meaning

Name: The project name (e.g., PMG, prod_demo, test)
Connectors Permission: Displays data source connectors (e.g., Google Drive, Sheets, Looker Studio). The number (e.g., +28) shows how many connectors are enabled.
Total Users: Users who have access to the project.
Admin: The project admin/owner.
Status: This toggle indicates whether a project is Active (blue) or Inactive (red). Inactive projects indicate that they have been created but are not currently used.
Action: Options to Edit or Delete the project (accessible via the three-dot menu)

Notable Functionalities

Search Bar: Allows users to search for a specific project by name.
Create New Project: A blue button at the top-right lets users start a new project.
Sorting: All columns support sorting (indicated by the ↕️ icons).
Pagination: Displays the number of projects visible per page. For example, if there are 16 projects in total and 8 are shown on the current page, navigation controls will appear at the bottom to browse the remaining projects.

User Controls

Admins can activate or deactivate projects using the toggle.
Connectors are visually represented, and likely clickable or hoverable for more details.
Admins can manage access and clean up test/demo projects.

Step 4: Click on “Create New Project.”

Step 5: Now, provide the necessary information to create a new project, including:

Project name (required): Field where the user provides the name for the new project.
Add connectors: This option allows users to select data source connectors (e.g., Google Sheets, SQL, etc.) to link to the project. A “Clear all” button removes selected connectors.
Add users: Users can be invited to the project by selecting from a user list. A role dropdown is provided alongside.
- User Role Selection: The dropdown next to the user selection field offers three roles:

Developer: Can build and manage workflows.
Viewer: Has read-only access to project content.
Admin: Has complete control, including editing project settings and managing users.

Step 6: After entering all the details, click the “Create” button, which confirms and creates the project with the specified configurations.

Step 7: Now, you can view the newly created project.

Step 8: Access the options by clicking the three dots to edit or delete the project.

User Management

Project admins can manage user access, roles, and activation status. It shows:

Project name: Dropdown to choose the project. For example, currently it’s PMG.
Created date: Shows the creation date of the selected project. For example, 23/02/2023.
Assign user button: Used to add new users to the project and assign roles.

Step 9: Once the project is selected, the user will find the following information:

Name: Full name of the user.
Email id: The email address is associated with each user.
Role: Assigned user role: either Admin or Guest. Admins have complete access.
Status: Toggle switch to activate or deactivate a user.
Action: Likely a context menu (three dots) for editing user roles.

Scheduler Management

Step 10: Here, you can view the overall count of schedules, including the number of active and paused schedules. Additionally, you can observe the count of schedules that have encountered failures.

Step 11: After selecting the project, navigate to the created schedules. Click on the specific schedule for which you want to access detailed information.

Step 12: You can also perform various functions on the schedule, including editing, pausing, deleting, and refreshing, either by selecting options at the top or clicking the three dots corresponding to the schedule.