Component Breakdown

1. Qualiz UI

A React-based interface that enables users to: - Design pipelines via drag-and-drop - Configure data sources, destinations, and transformation steps - Define data cleansing and deduplication rules - View dashboards and audit logs - Manage users, roles, projects, and settings

2. Backend Services

Spring Boot services that: - Provide REST APIs for all UI operations - Generate and manage pipeline workflow definitions dynamically - Interact with metadata store and task execution layer - Handle scheduling, triggering, and rule resolution

3. Metadata Database

Stores: - Pipeline definitions - Rule configurations - Execution history and audit logs - User/project settings

4. Execution Engine

Executes task types independently, including: - Python scripts - SQL queries - Data cleansing/standardization - Deduplication logic - Sub-pipeline invocation

Each task runs in a sandboxed container with resource isolation.

5. Connector Engine

Handles ingestion logic, including: - Source configuration (e.g., JDBC, S3, APIs) - Destination setup (e.g., databases, file stores) - Schema mapping and field-level control

All connections are configured through the Qualiz UI.

6. Distributed Compute (Optional)

For resource-heavy processing, Qualiz can delegate task execution to a cluster: - Built on Apache Beam with Flink as the runner - Ideal for parallel or streaming workloads - Supports complex aggregations, large joins, etc.

7. Monitoring & Logs

Real-time and historical observability: - Job-level and task-level audit logs - Alerts and email notifications (configurable) - Optional integration with log aggregators like ELK or Grafana