Skip to content

Connections / Mappings (Data Ingestion)

In Qualiz, a connection defines the link between a data source and a destination within a workflow. Mappings specify how data is transferred, transformed, and aligned from the source to the destination, ensuring accurate and reliable ingestion.

Key Concepts

Connection

  • A connection represents the configuration required to move data from a source to a destination.
  • Each connection includes:
  • Source Configuration: Details such as database credentials, API endpoints, or file locations.
  • Destination Configuration: Target database, file storage, or data warehouse information.
  • Security and Access: Credentials, encryption, and access control scoped per project.

Mapping

  • Mappings define how data fields from the source are translated and inserted into the destination.
  • Includes:
  • Field Mapping: Specifies which source columns map to which destination columns.
  • Transformation Rules: Optional rules to clean, enrich, or convert data during ingestion.
  • Validation Rules: Optional checks to ensure data quality before it is written to the destination.

Data Ingestion Flow

  1. Source Selection: Choose the source system or file from which data will be extracted.
  2. Destination Selection: Choose where the processed data will be delivered.
  3. Mapping Configuration: Define field-level mappings, transformations, and validations.
  4. Execution: Workflow tasks execute the ingestion according to the defined connection and mapping rules.
  5. Monitoring: Track ingestion status, success/failure, and logs centrally.

Extensibility

  • New sources, destinations, and mapping rules can be added without impacting existing connections.
  • Supports evolving business needs and integration with new systems.

Centralized Management

  • All connections and mappings are managed centrally for visibility, governance, and reuse across workflows.
  • Provides an overview of active, inactive, and historical configurations for auditing and compliance.

Summary

Connections and mappings in Qualiz define how data flows from sources to destinations. By providing flexible field mapping, transformation, validation, and centralized management, they ensure secure, reliable, and accurate data ingestion across workflows and projects.