Skip to content

Sources

In Qualiz, a source represents the origin of data that feeds into workflows. Sources define where and how data is ingested, ensuring that workflows receive accurate, structured, and reliable input for processing and quality operations.

Key Concepts

Source Types

  • Qualiz supports multiple types of data sources to meet diverse business needs.
  • Common source types include:
  • Databases: Relational databases like PostgreSQL, MySQL, Snowflake, BigQuery, and others.
  • APIs: REST or GraphQL endpoints providing structured data.
  • Files: CSV, JSON, Parquet, or other structured files stored locally or in cloud storage.
  • Streaming Sources (coming soon): Real-time data feeds from message brokers or streaming platforms.

Connection Configuration

  • Sources require connection details such as credentials, host, port, and database name.
  • Configurations are stored securely within Qualiz and scoped per project.
  • Supports dynamic configuration to connect to multiple environments or tenants.

Schema and Metadata

  • Sources expose schema information such as tables, columns, and data types.
  • This metadata is used to map data accurately to workflows and tasks.
  • Enables validation, transformation, and lineage tracking within the workflow.

Source Isolation

  • Each source is associated with a specific project, ensuring that connections and metadata are isolated.
  • Prevents cross-project data leakage and supports multi-tenant architectures.

Extensibility

  • New source types can be added without affecting existing workflows.
  • Supports growth and integration with emerging data platforms or technologies.

Centralized Management

  • All sources are registered and managed centrally for governance and visibility.
  • Provides an overview of active, inactive, and historical source configurations.

Summary

Sources in Qualiz define the origin of data for workflows, enabling structured, reliable, and isolated ingestion. By supporting multiple source types, secure connection management, and centralized visibility, sources provide a strong foundation for scalable and maintainable data pipelines across projects.