Skip to content

Tasks

In Qualiz, a task represents an individual unit of work within a workflow. Tasks are the building blocks of workflows, encapsulating specific operations such as data ingestion, transformation, validation, or cleansing. They define how data is processed step by step while maintaining consistency, reliability, and traceability.

Key Concepts

Workflow Association

  • Every task belongs to a specific workflow.
  • This association ensures that tasks are executed in the correct sequence and context within the workflow.
  • Supports logical grouping and dependency management between tasks.

Task Types and Supported Tasks

  • Tasks can represent a variety of operations, providing flexibility and modularity.
  • Supported tasks include:
  • Data Ingestion: Pull data from external systems like databases, APIs, or files using Airbyte (through Qualiz UI).
  • Run Python Script: Execute custom Python logic.
  • Run SQL Script: Run custom SQL queries using a secure backend.
  • Data Cleansing: Standardize or enrich data using cleansing rules.
  • Data Deduplication: Detect and remove duplicate records using algorithms.
  • Invoke Another Job: Trigger a different job as part of the current pipeline.
  • Run Python with Apache Beam (coming soon): Execute Python code on a distributed Beam cluster.
  • Email Notification (coming soon): Send results or alerts via email post execution.

Execution and Dependencies

  • Tasks can define dependencies on other tasks to ensure proper execution order.
  • Supports conditional execution, parallelization, and branching based on workflow requirements.
  • Ensures accurate and reliable processing even in complex workflows.

Isolation and Scope

  • Tasks inherit the workflow and project isolation settings.
  • Ensures that task data, configurations, and metadata are kept separate across workflows and projects for security and compliance.

Extensibility

  • New tasks can be added to a workflow without impacting other tasks.
  • Supports a scalable architecture where workflows can grow in complexity while maintaining clarity and manageability.

Centralized Monitoring

  • Tasks are tracked centrally for execution status, performance, and logging.
  • Provides visibility into workflow progress and simplifies debugging and auditing.

Summary

Tasks in Qualiz are the core operational units within workflows. By clearly defining execution, dependencies, type-specific behavior, and supported task types, tasks ensure that data processing is modular, scalable, and traceable. They provide the foundation for building robust, maintainable, and secure workflows across projects and domains.