Skip to main content

Pipelines and Workflows

A solid understanding of pipelines and workflows is essential for leveraging the full potential of our data processing platform. This guide explains these core concepts and their interrelationship.

Pipeline​

A pipeline refers to a linear or sequential flow of data through various stages. Each node in the pipeline leads directly to the next, making it easy to track the transformation of data from beginning to end.

Workflow​

A workflow represents a more comprehensive concept that involves complex processes. It can incorporate multiple pipelines, branching, merging, and parallel operations, providing greater flexibility and control over data flow.

info

Relationship between Pipelines and Workflows
💡 A pipeline is a subset of a workflow. A workflow can contain multiple interconnected pipelines, enabling more sophisticated data processing.

Visual Representation​

To visualize these concepts more clearly, consider the following examples:

pipeline and workflow sample

  1. Workflow with Multiple Pipelines: The upper section of the diagram illustrates a workflow containing two pipelines. These pipelines branch after the data cleanup stage.

  2. Simple Pipeline (also a Workflow): The lower section depicts a straightforward pipeline that follows a linear sequence, which is also considered a basic workflow.

caution

When designing complex workflows with multiple pipelines, ensure that data flows correctly between different stages and that all branches are properly configured to prevent errors.

By mastering the concepts of pipelines and workflows, you'll be equipped to design efficient and scalable data processing solutions using our platform.