Jepsen Analyzes Orkes Conductor 5.3.2 Workflow Orchestrator

Workflow Orchestration Overview

A workflow orchestrator is a software system designed to execute computations composed of smaller tasks. These tasks are connected in a workflow, which dictates the routing of outputs to inputs, task scheduling, and overall execution flow. The orchestrator manages workflows, their specific executions, task processing, state persistence, and aims to drive each execution to completion.

Orkes Conductor Background

Conductor, originally developed and open-sourced by Netflix in 2016, is now maintained by the Conductor OSS Foundation. Orkes, whose founders were involved in Conductor's development at Netflix, leads this foundation and offers a hosted version called Orkes Conductor. This analysis specifically focuses on Orkes Conductor version 5.3.2.

Workflow Definition and Execution

Workflow definitions in Conductor are JSON structures that specify tasks, their order, and input/output requirements. Each definition has a unique name and version. A single workflow definition can be run multiple times, with each instance referred to as a workflow execution, identified by a unique workflow ID. Tasks within an execution can also have multiple attempts, each with a unique execution ID.

Task Types and System Components

Conductor includes various task types: built-in system tasks for HTTP requests or Kafka events, operators for control flow like conditionals and loops, and worker tasks for user-defined business logic. Operators can execute ECMAScript code. Workflows can be initiated via API calls, message broker events, or Conductor's cron-style scheduler. Once acknowledged, Conductor schedules tasks and works towards completion.