HS

Himanshu Sharma

Lead Platform Engineer

← All blogs

Architecture • iPaaS • Conductor

Blueprint: High-Availability iPaaS on Netflix Conductor

The Freshworks integration platform orchestrates 30+ workflows at 99.9% release reliability. This is the control-plane / data-plane split, multi-region topology, and adoption path we use—including workflows that drive AI agents.

Author: Himanshu Sharma Updated: Aug 2026 13 min read
99.9%release reliability
30+critical workflows
2AWS regions

Control plane vs data plane

Conductor is the control plane. Workflow YAML lives in Git and promotes through Jenkins. Workers and adapters are the data plane on Kubernetes (spot + on-demand mix).

CONTROL PLANE Git workflow packs + owners Conductor server + summary index Jenkins · contract tests shadow env · canary · rollback DATA PLANE K8s workers (Java / Node) Adapters · bots · agent tools DynamoDB global + CDC MySQL replicas for reporting promote
Git → Jenkins promotion Worker execution plane
Definitions and promotion live left; execution and state live right.

Multi-region topology

Active/warm standby across two AWS regions. CDC keeps state equivalent; worker images and pipelines are mirrored so failover is a DNS + queue drain, not a rebuild.

Region A · active Conductor + workers Local queues + caches Region B · warm Conductor (warm) + workers Replicated queues + caches CDC Shared: DynamoDB global tables · Jenkins · Grafana · PagerDuty
CDC replication stream Warm standby workers
Warm region is always running workers; promotion is traffic, not bootstrap.

Release pipeline with shadow env

Every workflow pack passes contract tests in a shadow Conductor cluster that mirrors prod task domains but uses synthetic tenants. Canary promotion watches workflow failure rate—not just pod health.

Git merge Contract tests Shadow run Canary 5% Full prod Auto-rollback if workflow error rate or p99 task latency exceeds SLO for 10 min
Shadow env catches schema drift before a canary touches real tenants.

Reliability tactics

  1. Automated runbooks: Conductor events feed PagerDuty and remediation Lambdas.
  2. Release safety: contract tests per workflow plus a shadow environment.
  3. Blast-radius limits: circuit breakers per task domain.
  4. Idempotent terminate: skip already-terminated workflows (PR #766).
Product teams consume declarative workflow packs. They get telemetry, chaos toggles, and Java/Node adapter libraries by default. Agent tool-calls are just another adapter family on this plane.

Adoption journey

1 Model

YAML + task domains + owners.

2 Adapt

Pair with platform for connectors.

3 Prove

Contracts + chaos in sandbox.

4 Graduate

Prod SLOs and automated guardrails.

AIAgentic platform architecture OrchestrationDurable agent workflows SagasCDC choreography