FinBox runs a lot of short-lived data transformation workflows. An event comes in, gets validated, enriched from a few sources, transformed into a target shape, and forwarded downstream. These aren't long-running processes — they complete in milliseconds to seconds.
The natural instinct is to reach for Temporal. But Temporal is designed for durable, long-running workflows that need to survive process restarts. For short-running in-memory work, it introduces unnecessary overhead: a separate Temporal server, worker processes, workflow history serialization. For a workflow that runs in 50ms, that infrastructure cost isn't justified.
DataDancer was built to fill that gap.
What it is
DataDancer is a Go library that implements the Serverless Workflow Specification for in-memory execution. Workflows are defined in JSON following the open standard — no custom DSL, no proprietary format. The engine executes them entirely in-process with zero external dependencies.
The workflow model is built around three state types: operation states that execute activities in sequence, switch states that branch on conditions evaluated against the current data, and sleep states for timed pauses. These three primitives cover the vast majority of real data processing workflows.
This workflow runs entirely in-process. If the process dies, the workflow doesn't resume — but for the class of short-running work we're targeting, that's the right tradeoff.
The JSON transformation engine
A significant part of the work was building the transformation layer. Real-world workflows don't just call functions in sequence — they need to reshape data between steps: extract fields, rename keys, compute derived values, filter arrays.
DataDancer includes a config-driven JSON transformation engine that handles this declaratively. You describe the transformation in the workflow definition, and the engine applies it without custom code. For more complex cases, the spec supports custom function activities that DataDancer can invoke directly.
DataDancer ships a standard library of built-in activities covering the most common operations: data transformation via JQ, HTTP requests to external services, encryption and decryption, and format conversion utilities for HTML and XML.
Custom activities are registered through a simple Go interface. Any method on a registered struct becomes callable by name from a workflow definition.
Extensibility
One of the explicit design goals was zero hard dependencies. Microservices that embed DataDancer should be able to stay lean.
The library achieves this through injection interfaces. Teams can inject:
- Custom activities — Go functions registered by name, callable from workflow definitions
- Custom loggers — any logger implementing the interface, defaulting to a no-op
- OpenTelemetry instrumentation — trace and span hooks that wire into the host service's observability setup
None of these are required. A service that just wants basic workflow execution adds no external dependencies at all.
Serverless Workflow spec compliance
Implementing a spec rather than building a proprietary engine was a deliberate choice. The Serverless Workflow Specification is an open standard with broad industry support. Building against it means workflows are portable, tooling exists, and the mental model is shareable.
The spec covers more than DataDancer currently implements — durable execution, event-driven triggers, compensation flows. We've scoped to what short-running in-memory execution needs. The spec's design made it possible to implement a strict subset without compromising correctness.
DataDancer is now used across 4+ microservices at FinBox. The consistent pattern — define the workflow in JSON, register the custom activities in Go — makes it easier to understand what a service does without reading imperative code.