Skip to content

Instrumentation & Observability ​

flux stays vendor-agnostic: the core module never imports OpenTelemetry. Instead it carries a small, portable tracing model on typed contexts and event envelopes. The official bridge that turns that model into OpenTelemetry spans and metrics lives in a separate repository: flux-opentelemetry (package fluxotel).

Vendor-Neutral Primitives in Core ​

Every typed context (flux.Context, command.Context, query.Context, event.Context, projection.Context, workflow.Context) exposes:

go
Instrumentation() Instrumentation
go
type Instrumentation struct {
	TraceID    string
	SpanID     string
	TraceFlags string // Canonical single-byte hex (e.g. "01")
}

Instrumentation.IsValid() is true when both TraceID and SpanID are non-empty.

Reparenting with WithParent ​

Handlers often need to attach a child span (or a deadline) without losing actor, correlation, causation, or envelope coordinates. Use the typed WithParent(parent context.Context) method (or flux.WithParent for the base context) so the Go context.Context is replaced while all flux metadata is preserved.

TIP

See Typed Contexts for the full interface surface, including event.EventMetadata.

Envelope Metadata Keys ​

When an aggregate is saved, AggregateRepository.Save copies a valid Instrumentation onto each new envelope using OpenTelemetry-compatible keys:

ConstantMetadata key
flux.MetadataTraceIDtrace_id
flux.MetadataSpanIDspan_id
flux.MetadataTraceFlagstrace_flags

When events are later handled, event.NewContext hydrates Instrumentation back from those keys so projectors and workflows continue the same distributed trace.

Outbox Continuity ​

Workflow stores persist the triggering event's Instrumentation on outbox messages. When the relay dispatches a command, that instrumentation is restored on the command.Context, so asynchronous command handling stays on the originating TraceID.

Official OpenTelemetry Integration ​

github.com/wotek/flux-opentelemetry translates flux instrumentation into OpenTelemetry. It provides:

  • Bus middlewares — spans and metrics for command and query handlers
  • Store decorators — client spans for event, snapshot, and projection stores
  • Async wrappers — projection and workflow handlers that continue remote parents
  • Low-cardinality metrics — latency, throughput, and error rates with locked ok / error status attributes

Installation ​

bash
go get github.com/wotek/flux-opentelemetry

Requires github.com/wotek/flux@v1.3.0 or later.

go
import (
	"go.opentelemetry.io/otel"

	fluxotel "github.com/wotek/flux-opentelemetry"
)

tracer := otel.Tracer("my-service")
meter := otel.Meter("my-service")

Command and Query Buses ​

go
cmdBus := command.New()
cmdBus.Use(fluxotel.CommandMiddleware(tracer, meter))

queryBus := query.New()
queryBus.Use(fluxotel.QueryMiddleware(tracer, meter))

When a command arrives from an outbox relay with a valid remote Instrumentation, CommandMiddleware parents the new span via trace.ContextWithRemoteSpanContext, preserving end-to-end continuity.

Storage Decorators ​

Wrap stores at composition time so domain code stays unaware of telemetry:

go
eventStore := fluxotel.WrapEventStore(rawEventStore, tracer, meter)
snapshotStore := fluxotel.WrapSnapshotStore(rawSnapshotStore, tracer, meter)
projectionStore := fluxotel.WrapProjectionStore(rawProjStore, tracer, meter)

Iterator spans

Read and Stream use a two-phase span lifecycle: a short setup span for opening the iterator, then an iteration span that ends when ranging finishes, stops early, or fails. Always consume returned iterators so spans close cleanly.

Snapshot misses

flux.ErrSnapshotNotFound is expected cache-miss behavior. Decorators keep span status ok, set flux.snapshot.miss=true, and increment flux.snapshot.load.misses. Real store failures still record errors.

Projection and Workflow Handlers ​

Wrap handlers before registration so each invocation is a child span under the event's remote parent:

go
projection.RegisterHandler(projector, fluxotel.WrapProjectionHandler(tracer, meter,
	func(ctx projection.Context, e OrderPlaced) error {
		// Read-model mutation
		return orderReadStore.Save(ctx, e.OrderID)
	},
))

workflow.RegisterHandler(orchestrator, wfStore, fluxotel.WrapWorkflowHandler(tracer, meter,
	func(ctx workflow.Context, wf *OrderFulfillmentWorkflow, e OrderPlaced) error {
		workflow.EnqueueCommand(ctx, SendWelcomeEmailCommand{OrderID: e.OrderID})
		return nil
	},
))

End-to-End Trace Continuity ​

A typical happy path shares one TraceID across synchronous and asynchronous work:

  1. Command bus — flux.command span; middleware stamps active span IDs onto the typed context.
  2. Aggregate save — repository writes trace_id / span_id / trace_flags into envelope metadata; event store append is a child client span.
  3. Projector — hydrates instrumentation from the envelope; wrapper starts flux.projection as a remote child of the originating command span.
  4. Workflow — same pattern with flux.workflow; enqueued commands carry the event's instrumentation in the outbox.
  5. Outbox relay — dispatches the command; CommandMiddleware remote-parents the follow-up flux.command span onto the same trace.

Metrics Catalog (Summary) ​

Status attributes are only "ok" or "error". High-cardinality values (stream IDs, aggregate IDs, user IDs) are never metric attributes—those belong on spans when useful for correlation.

AreaInstruments
Commands / queriesflux.command.duration, flux.command.count, flux.query.duration, flux.query.count
Event storeflux.event_store.append.duration, .concurrency_conflicts, .events, flux.event_store.read.duration
Snapshotsflux.snapshot.load.duration, .save.duration, .load.misses
Projectionsflux.projection.store.update.duration, flux.projection.handler.duration, .count
Workflowsflux.workflow.handler.duration, flux.workflow.handler.count

Outbox queue depth / dispatch metrics are reserved for a later fluxotel release; v1 still traces outbox-driven commands through CommandMiddleware.

Logs via OpenTelemetry ​

flux does not ship a logging backend. Typed contexts expose Logger() *slog.Logger, built from slog.Default() plus actor / correlation_id / causation_id. To send those logs to an OpenTelemetry Collector (and any OTLP logs backend), bridge slog → OTel logs in your application bootstrap—the same place you install the tracer and meter providers.

Shape ​

text
ctx.Logger() / slog.Default()
        │  otelslog bridge
        ▼
  LoggerProvider (sdk/log) + BatchProcessor
        │  OTLP HTTP (/v1/logs)
        ▼
  OpenTelemetry Collector  →  your logs backend

Core flux stays free of OpenTelemetry imports. Prefer putting shared helpers in flux-opentelemetry when you add a reusable bridge; apps still own the LoggerProvider, exporter, and shutdown.

PackageRole
go.opentelemetry.io/otel/sdk/logLoggerProvider
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpOTLP HTTP exporter (honors OTEL_EXPORTER_OTLP_ENDPOINT)
go.opentelemetry.io/contrib/bridges/otelslogslog.Handler / logger that emits OTel log records

Pin versions that resolve cleanly with the same go.opentelemetry.io/otel major line you use for traces and metrics.

Bootstrap sketch ​

go
import (
	"context"
	"log/slog"

	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	"go.opentelemetry.io/otel/sdk/resource"
	"go.opentelemetry.io/otel/attribute"
	logglobal "go.opentelemetry.io/otel/log/global"
)

func initLogging(ctx context.Context, serviceName string) (func(context.Context) error, error) {
	exp, err := otlploghttp.New(ctx) // uses OTEL_EXPORTER_OTLP_ENDPOINT
	if err != nil {
		return nil, err
	}
	res, err := resource.New(ctx, resource.WithAttributes(
		attribute.String("service.name", serviceName),
	))
	if err != nil {
		_ = exp.Shutdown(ctx)
		return nil, err
	}
	lp := sdklog.NewLoggerProvider(
		sdklog.WithProcessor(sdklog.NewBatchProcessor(exp)),
		sdklog.WithResource(res),
	)
	logglobal.SetLoggerProvider(lp)
	slog.SetDefault(otelslog.NewLogger(serviceName)) // flux.NewContext picks this up
	return lp.Shutdown, nil
}

Call this before creating command/query contexts so ctx.Logger() is OTel-backed.

Correlation ​

Use context-aware slog APIs so records attach to the active span:

go
ctx.Logger().InfoContext(ctx, "order placed", "order_id", orderID.String())

Logger().Info(...) without a context usually won't get trace_id / span_id. Prefer InfoContext / ErrorContext (also required by flux project conventions).

Collector pipeline ​

Apps keep a single OTEL_EXPORTER_OTLP_ENDPOINT. Route logs in the collector (Jaeger is traces-oriented and is not a logs backend):

yaml
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/your-logs-backend]

What this example does not do ​

The Temporal Payment compose stack exports traces to Jaeger. It does not wire OTLP logs. Use the pattern above when you want logs in the same OTLP path as traces/metrics.

Further Reading ​