Skip to main content
Omni exports OpenTelemetry traces from the core services: omni-web (Node/SvelteKit), omni-ai (Python/FastAPI), omni-searcher (Rust), omni-indexer (Rust), and omni-connector-manager (Rust). Connectors capture traces individually through the connector SDK. Metrics and OTLP log export are not currently implemented; traces are exported as OTLP/HTTP to the configured endpoint.

​
Architecture & Signal Flow

Omni does not bundle or require an OpenTelemetry Collector or backend. Traces are sent to whatever endpoint is configured in OTEL_EXPORTER_OTLP_ENDPOINT. The default value is empty, which means instrumentation is active but signals are dropped / not exported or stored. When a non-empty endpoint is set, traces are sent to ${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces. Point OTEL_EXPORTER_OTLP_ENDPOINT at any compatible OTLP/HTTP collector or backend. See Production & Privacy Guidance for production recommendations.

​
Base OTLP Endpoint Semantics

​
Resource Attributes

Every trace carries:
  • service.name: the logical service name (omni-searcher, omni-indexer, omni-ai, omni-web, omni-connector-manager)
  • service.version: from SERVICE_VERSION
  • deployment.environment: from OTEL_DEPLOYMENT_ENVIRONMENT
  • deployment.id: from OTEL_DEPLOYMENT_ID

​
Shutdown Behaviour

On graceful shutdown (SIGTERM/SIGINT), each service flushes pending spans before exiting. In Rust services the OTLP exporter timeout is hard-coded at 10 seconds in the opentelemetry-rust SDK; OTEL_EXPORTER_OTLP_TIMEOUT is not honoured. Busy shutdowns that exceed the Rust SDK timeout may lose the final batch. Python (omni-ai) and Node (omni-web) use their SDK defaults.

​
Trace Continuity

​
HTTP Request Tracing

Rust services (searcher, indexer, connector-manager) use custom axum middleware. SERVER spans use METHOD /path names (e.g. GET /v1/search); the trace_id is also propagated as a request id. W3C traceparent / tracestate is propagated in HTTP headers: inbound requests extract the parent context, and outbound HTTP calls inject it via the shared http_client helper. Node (omni-web) and Python (omni-ai) use auto-instrumentation provided by the respective OpenTelemetry SDKs (Node auto-instrumentations with the filesystem instrumentation disabled; Python FastAPI and HTTPX instrumentation). Span names follow the SDK convention rather than a unified scheme. Outbound HTTP requests create a CLIENT span, producing parent/child relationships within a single trace (not links) when the web service calls internal APIs. The same traceId is shared across the web→searcher and web→AI call chain.

​
Queue Processing

The indexer and connector-manager consume events from Postgres-based queues (connector_events_queue, embedding_queue). Queue rows do not currently carry trace context, so consumer processing starts a new root trace independent of the producer.

​
Logs

Logs are written to the container’s stdout (standard docker logs), not exported via OTLP. Rust services emit structured JSON logs through tracing-subscriber; Python and Node use their standard logging. No global PII scrubber is applied. Runtime logging call sites are audited to avoid sensitive and high-cardinality fields, but log content is produced by the application as-is. In Rust services, when an active trace span exists at the log call site, trace_id/span_id are attached to the log record via the tracing–OpenTelemetry layer.

​
Troubleshooting

​
Production & Privacy Guidance

When using an external OTLP collector or backend in production:
  1. Use TLS. Enable TLS on the OTLP endpoint, or use gRPC with TLS.
  2. Authenticate. Use API keys or mTLS to secure the connection.
  3. Configure sampling. Use a probabilistic_sampler or tail_sampling processor in your collector for high-throughput services.
  4. Set resource limits. Configure memory limits and spike limits in the collector or backend.
  5. Never expose the OTLP endpoint publicly. It has no built-in authentication. Bind to loopback or use a private network.
  6. Filter sensitive data. Omni applies no PII scrubbing. Redact sensitive fields in the collector or at the application level.