Pipeline & Runtime

Pipeline Stages

Every message flows through an ordered sequence of stages. Each stage is optional — the engine skips any stage that has no handler configured.

Source
  → Preprocessor
    → Validator
      → Source Filter
        → Transformer
          → Destination Filter
            → Destination Transformer
              → Send
                → Response Transformer
                  → Postprocessor

Stage	Purpose
Source	Receives raw bytes from the listener (TCP, HTTP, Kafka, file, etc.) and wraps them in an IntuMessage
Preprocessor	Runs before any business logic — decode character sets, decompress, strip wrappers
Validator	Checks structural and semantic validity (e.g. HL7v2 schema conformance, FHIR profile validation)
Source Filter	Decides whether to accept or drop the message based on source-side criteria
Transformer	Core business logic — map fields, translate codes, reshape payloads
Destination Filter	Per-destination routing — include or exclude the message for each configured destination
Destination Transformer	Per-destination payload adjustments (e.g. different FHIR profiles for different receivers)
Send	Delivers the message to the destination connector
Response Transformer	Processes the destination’s response before returning it upstream
Postprocessor	Runs after everything else — audit logging, metrics, cleanup

Note

Stages run in-process and share the same IntuMessage reference. Mutations made in one stage are visible to all subsequent stages.

Error Handling

When a stage throws an error or returns a failure status, the engine:

1. Records the error on the message (status, error message, stack trace).
2. Skips remaining stages — the message does not continue through the pipeline.
3. Triggers the retry policy if one is configured on the channel.
4. Returns an error response to the source connector (e.g. an HL7v2 AE ACK or an HTTP 500).

Common error status codes:

Code	Meaning
VALIDATION_FAILED	The validator rejected the message
TRANSFORM_ERROR	The transformer threw an exception
FILTER_REJECTED	A filter explicitly dropped the message
DESTINATION_ERROR	The destination connector could not deliver
TIMEOUT	A stage exceeded its deadline

Caution

Unhandled exceptions in TypeScript code are caught by the engine and surfaced as TRANSFORM_ERROR. Always validate assumptions inside your transformer to produce clearer error messages.

Retry & Dead-Letter Queue

Retry Configuration

Failed deliveries can be retried automatically. Configure the policy in channel.yaml:

retry:
  max_attempts: 5
  backoff: exponential     # constant | linear | exponential
  initial_delay_ms: 500
  max_delay_ms: 30000
  jitter: true

Key	Type	Default	Description
max_attempts	integer	3	Total delivery attempts (including the first)
backoff	string	constant	Delay strategy between attempts
initial_delay_ms	integer	1000	Delay before the first retry
max_delay_ms	integer	60000	Upper bound on delay (for linear/exponential)
jitter	boolean	false	Add random jitter to prevent thundering-herd

Backoff strategies:

• constant — wait initial_delay_ms between every attempt
• linear — increase by initial_delay_ms each attempt (500, 1000, 1500 …)
• exponential — double the delay each attempt (500, 1000, 2000, 4000 …), capped at max_delay_ms

Dead-Letter Queue

When all retry attempts are exhausted, the message is routed to the dead-letter queue (DLQ). The DLQ stores the original message, the last error, and all attempt metadata so operators can inspect and replay failures.

retry:
  max_attempts: 5
  backoff: exponential
  initial_delay_ms: 1000
  max_delay_ms: 30000
  dlq:
    destination: dlq_store

Tip

Set up an alert on DLQ depth. A growing DLQ often indicates a downstream system is misconfigured or unreachable.

Hot Reload

When runtime.hot_reload is true, the engine uses fsnotify to watch for file changes:

• channel.yaml — channel configuration changes trigger a graceful restart of only the affected channel.
• .ts files — TypeScript transformer, validator, and filter files are recompiled automatically and hot-swapped.
• intu.yaml — root config changes require a full engine restart.

Only the affected channel is restarted. Other channels continue processing without interruption.

Note

Hot reload is designed for development. In production, set hot_reload: false and use rolling deployments or intu reload for controlled restarts.

Batch Processing

Some sources deliver multiple messages in a single payload. intu can split these into individual messages using splitters:

listener:
  type: tcp
  port: 6661
  content_type: hl7v2
  splitter: hl7_batch

Splitter	Input	Behaviour
hl7_batch	HL7v2 batch (BHS/BTS wrapped)	Extracts each MSH segment group
fhir_bundle	FHIR Bundle resource	Extracts each Bundle.entry.resource
newline	Newline-delimited text	Splits on \n
xml_root	XML with repeated child elements	Splits on the first-level child element

Each sub-message is wrapped in its own IntuMessage and processed independently through the full pipeline. Failures in one sub-message do not affect others.

Map Variables

intu provides four scoped key-value maps accessible from TypeScript code:

Map	Scope	Lifetime	Use Case
globalMap	All channels	Engine lifetime	Shared lookup tables, cross-channel state
channelMap	Single channel	Channel lifetime	Channel-scoped counters, caches
connectorMap	Single connector	Connector lifetime	Connection-specific state (sequence numbers)
responseMap	Single message	Request → response cycle	Pass data from the transformer to the response transformer

export default function transform(msg: IntuMessage, ctx: IntuContext) {
  const counter = (ctx.channelMap.get("seq") as number ?? 0) + 1;
  ctx.channelMap.set("seq", counter);

  const lookup = ctx.globalMap.get("facilityNames") as Map<string, string>;
  const name = lookup?.get(msg.body.MSH[4]) ?? "UNKNOWN";

  ctx.responseMap.set("processedAt", new Date().toISOString());
  return msg;
}

Caution

globalMap is shared across all channels and workers. Use it for read-heavy reference data. Frequent writes from multiple channels can cause contention.

Supported Data Types

intu natively parses and generates the following healthcare and general data formats:

Type	Content Type	Description
HL7v2	hl7v2	ADT, ORM, ORU, MDM, and all standard message types
FHIR R4	fhir+json, fhir+xml	Resources, Bundles, Parameters
X12	x12	837, 835, 270/271, 276/277 transaction sets
CDA/CCDA	cda+xml	Clinical Document Architecture
DICOM	dicom	Metadata extraction and routing (no pixel data)
JSON	json	Generic JSON payloads
XML	xml	Generic XML documents
CSV	csv	Delimited text with configurable separators

Code Template Libraries

Shared TypeScript utilities can be placed in code_templates/ directories at the project root or inside a channel folder:

my-project/
├── code_templates/          # project-wide shared code
│   ├── hl7-helpers.ts
│   └── fhir-mappers.ts
└── channels/
    └── adt-to-fhir/
        ├── channel.yaml
        ├── transform.ts
        └── code_templates/  # channel-specific shared code
            └── adt-utils.ts

Import shared code in your transformers using relative paths:

import { buildPatientResource } from "../code_templates/fhir-mappers";

Channel-level code_templates/ are resolved first, then project-level — allowing channels to override shared utilities.

Message Storage

Storage mode is configurable globally in intu.yaml and can be overridden per channel:

# Global default
storage:
  driver: postgres
  mode: status

# Per-channel override in channel.yaml
storage:
  mode: full

Mode	What is Stored	Typical Use
none	Nothing	High-throughput fire-and-forget pipelines
status	Message ID, status, timestamps, errors	Monitoring and alerting without payload overhead
full	Complete IntuMessage including body	Audit trails, debugging, message replay

Tip

Use full storage during development and initial go-live, then switch to status once the channel is stable to reduce storage costs.