File Source

The File source watches a directory at a configurable poll interval and feeds each new file into the channel pipeline. It supports local filesystems, FTP, Amazon S3, and SMB network shares through a unified configuration.

After a file is successfully processed it is moved to an archive directory. Files that fail processing are moved to an error directory so they can be reviewed without blocking the queue.

Configuration

source:
  type: file
  scheme: local
  directory: /data/inbound
  file_pattern: "*.hl7"
  poll_interval: 30s
  move_to: /data/archive
  error_dir: /data/error
  sort_by: date

Properties

Property	Type	Required	Default	Description
scheme	string	Yes	—	Transport scheme: local, ftp, s3, or smb.
directory	string	Yes	—	Path to the directory to poll.
file_pattern	string	No	*	Glob pattern to match filenames (e.g. *.hl7, ADT_*.txt).
poll_interval	string	No	30s	How often to check for new files. Supports s, m, h suffixes.
move_to	string	No	—	Directory to move files to after successful processing.
error_dir	string	No	—	Directory to move files to when processing fails.
sort_by	string	No	name	Order in which files are processed: name, date, or size.

S3 Sub-Properties

When scheme is s3, the following additional properties are available under the s3 key:

Property	Type	Required	Default	Description
bucket	string	Yes	—	S3 bucket name.
region	string	Yes	—	AWS region (e.g. us-east-1).
prefix	string	No	—	Key prefix to filter objects.
auth	object	No	—	Explicit credentials. Falls back to the default AWS credential chain when omitted.

source:
  type: file
  scheme: s3
  directory: /incoming
  file_pattern: "*.json"
  poll_interval: 1m
  s3:
    bucket: ehr-integration-inbox
    region: us-east-1
    prefix: facility-01/
    auth:
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Complete Example

Poll a local directory for HL7v2 files and forward them over MLLP:

channel.yaml

channel:
  name: lab-file-pickup
  description: Pick up lab result files and forward to the LIS

source:
  type: file
  scheme: local
  directory: /data/lab/inbound
  file_pattern: "*.hl7"
  poll_interval: 30s
  move_to: /data/lab/archive
  error_dir: /data/lab/error
  sort_by: date

transformer:
  type: typescript
  file: ./transforms/enrich-lab.ts

destination:
  type: tcp
  host: lis.hospital.local
  port: 2575
  mode: mllp

Tip

Set sort_by: date when processing order matters, such as sequential lab results or medication orders. This ensures older files are processed first.

Note

When using scheme: ftp or scheme: smb, provide connection credentials under auth the same way you would for S3. The directory value becomes the remote path on the server.