BDD Testing¶

Overview¶

BDD (Behaviour-Driven Development) tests verify SolidSyslog end-to-end: a C program sends an RFC 5424 syslog message over UDP, TCP, TLS or mutual TLS to a real syslog server, and Behave (Python) asserts that the received message was parsed into the expected fields.

This catches conformance issues that unit tests cannot: syslog-ng's RFC 5424 parser is the test oracle, so a malformed message that happens to match a byte-level expectation in a unit test will fail here.

Architecture¶

                                  ┌──────────────┐
                                  │    Behave     │
                                  │  (container)  │
                                  └──┬────────┬──┘
                                     │ runs   │ reads
                                     ▼        │
                              ┌──────────────┐│
                              │  BDD target  ││
                              │    binary    ││
                              └──────┬───────┘│
                                     │        │
                            UDP 5514 / TCP 5514│
                            TLS 6514 / mTLS 6515
                                     │        │
                                     ▼        │
                              ┌──────────────┐│
                              │   syslog-ng  ││
                              │  (container) ││
                              └──────┬───────┘│
                                     │ writes │
                                     ▼        ▼
                              Bdd/output/received*.log

BDD targets pair with their own oracle so jobs and developers switching containers never interfere. Each pair lives on its own bridge network so the syslog-ng DNS alias is scoped per-pair; see Bdd/README.md for the per-target compose layout, the BDD_TARGET env-var contract, and FreeRTOS local fault-finding tips.

Target	Behave runner	Oracle
Linux	`behave-linux`	`syslog-ng-linux`
FreeRTOS (Plus-TCP)	`behave-freertos` (cross image carries QEMU + Behave)	`syslog-ng-freertos` (shared netns; QEMU slirp `10.0.2.2` reaches it on loopback)
FreeRTOS (lwIP Raw API)	`behave-freertos-lwip`	`syslog-ng-freertos-lwip`, same shape
CMSIS-RTOS2 (lwIP Sockets API)	`behave-cmsis-lwip`	`syslog-ng-cmsis-lwip`, same shape
Windows	`behave` on the runner	`otelcol-contrib` (no compose; runner-direct)
VxWorks 6.4	`behave` natively on the development machine, driving QEMU on a build machine through the VxWorks 6.4 runner (`Bdd/Targets/VxWorks64/Run-VxWorks64Bdd.ps1`)	`otelcol-contrib` on the development machine, on UDP and TCP 5514 (`Bdd/otel/config.vxworks64.yaml`)

The Linux BDD target binary is built in the gcc container but executed by Behave via subprocess.run. Both services share the workspace mount, so Bdd/output/received.log is visible to both syslog-ng (writer) and Behave (reader) without any network file transfer. The FreeRTOS ELF is built in the cross image and run under qemu-system-arm from inside the freertos-target service; Behave drives it through the QEMU UART (-serial stdio).

Key files¶

File	Purpose
`Bdd/syslog-ng/syslog-ng.conf`	syslog-ng configuration - UDP, TCP, TLS and mutual-TLS sources, a TLS source the target cannot chain to, key=value template output
`ghcr.io/cososo-ltd/behave`	GHCR image - Debian trixie + Python + Behave (source)
`Bdd/output/`	Shared directory - syslog-ng writes here, Behave reads
`Bdd/features/`	Gherkin feature files and step definitions
`Bdd/Targets/Linux/main.c`	Linux BDD target: PosixMessageQueueBuffer + service pthread + SwitchingSender. Accepts `--facility`, `--severity`, `--transport`, `--app-name`, store-config flags
`Bdd/Targets/CMakeLists.txt`	Builds the BDD target binary, linked against the SolidSyslog library

syslog-ng configuration¶

The syslog-ng config (Bdd/syslog-ng/syslog-ng.conf) declares four syslog() sources - UDP and TCP on 5514, server-auth TLS on 6514, and mutual TLS (peer-verify(required-trusted)) on 6515 - all parsing RFC 5424 natively. Every message is tee'd to both received.log (catch-all) and a per-transport log file (received_udp.log, received_tcp.log, received_tls.log, received_mtls.log) using a key=value template:

PRIORITY=14 TIMESTAMP=2009-03-23T00:00:00+00:00 HOSTNAME=TestHost APP_NAME=TestApp PROCID=1234 MSGID=TestMsgId STRUCTURED_DATA= MSG=Test message

Important syslog-ng behaviours¶

These are deliberate syslog-ng design choices, not bugs:

Behaviour	Detail
`keep-hostname(yes)`	Required in the source config. Without it, syslog-ng replaces the RFC 5424 HOSTNAME field with the sender's transport-level IP address. `$HOST` then reflects the parsed payload hostname; `$HOST_FROM` always contains the sender IP regardless.
Timestamp reformat	syslog-ng normalises timestamps via `$ISODATE`. A sent value of `...T00:00:00.000Z` will appear as `...T00:00:00+00:00`.
Nil structured data	The RFC 5424 nil value `-` for SDATA produces an empty string in `$SDATA`, not `-`.
MSG leading space	BSD syslog convention may prepend a space to the message body. Step definitions should use `.strip()`.

flush_lines(1)¶

The destination uses flush_lines(1) so that each message is written to disk immediately. This avoids flaky BDD tests that poll for output before the buffer is flushed.

Reconfiguring syslog-ng from Behave¶

The syslog-ng and behave containers share a named Docker volume (syslog-ng-ctl) mounted at /var/lib/syslog-ng. This exposes syslog-ng's Unix control socket to the behave container, allowing Behave to trigger a config reload without SSH or docker compose exec.

Since the syslog-ng config file is bind-mounted from the workspace (Bdd/syslog-ng/syslog-ng.conf), Behave can write a new config to that path via the shared workspace mount and then reload:

import socket

sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect("/var/lib/syslog-ng/syslog-ng.ctl")
sock.sendall(b"RELOAD\n")
response = sock.recv(1024)  # b'OK Config reload successful\n.\n'
sock.close()

This is exercised by scenarios that need a different syslog-ng source configuration, for example, the TLS and mTLS scenarios swap in tighter peer-verify settings and reload without restarting the container.

Feature tags¶

Features are tagged by capability, what the system under test must provide for the scenario to be meaningful. Runners declare which capabilities they have by excluding the tags they lack. This lets us add new runners (Windows, bare-metal RTOS, TLS-only client, etc.) without rewriting feature tags.

Tag	Meaning
`@udp`	Needs UDP transport
`@tcp`	Needs TCP transport (RFC 6587 framing)
`@tls`	Needs TLS transport (RFC 5425, server-auth). Excluded on VxWorks 6.4, which carries no TLS.
`@mtls`	Needs mutual TLS (client cert + key)
`@buffered`	Needs a buffered wiring (CircularBuffer + service thread, PosixMessageQueueBuffer, etc.) beyond the single-task PassthroughBuffer path - e.g. file-backed block store, switching sender between transports, syslog-ng reload via the UNIX control socket. Every target binary carries a buffered wiring, so `@buffered` is not excluded by any runner. TLS and mTLS do not carry `@buffered`, because the OTel oracle provides TLS receivers (Windows otelcol-contrib listens on 6514 / 6515 with `client_ca_file` for mTLS).
`@store`	Needs file-backed `SolidSyslogBlockStore` capability in the target (write blocks to disk, replay across restart, threshold callbacks). Carried alongside `@tcp @buffered` on the store-and-forward / capacity / power-cycle / block-lifecycle features. Run on every target - Linux and Windows over `SolidSyslogPosixFile` / `SolidSyslogWindowsFile`, the three QEMU targets over `SolidSyslogPlusFatFile` (FreeRTOS-Plus-TCP), `SolidSyslogFatFsFile` (lwIP Raw API) and `SolidSyslogLittleFsFile` (CMSIS-RTOS2), and VxWorks 6.4 over `SolidSyslogVxWorks64File`.
`@host_store`	Needs the target's store on the host's filesystem, where a step can spoil the store's path before the target starts. Run on Linux and Windows; excluded on the QEMU targets and VxWorks 6.4, whose store media the harness cannot reach.
`@rtc`	Scenario assumes a real-time clock with synchronised wall-clock time - asserts a known absolute TIMESTAMP. Run on Linux, Windows and VxWorks 6.4, whose harness sets the target's clock at startup; excluded on the FreeRTOS and CMSIS-RTOS2 targets, which model a no-RTC product per RFC 5424 section 6.2.3.1.
`@no_rtc`	Scenario asserts the no-RTC product behaviour over the wire (`tzKnown="0"`, `isSynced="0"`). Run on the FreeRTOS and CMSIS-RTOS2 targets; excluded on Linux, Windows and VxWorks 6.4. The complementary pair of `@rtc`.
`@requires_message_size_1500`	Scenario requires `SOLIDSYSLOG_MAX_MESSAGE_SIZE` to be at least 1500 bytes - used by the UDP path-MTU clipping feature, which has to drive an oversized payload through `EMSGSIZE`. The library default is below that by design, so the Linux and Windows targets are built against the tunable override that supplies it (`tunable-override-debug` / `msvc-tunable-override`); the QEMU targets run at the default, so the gate skips the feature there.
`@hmac`	Needs an HMAC-SHA256 at-rest policy wired in the target, so a record sealed on write is verified on replay-read. Carried alongside `@store`, and excluded on VxWorks 6.4, which carries no TLS library to provide the policy.
`@aesgcm`	Needs an AES-256-GCM at-rest policy wired in the target - authenticated encryption rather than integrity alone. Carried alongside `@store`, and excluded on Windows, which wires no AES-GCM policy, and on VxWorks 6.4, which carries no TLS library to provide it.
`@tls13`	Scenario asserts the handshake against a server that refuses everything below TLS 1.3, so delivery alone proves the version negotiated.

Rollout markers are also used (temporary; remove once the scenario passes):

Tag	Meaning
`@wip`	Skip everywhere - work in progress. Not currently in use on any scenario.
`@windows_wip`	Skip on Windows only - should work but not yet verified. Currently on `tcp_singletask.feature` (whole feature) and one scenario in `udp_mtu.feature`.
`@freertoswip`	Skip on the QEMU targets only - scenario currently fails or errors on them and is gated until the relevant capability lands. Each tagged scenario is a follow-up tied to a specific gap; the tag is removed scenario-by-scenario as the gap closes. Not currently in use on any scenario.
`@vxworks64wip`	Skip on the VxWorks 6.4 target only - the scenario needs a capability that target does not have yet, and loses the tag when the story that adds it lands.

Which runner excludes which tags is set on each Behave service's command in ci/docker-compose.bdd.yml, on the Windows step in .github/workflows/ci.yml, and in Bdd/Targets/VxWorks64/Run-VxWorks64Bdd.ps1. Read the filters there rather than from a copy here: a runner is added or a tag retired by editing those files, and a table restating them is wrong the moment one changes.

Two oracles, one step file¶

Step definitions read the active oracle from ORACLE_FORMAT:

Oracle	`ORACLE_FORMAT`	`RECEIVED_LOG` default	Runs on
syslog-ng (key=value text)	`syslog-ng`	`Bdd/output/received.log`	Linux container
OTel Collector Contrib (JSON Lines)	`otel-jsonl`	`Bdd/output/received.jsonl`	Windows native, and the VxWorks 6.4 run from the development machine

parse_oracle_line dispatches to the right parser; both produce the same flat field dict (PRIORITY, TIMESTAMP, HOSTNAME, APP_NAME, PROCID, STRUCTURED_DATA, ...) so the Then steps don't need to know which oracle is in use. The OTel parser re-renders the SD-ELEMENT back into RFC 5424 wire form via _render_otel_structured_data so the structured-data scenarios run cross-platform against the same assertions.

Local Windows BDD setup¶

Prerequisites: Python 3.13+ on PATH (winget install Python.Python.3.13), MSVC + vcpkg for the msvc-tunable-override build.

Shell: the recipe below uses bash syntax (background &, inline VAR=value command). Run it from Git Bash for Windows, which ships with the Git installer that's already required for this repo. PowerShell users either translate (Start-Process for backgrounding, $env:VAR = "..." for env vars) or invoke bash -c "...".

# 1. Install behave (pinned version matches the Linux container image)
pip install -r Bdd/requirements.txt

# 2. Download otelcol-contrib (pinned version, SHA-256 verified)
powershell -ExecutionPolicy Bypass -File Bdd/otel/Install-OtelCollector.ps1

# 3. Build the Windows example
cmake --preset msvc-tunable-override
cmake --build --preset msvc-tunable-override --target SolidSyslogBddTarget

# 4. Start the OTel oracle (binds 127.0.0.1:5514 udp+tcp, 6514 tls, 6515 mtls)
./Bdd/otel/bin/otelcol-contrib.exe --config=Bdd/otel/config.yaml &

# 5. Run the Windows-eligible scenarios
EXAMPLE_BINARY=build/msvc-tunable-override/Bdd/Targets/Debug/SolidSyslogBddTarget.exe \
RECEIVED_LOG=Bdd/output/received.jsonl \
ORACLE_FORMAT=otel-jsonl \
behave --tags='not @wip and not @windows_wip and not @no_rtc and not @aesgcm' Bdd/features/

If port 5514 is busy, stop the dev-container's syslog-ng: docker compose -f .devcontainer/docker-compose.yml stop syslog-ng-linux

Test isolation¶

Behave uses line-count tracking for test isolation. The Given step records the current line count of received.log, and the When step waits for a new line to appear before parsing it. This avoids restarting the syslog-ng container between scenarios.

Running BDD tests locally¶

From WSL or a host terminal (not inside the devcontainer):

# Start the Linux oracle
docker compose -f .devcontainer/docker-compose.yml up -d syslog-ng-linux

# Build the example binary (inside the gcc container)
docker compose -f .devcontainer/docker-compose.yml exec gcc \
    cmake --preset tunable-override-debug
docker compose -f .devcontainer/docker-compose.yml exec gcc \
    cmake --build --preset tunable-override-debug --target SolidSyslogBddTarget

# Run Behave
docker compose -f .devcontainer/docker-compose.yml run --rm behave-linux \
    behave Bdd/features/ --tags='not @wip and not @no_rtc'

For the QEMU targets (cross-build the ELF, then run Behave via QEMU inside freertos-target), see Bdd/README.md.

Verifying syslog-ng manually¶

To send a test message and check that syslog-ng is receiving and parsing correctly:

echo '<14>1 2009-03-23T00:00:00.000Z TestHost TestApp 1234 TestMsgId - Test message' \
    | nc -u -w1 localhost 5514

cat Bdd/output/received.log

If the file is not created, check that:

The syslog-ng container is running: docker compose -f .devcontainer/docker-compose.yml ps
The config is mounted correctly: docker compose -f .devcontainer/docker-compose.yml exec syslog-ng-linux cat /etc/syslog-ng/syslog-ng.conf
After editing syslog-ng.conf, reload the config, either via the control socket (see above) or by recreating the container: docker compose -f .devcontainer/docker-compose.yml rm -sf syslog-ng-linux && docker compose -f .devcontainer/docker-compose.yml up -d syslog-ng-linux