Transport Benchmarks

Transport benchmarks live under benchmarks/transport.

They measure the baseline communication surfaces directly, before timed-execution metadata is layered on top.

Public Installed-User Flow

Assume the normal public setup is already complete.

1. install Librux under /opt/librux
2. source /opt/librux/env.sh
3. run benchmark commands from /opt/librux

If you do not already have a control backend process running, these benchmark runners will start the installed control backend automatically. The equivalent manual development command is librux control --config "$LIBRUX_PREFIX/config.yaml" after sourcing /opt/librux/env.sh.

Run only one benchmark at a time on a host. Multiple benchmark runners share the same local control backend ports and can distort each other's measurements if started concurrently.

JSON benchmark reports include a top-level clock_status copied from the local control backend while the run is active. This is the self-contained source for time-sync acceptance fields such as correction_ns, correction_applied, threshold_kind, domain_compatible, domain_reason, and upstream_*. Per-process clock snapshots are retained as SDK-local timing state and may contain only the compact ABI fields.

For a full payload/subscriber matrix, keep the same reporting discipline as the public runners. Record the matrix plan, raw JSON reports, host clock status, and the Markdown summary together. Production cross-host one-way cases should be gated by the time-sync acceptance rules before payload/subscriber sweeps begin.

The communication matrix helper supports staged matrix profiles.

Profile	Use	Shape
smoke	quick install and regression sweep	small payload/count subset, fanout through N=16
coarse	trend check before long runs	payloads from bytes to MiB class, fanout through N=256
full	official ROS 2 comparison candidate	payloads 8 B-1 MiB, fanout through N=256
stress	high-fanout limit finding	8 B Event fanout at N=512 and N=1024

Advanced comparison matrix tooling lives in the SDK source checkout under tools/qa. Clone librux-sdk before using those matrix helpers.

Generate a plan before running long fanout cases.

python3 tools/qa/run_comms_matrix.py \
  --host-a <host-a> --host-b <host-b> \
  --matrix-profile full \
  --event-fanout --event-rtt --plan-only

The plan records case_plan and resource_plan.shm in the JSON and Markdown outputs. The official comparison envelope stops at N=256. N=512 and N=1024 are stress cases for capacity and scheduler sensitivity; keep those reports separate from accepted comparison tables. Review shm_slot_size, shm_slot_count, burst_bytes, and warnings against the hosts' /dev/shm size before running stress fanout. Large-payload fanout should be split into staged runs when the burst estimate exceeds the configured SHM budget.

Shared Matrix Runner

Use tools/qa/run_transport_matrix.py when the goal is to keep Librux and ROS 2 comparison cases aligned. The profile is a declarative matrix; the runner expands it into normalized cases and writes JSON, CSV, and Markdown plans under build/reports.

The comparison axes are explicit.

Surface	Axes	Meaning
Event	payload_size x topic_count x subscriber_count	publish/subscribe scaling
Control	packet_kind x payload_size x connection_count	request/response and real-time round-trip scaling

Plan the default smoke matrix without touching sandbox hosts.

python3 tools/qa/run_transport_matrix.py \
  --profile tools/qa/matrix_profiles/transport_comparison_smoke.yaml

Available shared profiles.

Profile file	Use
transport_comparison_smoke.yaml	setup validation and quick artifact check
transport_comparison_full.yaml	official-candidate matrix through Event N=256 and Control connection N=16
transport_comparison_stress.yaml	limit finding for Event N=512/1024 and high Control connection counts

Execute only after checking the generated plan.

python3 tools/qa/run_transport_matrix.py \
  --profile tools/qa/matrix_profiles/transport_comparison_smoke.yaml \
  --runtime librux \
  --surface event \
  --host-a <host-a> --host-b <host-b> \
  --execute

Unsupported cases remain in the report. This is intentional because ROS 2 service, ROS 2 action, ROS 2 timed-control reference, and concurrent Control connection adapters need dedicated runners before those axes can be accepted as official comparison data.

For Librux-only Event multi-topic sweeps, the lower-level matrix runner exposes the topic axis directly.

python3 tools/qa/run_comms_matrix.py \
  --host-a <host-a> --host-b <host-b> \
  --event-payloads 64,1024,4096 \
  --event-topic-counts 1,4,16 \
  --counts 1000 \
  --skip-control

Coverage

Benchmark	Purpose
transport/event	publish/subscribe latency and drop behavior
transport/control	Control request/response latency, timeout behavior, and priority contention

Reference Results

Reference values come from Ubuntu sandbox runs. Treat them as environment-specific sizing references; run the benchmark scripts on your own hosts for acceptance.

Primary evidence.

• build/reports/comms_matrix_fanout_startup_delay_full.md
• build/reports/comms_matrix_timeout_cache_payload_matrix.md
• accepted installed-user testing reports under reports/testing

Track	Mode	Condition	Reference result
Event 1:N fanout	same-host	64 B-4 KiB, N=1/2/4/8/16, 1000 Hz	lossless, subscriber mean 23.413-39.963 us
Event 1:N fanout	cross-host	64 B-4 KiB, N=1/2/4/8/16, 1000 Hz	lossless, subscriber mean 116.631-175.347 us
Event 1:N fanout stress	cross-host	8 B, single-process subscriber group, N=512, 1000 Hz	lossless in sandbox, subscriber mean about 541 ms
Event 1:N fanout stress	cross-host	8 B, single-process subscriber group, N=1024, 1000 Hz	partial completion in sandbox, 1008/1024 endpoints complete
Event RTT	same-host	64 B-64 KiB, 10000 samples	ping avg 43.125-115.346 us, failures 0
Event RTT	cross-host	64 B-64 KiB, 10000 samples	ping avg 239.061-479.578 us, failures 0
Control	same-host C++	8 B, 1 kHz	avg 82.174 us, p95 99.097 us, failures 0
Control	cross-host C++	8 B, 1 kHz	avg 383.970 us, p95 793.900 us, failures 0
Control	cross-host C++	1 MiB, about 30 Hz	avg 14.890 ms, p95 16.038 ms, failures 0

The Event fanout and Event RTT rows are the primary communication-scale references. The stress rows are limit-finding evidence, not official acceptance criteria. The Control rows remain representative control-path references until a full refreshed Control matrix is captured; keep them separate from strict acceptance criteria when reporting final numbers.

Cross-host Event fanout rows assume the benchmark readiness gate has already observed the subscription path and that startup guard traffic is excluded from the requested measured sample count. If a one-way cross-host smoke test starts publishing before the remote subscription path is applied, prefix loss should be reported as a warning rather than hidden.

The public Event tutorials include a gate helper for this sequence. Start the publisher with --start-gate-file, run tools/qa/fixtures/transport/event/python/wait_event_path.py against the publisher-side and subscriber-side control status URLs, and let the helper create the gate file only after remote_subscribe_topics and inbound_ready_topics contain the target topic. For cross-host one-way latency, also pass --require-time-sync --stable-polls 3 so the gate opens only after the control backend reports synchronized time for consecutive polls. A readiness-only gate proves delivery-path setup only; do not use its one-way latency numbers as accepted performance results.

If the gate helper output is piped through tee, run the shell with set -o pipefail, or pass --report-path and validate that the final JSON has "ready": true. Without that check, tee can mask a failed helper exit code.

Event

Path - benchmarks/transport/event

Quick validation command (Python-only, no extra compiler toolchain required).

python3 benchmarks/transport/event/run_benchmark.py --publisher-language python --subscriber-language python --count 20 --warmup 5

dropped in the event report is a measurement field. The benchmark runner exits successfully when the publisher and subscriber complete and reports the observed drop count for interpretation. For CI or release gates, add --fail-on-warning so nonzero dropped or too few measured subscriber samples returns a nonzero exit code after the report is written.

The local Event path uses a bounded per-publisher ring for Event topics instead of a single overwrite slot. Small payload topics use a default queue depth of 64, medium payloads use 16, and larger payloads use 4. Override this with --queue-depth in the benchmark runner or LIBRUX_EVENT_QUEUE_DEPTH when testing a specific workload. A nonzero dropped count remains a failure signal for strict validation; the queue reduces avoidable subscriber overrun but does not make Event a reliable control-command transport.

Native representative command.

python3 benchmarks/transport/event/run_benchmark.py --publisher-language cpp --subscriber-language cpp --count 100 --warmup 10

The top-level benchmark report includes result, warnings, and summary. Use --fail-on-warning when a warning should fail a CI or release validation run after the report is written.

publisher_slack is startup guard traffic. It increases the publisher's total messages but should not increase the requested measured sample count in the subscriber report. Extra measured samples are reported as warnings.

For high-frequency Event validation, make the queue depth explicit.

python3 benchmarks/transport/event/run_benchmark.py --publisher-language cpp --subscriber-language cpp --payload-size 8 --period-us 1000 --count 10000 --warmup 1000 --queue-depth 64 --fail-on-warning

Control

Path - benchmarks/transport/control

Quick validation command.

python3 benchmarks/transport/control/run_benchmark.py --controller-language python --gateway-language python --count 20 --warmup 5

Native representative command.

python3 benchmarks/transport/control/run_benchmark.py --controller-language cpp --gateway-language cpp --count 100 --warmup 10

The Control runner uses the same top-level result, warnings, summary, and --fail-on-warning convention as the event runner.

Priority contention scenario.

python3 benchmarks/transport/control/run_priority_benchmark.py --controller-language python --gateway-language python

If you want the C++ variants instead, install a compiler toolchain first on Ubuntu / Debian. Installed Librux Linux C++ SDK artifacts bundle yaml-cpp internally, so the normal installed-user benchmark path does not depend on the host libyaml-cpp.so.* version. libyaml-cpp-dev is required only when you rebuild the public C++ SDK source itself.

sudo apt update
sudo apt install -y build-essential libyaml-cpp-dev

Relationship To Tutorials

Use Add Component Subsystems when you want runnable communication samples in the cleaner robot tutorial. Use these benchmark runners when you want latency, throughput, or fairness data.