SBK

Storage Benchmark Kit

SBK architecture and code flow

This guide explains how SBK is assembled and how one benchmark command moves through the code. It is intended for software engineers, reviewers, operators, and coding agents. For algorithm-level detail, continue with sbk-internals.md.

System context

SBK separates workload control from storage-specific I/O. The same harness can drive a local file, an S3 endpoint, a message broker, a database, or an in-memory queue because all backends implement the same Storage<T> contract.

flowchart TB
    USER[User, script, or YML file]
    SBK[SBK single-node launcher]
    DRIVER[Storage driver]
    BACKEND[Storage system]
    PERL[PerL measurement engine]
    LOGGER[Output logger]
    SBM[SBM distributed aggregator]
    GEM[SBK-GEM SSH orchestrator]

    USER --> SBK
    GEM --> SBK
    SBK --> DRIVER
    DRIVER --> BACKEND
    SBK --> PERL
    PERL --> LOGGER
    LOGGER -->|GrpcLogger| SBM
    GEM --> SBM

Build-time module boundaries

flowchart LR
    PERL[perl]
    API[sbk-api]
    DRIVERS[drivers projects]
    ROOT[root distribution]
    SBM[sbm]
    GEM[sbk-gem]
    YAL[sbk-yal]
    GYAL[sbk-gem-yal]

    API --> PERL
    DRIVERS --> API
    ROOT --> API
    ROOT --> DRIVERS
    SBM --> API
    GEM --> SBM
    YAL --> API
    GYAL --> GEM
Module Owns Does not own
perl Timestamp abstraction, concurrent queues, latency windows, percentiles, periodic/total recording Storage semantics or CLI driver configuration
sbk-api Storage/logger SPIs, bootstrap, common CLI, worker orchestration, payload types Vendor SDK calls
drivers/* Backend configuration and I/O adaptation Benchmark scheduling or general percentile computation
sbm SBP/gRPC ingestion and multi-client aggregation Remote process launch
sbk-gem SSH connections, remote distribution/launch, embedded SBM lifecycle Driver implementation
sbk-yal Mapping a YML document to SBK arguments A separate benchmark engine
sbk-gem-yal Mapping YML to SBK-GEM arguments A separate distributed protocol

Single-node bootstrap

The installed script starts io.sbk.main.SbkMain, configured by the Gradle application plugin.

sequenceDiagram
    participant Main as SbkMain
    participant Boot as Sbk
    participant Scan as Package scanners
    participant Driver as Storage driver
    participant Logger as RWLogger
    participant Bench as SbkBenchmark

    Main->>Boot: run(args)
    Boot->>Scan: scan driver and logger packages
    Scan-->>Boot: matching classes
    Boot->>Driver: construct selected -class
    Boot->>Logger: construct selected -out
    Boot->>Driver: addArgs(parameters)
    Boot->>Logger: addArgs(parameters)
    Boot->>Boot: parse and validate merged CLI
    Boot->>Driver: parseArgs(parameters)
    Boot->>Logger: parseArgs(parameters)
    Boot-->>Main: new SbkBenchmark(...)
    Main->>Bench: start()

Important details:

Primary sources:

Benchmark lifecycle

SbkBenchmark is the lifecycle owner. Its states prevent a benchmark from being started or stopped twice.

stateDiagram-v2
    [*] --> BEGIN
    BEGIN --> RUN: start
    RUN --> END: workers finish
    RUN --> END: duration timeout
    RUN --> END: shutdown or error
    END --> [*]

At start() it:

  1. Opens the selected RWLogger.
  2. Calls Storage.openStorage() once for shared driver resources.
  3. Calls createWriter(id, params) and createReader(id, params) for configured workers.
  4. Wraps driver objects in SbkWriter and SbkReader harness workers.
  5. Starts write/read PerL recorders where the selected action needs them.
  6. Distributes count-based records across workers, preserving the remainder on the last worker.
  7. Starts workers in configured steps and optionally delays between steps.
  8. Schedules stop() for timed runs and also chains shutdown after worker completion.

At shutdown it stops new work, closes driver readers and writers, closes the storage and logger, shuts down executors, and completes the benchmark future. Duration-based shutdown can interrupt an SDK call, so drivers must tolerate interruption-related exceptions during normal teardown.

Primary source: sbk-api/src/main/java/io/sbk/api/impl/SbkBenchmark.java.

Operation path and timing

The driver-facing types are deliberately layered:

Synchronous write

sequenceDiagram
    participant Worker as SbkWriter
    participant Writer as Driver Writer
    participant Time as Time
    participant Channel as PerlChannel

    Worker->>Writer: recordWrite(...)
    Writer->>Time: getCurrentTime()
    Writer->>Writer: writeAsync(payload)
    Note over Writer: returns null for synchronous completion
    Writer->>Time: getCurrentTime()
    Writer->>Channel: send(start, end, records, bytes)

Asynchronous write

When writeAsync() returns a CompletableFuture, the default implementation records completion time in the future callback. Exceptional completion is passed to the PerL exception handler. A driver must not report a future as complete before the storage operation has reached the completion semantics promised by that driver.

Read

Reader.read() returns one payload, while default reader methods time the operation, derive the byte count through DataType<T>, and submit the result. Callback or batch-oriented backends can implement the more specialized reader abstractions.

The harness already measures calls. A driver should only override timing helpers when its backend semantics require it, such as measuring a batch, embedding a producer timestamp, or avoiding an unavoidable adapter copy.

Primary sources:

PerL measurement pipeline

PerL decouples measurement ingestion from statistics calculation.

flowchart LR
    W1[Worker 1] --> C[PerlChannel]
    W2[Worker 2] --> C
    WN[Worker N] --> C
    C --> Q[Selected timestamp queue array]
    Q --> R[Performance recorder]
    R --> P[Periodic window]
    R --> T[Total window]
    P --> L[RWLogger]
    T --> L

Each measurement contains start time, end time, record count, and byte count. PerL records all submitted operations rather than sampling them. The recorder drains concurrent queues and updates latency storage and counters away from the I/O worker.

By default, the selected array contains intrusive TimeStampMpscQueue instances. Each submitted TimeStampNode is both the measurement payload and its linked-queue node, so enqueue does not allocate a second wrapper. -mpscqueue false selects the compatibility path based on JDK ConcurrentLinkedQueue<TimeStamp>; -mpscqueue true selects the intrusive path. The default comes from MpscQueueEnable in sbk-api/src/main/resources/sbk.properties.

Queue topology is a separate concern. qPerWorker and maxQs remain property-backed settings rather than public CLI options. SBK prints both the effective queue implementation and topology after argument parsing, before the benchmark starts.

The timestamp queues have a multiple-producer, single-consumer workload: worker threads produce measurements and the PerL recorder consumes them. TimeStampMpscQueue specializes for that contract; the JDK fallback retains general MPMC Collection behavior. See the queue research guide for linearization, memory-ordering, reclamation, complexity, and benchmark evidence.

TimeStampMpscQueue uses one single-use TimeStampNode as both payload and link. Producers publish through a CAS on the predecessor’s next reference; the single consumer owns head. Every 16 dequeues, the consumer release-publishes a recovery head and self-links the retired predecessor batch. A producer paused on an old node detects that self-link and resumes from the recovery head. This bounds stale-chain retention without pooling nodes or adding a consumer-side head CAS. The specialization is appropriate only for PerL’s many-producer/one-consumer hand-off; it is not intended for multiple consumers, iterators, arbitrary removals, or general collection use.

The default recorder also avoids querying the clock on every queue scan. While records are available it reuses TimeStamp.endTime. While all queues are empty, ElasticWait parks and checks the clock only after an adaptively calibrated batch. The learned parks-per-millisecond rate is retained in an exponential moving average. When activity briefly interrupts idleness, the first subsequent empty scan starts a clean idle sample from the last record timestamp while retaining that learned rate. This prevents active time from diluting calibration and adds no new clock read. Setting sleepMS > 0 selects the simpler sleeping recorder and bypasses ElasticWait.

The phrase “lock-free hot path” applies to harness measurement transport. It does not promise that a vendor SDK, filesystem, JVM scheduler, allocator, or backend is lock-free. Driver wrappers should avoid adding their own locks to the per-operation path.

Primary sources:

Output boundary

RWLogger is both a lifecycle interface and a metrics sink. Implementations can print, write CSV, expose Prometheus metrics, or forward data over gRPC.

Logger Use
SystemLogger Default terminal output
Sl4jLogger SLF4J logging path
CSVLogger CSV result persistence
WebLogger Console/CSV behavior plus dependency-free local live graphs; see WebLogger guide
PrometheusLogger Metrics endpoint plus inherited result behavior
GrpcLogger SBP/gRPC forwarding to SBM

Logger discovery follows the same class-scanning pattern as drivers. New loggers belong under io.sbk.logger and must implement RWLogger, usually by extending the existing abstract implementation.

Distributed flow

SBM aggregates measurements; it does not generate storage load. SBK-GEM orchestrates load generators; it does not replace the driver or the single-node harness.

sequenceDiagram
    participant Operator
    participant GEM as SBK-GEM
    participant SBM
    participant HostA as Remote SBK A
    participant HostB as Remote SBK B
    participant Storage

    Operator->>GEM: start connections and SBK arguments
    GEM->>SBM: start embedded aggregator
    GEM->>HostA: reconcile Java and SBK, export SBK_JAVA_HOME, start SBK
    GEM->>HostB: reconcile Java and SBK, export SBK_JAVA_HOME, start SBK
    HostA->>Storage: driver operations
    HostB->>Storage: driver operations
    HostA->>SBM: GrpcLogger measurements
    HostB->>SBM: GrpcLogger measurements
    SBM-->>Operator: aggregate windows and totals

SBP messages and gRPC services are generated from protobuf definitions in sbk-api/src/main/proto. SBM receives client registrations and latency records through SbmGrpcService, queues them in SbmLatencyBenchmark, and merges them into periodic and total windows. The default gRPC port is 9717.

SBK-GEM uses Apache MINA SSHD for connection, copy, and command execution. It supports passwordless public-key authentication through the launching user’s SSH agent and OpenSSH-configured identity files; an explicitly configured password is an optional fallback. Server identities are verified against that user’s ~/.ssh/known_hosts, or the path selected by -knownhosts, so unknown or changed host keys fail before deployment. Strict checking is the default; -hostkeycheck false is an explicit, insecure opt-out for isolated environments. Before launch it checks the requested Java major version and exact SBK version on every host. It can reuse Java from PATH or a configured Java home, or copy its local JVM when provisioning is enabled. Each launch exports the verified node-specific SBK_JAVA_HOME. With copy=true (the default), SBK is copied only to missing or mismatched targets. delete=true removes a mismatched installation before replacement, while deleteafter=false preserves the verified deployment after the run. GEM verifies copied versions, resolves each executable to a node-specific absolute path, and constructs remote SBK arguments with GrpcLogger pointing back to its embedded SBM instance.

Configuration layers

Configuration comes from several layers:

  1. Gradle/application defaults establish program name, main class, and application home.
  2. Resource property files provide harness, logger, SBM, GEM, and driver defaults.
  3. Common CLI options are registered by SbkParameters.
  4. The selected logger and driver add their CLI options.
  5. Parsed CLI values override applicable defaults.
  6. YAL variants translate YML entries into the same argument model; they do not bypass normal validation.

For the PerL transport, MpscQueueEnable supplies the default and the common -mpscqueue true|false option overrides it for that benchmark. The topology properties qPerWorker and maxQs are validated when SbkParameters loads sbk.properties, but are not exposed as command-line options. This keeps an A/B queue comparison to one explicit runtime switch while preventing accidental topology changes between runs.

Use generated -help output as the authority for accepted options. Use the relevant resource property file as the authority for defaults that are not printed in help.

Packaging and class loading

The root distribution includes sbk-api and all drivers declared as API dependencies in build-drivers.gradle. Drivers also must be included in settings-drivers.gradle so Gradle creates their projects.

The generated launcher uses a pathing JAR whose manifest points at runtime dependencies. After changing dependencies, force regeneration:

./gradlew clean :pathingJar installDist --rerun-tasks

A driver can compile successfully yet be unavailable at runtime if either registration file is missing or the distribution/pathing JAR is stale.

Safe extension boundaries

Desired change Primary location
Support a backend drivers/<name>/ plus both driver registration files
Add a workload-wide CLI option SbkParameters and ParameterOptions contracts
Add a payload representation io.sbk.data implementation
Add result output io.sbk.logger implementation
Change latency storage or percentile behavior perl
Change timestamp queue selection or topology perl, SbkParameters, and SbkBenchmark
Change distributed aggregation sbm and protobuf compatibility review
Change remote launch sbk-gem
Change YML mapping the applicable YAL module

Crossing these boundaries should be an explicit design decision. In particular, do not put vendor behavior into sbk-api or generic measurement behavior into a driver.

Failure propagation

When diagnosing a failure, identify the boundary first: argument discovery, distribution/classpath, driver open, worker I/O, measurement recorder, logger, gRPC aggregation, or SSH orchestration.

Source reading order

For a practical code walkthrough:

  1. sbk-api/src/main/java/io/sbk/main/SbkMain.java
  2. sbk-api/src/main/java/io/sbk/api/impl/Sbk.java
  3. sbk-api/src/main/java/io/sbk/api/Storage.java
  4. One simple driver, such as drivers/file/
  5. sbk-api/src/main/java/io/sbk/api/impl/SbkBenchmark.java
  6. SbkWriter, SbkReader, Writer, and Reader
  7. perl/src/main/java/io/perl/api/impl/PerlBuilder.java
  8. The PerL queue and recorder selected by that builder
  9. drivers/perlbench/ and the queue research guide for a controlled end-to-end queue comparison
  10. GrpcLogger and SbmGrpcService for distributed reporting
  11. SbkGem and SbkGemBenchmark for remote orchestration

Architectural invariants

Review changes against these invariants: