Storage Benchmark Kit
SBK-GEM runs one SBK workload across multiple hosts. It uses Apache MINA SSHD to connect to remote machines, reconcile the expected SBK distribution on every host, execute the same benchmark arguments, and aggregate client measurements through an embedded SBM instance.
flowchart TB
OP[Operator] --> GEM[SBK-GEM]
GEM --> SBM[Embedded SBM]
GEM -->|SSH, verified runtime bundle| A[Remote host A / SBK plus Java]
GEM -->|SSH, verified runtime bundle| B[Remote host B / SBK plus Java]
A --> STORAGE[Target storage]
B --> STORAGE
A -->|GrpcLogger| SBM
B -->|GrpcLogger| SBM
SBK-GEM owns remote launch and aggregate lifecycle. The remote SBK processes still own driver discovery, workload scheduling, and storage I/O. SBM owns aggregation.
-fullcopy false, the default) or the full controller JDK
(-fullcopy true) as a separate content-addressed tree. Java is never
embedded in the SBK runtime archive.known_hosts file is needed only when host-key checking is explicitly enabled.9717. By default,
SBK-GEM advertises the numeric controller address selected by each authenticated SSH route, avoiding a dependency
on remote DNS. Use -localhost <address> only when an explicit shared callback address is required (for example,
through NAT or a load balancer).Linux or
macOS). CPU architecture is not part of deployment compatibility. Windows
and mixed Linux/macOS runs are rejected.scp and tar executables are required for bulk transfer
and first-time Java extraction. No checksum utility or generated shell script
is required; the packaged Java agent performs SBK extraction, verification,
and launch through Java APIs.Use dedicated benchmark hosts and least-privilege SSH credentials. Do not put passwords or private keys in committed files.
Passwordless public-key login is the recommended configuration. Run SBK-GEM as
the same local account whose SSH configuration and credentials can reach the
remote benchmark account. SBK-GEM can use keys exposed by the local SSH agent
(SSH_AUTH_SOCK) and identity files selected by the local OpenSSH configuration,
including the conventional files under ~/.ssh. An agent is the preferred way
to use passphrase-protected keys because their passphrases do not have to be put
in SBK configuration.
The -gempass option and SBK_GEM_SSH_PASSWD environment variable are optional
password-authentication inputs. When a password is supplied, SBK-GEM asks Apache
MINA to try password authentication first and then falls back to the SSH agent
and configured key files if the server rejects the password. A supplied password
also disables SSH host-key checking, so stale or changed known_hosts entries do
not prevent login. Leave both unset for passwordless login. Do not store gempass
in a committed YML or properties file.
Remote host-key checking is disabled by default so unattended passwordless SSH
can use the configured SSH agent or key files without known_hosts blocking the
connection. Set -hostkeycheck true to enable accept-new checking against the
local user’s OpenSSH ~/.ssh/known_hosts data, or use -knownhosts <path> to
select a dedicated trust file. A previously unknown host key is then accepted
and persisted on first contact (TOFU/accept-new), and a changed key is rejected.
Host-key checking is bypassed whenever -gempass or SBK_GEM_SSH_PASSWD
supplies a password, regardless of -hostkeycheck.
Verify newly recorded fingerprints through an independent channel when the
environment requires stronger first-contact assurance. -hostkeycheck false
is an explicit opt-out for isolated, disposable environments and also disables
changed-key protection. A successful command-line ssh connection is a useful
preflight check, but run it as the exact operating system user that will launch
SBK-GEM so it reads the same agent, key files, SSH configuration, and
known_hosts file.
./gradlew :sbk-gem:check
./gradlew installDist
./gradlew generateSbkCompactJavaRuntime
Use the root distribution because it contains the complete enabled-driver dependency graph that GEM packages for remote SBK processes. The third command is optional and demonstrates the standalone compact Java image build; SBK-GEM generates and caches the same image automatically when compact copying is used.
Display the current connection, remote-installation, SBM, and benchmark options:
./build/install/sbk/bin/sbk-gem -help
SBK-GEM accepts GEM-specific options and forwards an SBK argument set to remote processes. Because authentication and connection-file formats are security-sensitive and evolve independently of a sample environment, use generated help and the checked-in example configuration files as the authority.
-idletimeoutseconds N is a shared lifecycle option with a default of 600
seconds. GEM forwards it to every remote SBK process and configures its embedded
SBM with the same value. The deadline is enabled only for fixed-record runs
(-records or fixed-mode -totalrecords) and is disabled whenever -seconds
is used. Each positive performance result restarts the full interval. A remote
PerL idle failure therefore terminates that SBK process and becomes a
host-tagged GEM failure; an embedded SBM idle failure also terminates GEM
immediately rather than leaving fixed-record orchestration waiting indefinitely.
The value must be strictly greater than the reporting interval used by the
remote SBK loggers and embedded SBM logger.
The final SBK-GEM lifecycle message reports successful -seconds or -records
mode completion, an explicit stop, an idle-timeout exit, or an internal remote,
SSH, cleanup, or embedded-SBM exception. When several failures occur, the first
failure remains authoritative and later failures are retained as suppressed
causes.
-gemport remains the default SSH port for every entry in -nodes. A node can
override that default by using a host:port endpoint. This supports multiple
SSH nodes exposed through different ports on one host, including disposable
Docker fixtures:
sbk-gem -nodes 127.0.0.1:2201,127.0.0.1:2202 -gemport 22 \
-class file -writers 1 -records 1000 -size 4096
Use brackets when an IPv6 address includes a port, for example
-nodes [::1]:2201. Hostnames and addresses without an endpoint-specific port
continue to use -gemport.
Use -records to preserve the standard SBK behavior: the supplied value is
forwarded to every remote SBK client. Use the GEM-only -totalrecords option
when the value must apply to the distributed benchmark as a whole:
# Execute exactly 1,000,000 records across all configured nodes.
sbk-gem -nodes node-a,node-b -class file -writers 4 -size 4096 \
-totalrecords 1000000
# Limit the aggregate write rate to exactly 120,000 records/second for 60 seconds.
sbk-gem -nodes node-a,node-b -class file -writers 4 -size 4096 \
-totalrecords 120000 -seconds 60
Without -seconds, GEM divides the fixed count across nodes and assigns any
remainder to the first nodes. With -seconds, it divides the aggregate
records/second rate in whole per-worker units so the requested total remains
exact. The rate must therefore be divisible by the active worker count and
must provide at least one record/second to every active worker on every node.
For mixed writer/reader runs, the aggregate rate applies independently to each
direction when writer and reader counts are equal, matching SBK’s shared
per-worker rate model. Timed aggregate rates reject unequal mixed worker counts.
-totalrecords is mutually exclusive with -records and -throughput.
Use -throughput to apply the supplied MB/s limit independently to every
remote SBK client. Use the GEM-only -totalthroughput option when one MB/s
limit must apply to the distributed benchmark as a whole:
# Limit two remote clients to 200 MB/s in aggregate for 60 seconds.
sbk-gem -nodes node-a,node-b -class file -writers 4 -size 4096 \
-totalthroughput 200 -seconds 60
# Execute exactly 1,000,001 aggregate records at 200 MB/s aggregate throughput.
sbk-gem -nodes node-a,node-b -class file -writers 4 -size 4096 \
-totalrecords 1000001 -totalthroughput 200
GEM divides the aggregate throughput across the remote clients and forwards a
node-specific -throughput value to each one. Decimal division retains twelve
fractional digits and assigns any remainder to the first client, so the
forwarded values sum to the requested aggregate value. Each allocation must
provide at least one record/second to every active worker after SBK converts
MB/s to its per-worker record rate.
-totalthroughput is mutually exclusive with -throughput. It may be combined
with fixed -records or fixed -totalrecords. It must not be combined with
timed -totalrecords, because in that mode -totalrecords already means an
aggregate records/second limit and the two options would define competing
rates. The option is also supported by SBK-GEM-YAL as totalthroughput.
Mixed writer/reader runs require equal writer and reader counts because SBK
uses one shared per-worker rate for both directions.
Before connecting to storage, SBK-GEM validates the local installDist
layout, including the pathing JAR and every dependency named by its manifest.
Every standard Gradle build, installDist, and distTar output remains a
complete SBK distribution. The independent generateSbkDriverRuntimes task
also writes one dependency manifest and pathing JAR per enabled driver under
worker-runtime/. These files describe the selected driver’s transitive
runtime closure without replacing or removing any normal launcher, library,
manifest, or documentation file from the complete distribution.
Gradle also exposes generateSbkCompactJavaRuntime as an independent build
task. It creates build/generated/sbk-compact-java-runtime directly from the
versioned module/options contract in
gradle/sbk-compact-java-runtime.properties; it does not require SBK-GEM or
change the contents of installDist and distTar. The standard distribution
packages only the small descriptor consumed by deployment tools, keeping the
customer tar portable rather than embedding a controller-platform Java image.
By default, -fullcopy false reduces both payloads. The remote SBK archive
uses the Gradle-generated closure for the selected -class, and missing remote
Java is supplied as a jlink runtime image generated from the versioned
worker-runtime/java-runtime.properties contract. For example, a File run
transfers SBK core, the File driver, and their runtime dependencies; a later
RocksDB run selects a different content identity containing the RocksDB
driver and its native dependency. Exact Java and SBK identities are still
reused independently. With packagescleanup=false, multiple inactive driver
identities may remain cached for fast switching; the default cleanup policy
retains only the current SBK identity after its leases become inactive.
Set -fullcopy true to provision the full controller JDK and complete SBK
distribution instead.
The Gradle distribution records an identity covering its runtime JARs,
launchers, Java bootstrap files, and remote agent. SBK-GEM packages the
runtime-only bin, lib, and identity files into a content-addressed plain
tar archive. When remote Java provisioning is required, the selected full or
compact Java tree is copied separately with its own content identity. Each identity covers every file,
contained relative symbolic link, and normalized file mode, so two builds
carrying the same SBK version but different dependencies cannot be mistaken
for one another. Links escaping the SBK or JDK source tree are rejected;
directory modes are normalized to 0755 in both the identity and archive.
The archive contains a platform descriptor and per-file SHA-256 manifest.
Every remote node must pass the homogeneous operating-system preflight. GEM then
automatically uploads only a missing content identity, verifies the archive SHA-256, extracts
to a unique staging directory, verifies every regular file, and atomically
renames the verified runtime into place. A partially copied or failed staging
directory is never used to launch SBK. Subsequent runs reuse the exact verified
content without copying it again. Local archives are cached under
~/.sbk/cache/sbk-gem by default; runtimeCacheDirectory in
gem.properties changes that location. A per-identity file lock serializes
cache writers across GEM processes, and a separately published SHA-256 sidecar
plus archive-size sidecar causes incomplete cached archives to be rebuilt before
use. An unchanged build identity reuses both sidecars without rehashing the
installed distribution or cached archive. The remote agent still verifies the
complete transferred archive; a digest mismatch rebuilds the local archive and
retries the affected transfer once. With
packagescleanup=true, the controller retains only the selected cached bundle;
a non-current archive being transferred by another GEM process is protected by
its cache lock and is removed after it becomes inactive.
Creating a new bundle hashes the runtime files once and calculates the archive
SHA-256 while writing the uncompressed tar, avoiding separate compression and
archive-hashing passes. The independently managed Java tree is hashed, including
executable/POSIX permission state, and copied as one cached tar file only when
its exact usable identity is unavailable remotely. Full mode requires both
bin/java and bin/javac; compact mode generates a runtime-only image and
requires bin/java. A matching marker with unusable required executables is
retired and repaired instead of being reused. Physical deployment work is grouped by SSH user, authenticated network
endpoint, port, and the resolved case-sensitive remote path. Host aliases and
multiple logical clients sharing that target therefore install the agent and
copy Java/SBK only once.
First-time Java deployment creates or reuses one cached plain-tar archive and sends
it through a single Apache MINA SCP stream per active target. The standard remote
tar executable extracts it into an atomic staging directory. SBK runtime archives
use the same single-file bulk transport and are extracted by the Java agent. SFTP
remains limited to remote-directory resolution and atomic installation of the
small Java agent; typed agent requests perform runtime lifecycle operations. JDK and SBK
deployment progress reports transferred bytes, percentage, MiB/s, and ETA;
after payload completion it explicitly reports remote metadata finalization.
During these potentially long disk-intensive steps, GEM emits an elapsed-time
heartbeat every 5 seconds by default. Runtime
bulk SCP copies also report the archive size, unique transfer-target count,
completed targets, and hosts still pending. Runtime lease acquisition and inactive-runtime
retirement use the same per-host progress reporting. Configure the shared bounded
update interval with runtimeProgressIntervalSeconds in gem.properties;
cached bundles and fast local-network copies normally complete without a heartbeat.
Controller execution resources remain bounded as the node inventory grows. SSH
connections and control-plane operations use a fixed-size platform-thread pool,
while JDK and SBK data movement uses a smaller independent transfer pool so a
large copy cannot starve probes, leases, or shutdown work. Remote commands that
remain open for the complete benchmark use lightweight Java virtual threads
instead of one platform thread per node. The pool limits are owned by
controlExecutorThreads and transferExecutorThreads in gem.properties.
Each bulk SCP stream uses the separately configurable sshCopyBufferBytes
read buffer (4 MiB by default); this affects controller memory per active
transfer but does not change the number of parallel transfers.
When the controller is also a selected SSH host, its deployment parent may be
the local SBK distribution directory. GEM reserves sbk-runtime-* and
.sbk-runtime-* top-level names for deployment state and excludes those trees
from bundle input. This prevents old managed SBK trees, leases, locks, and
transfer staging files from changing or recursively expanding the next bundle.
The deployment lifecycle is automatic: exact verified SBK and Java identities
are reused independently, while missing content is uploaded, verified, and
activated without a separate copy switch. The current identities are retained.
Remote-agent leases protect runtimes used by concurrent GEM executions. The
controller sends one typed Java-agent request per host to reserve, acquire,
refresh, or release a lease; the agent performs the corresponding filesystem
work locally on that host. A reservation is created before probe/copy/activation
and refreshed during the benchmark, so overlapping GEM processes cannot retire an
identity another process is preparing or using. Therefore
an active non-current identity may coexist temporarily and is removed after its
final lease exits. Cleanup compares immutable identities, not version numbers,
so both lower and higher inactive SBK versions are removed. An inactive runtime
is first atomically renamed out of the managed sbk-runtime-* namespace while
the lifecycle lock is held. Its potentially large directory tree is then
deleted by the remote Java agent after leases are released, so recursive deletion cannot
hold the lifecycle lock or block lease acquisition and remote benchmark startup.
No login-shell program, zsh construct, nohup, PID probe, shell glob, remote
archive command, or checksum command participates in deployment. Apache MINA
SFTP resolves paths and verifies or installs the small Java agent in one
operation per physical target. SBK-GEM hashes the local agent once and starts
each target’s Java probe immediately after that target is ready, while bounded
progress logs identify any hosts still bootstrapping. The agent performs
lifecycle metadata updates, verification, archive activation, cleanup, and
benchmark process launch through local Java filesystem APIs.
Every SSH/SFTP operation has a bounded deadline. A timeout actively interrupts its worker and closes the operation-owned MINA SFTP filesystem; it does not merely mark a future failed while transfer work continues in the background. Startup holds the lifecycle lock only while changing state, so Ctrl+C or an internal failure can cancel connection, deployment, or verification work without waiting for the complete startup sequence. Shutdown then releases any unlaunched lease it can still reach, closes all SSH sessions, stops the embedded SBM, and forcibly terminates surviving remote-agent child processes after the grace interval.
The generated bin/sbk-gem and bin/sbk-gem-yal launchers select their complete
local SBK installation through the internal sbk.appHome system property.
SBK-GEM validates that standard installation contract and the remote agent
launches SbkMain directly from the verified JARs. The distribution cannot be
redirected through a command-line or YAML option.
The deployment lifecycle option is:
-packagescleanup true|false removes every inactive non-current remote
SBK-GEM-managed runtime identity and controller-side cached bundle after
verified activation and lease/transfer release, regardless of whether its
SBK version is lower or higher; the default is
true. It never deletes the current identity, a live leased identity, an
unmanaged directory, or a user-managed JDK selected with -javadir. Cleanup
is independent of -fullcopy: a minimal run removes inactive old full
distributions and inactive runtimes for other drivers, while a full run
removes inactive full and driver-scoped runtimes.-fullcopy true|false provisions the complete controller JDK and complete
SBK distribution when true. The default, false, selects both the compact
Gradle Java-runtime contract and the driver-specific SBK runtime closure.
The locally built installDist and distTar artifacts remain complete and
independently usable at customer sites in either mode.The rule applies to every deployment target in -nodes, including the
controller host when it is selected as a node. It does not scan or delete
arbitrary SBK/JDK installations outside the SBK-GEM-managed deployment parent;
doing so would risk deleting user-owned software.
The former -copy, -delete, -deleteafter, -sbkcommand, -sbkdir,
-javacopy, -javaversion, -copyonlydrivers, -compactruntimecopy, and
-compactcopy options are rejected with
migration guidance. This prevents disabling required provisioning, deleting
the newly verified runtime at benchmark shutdown, or bypassing the verified
standard launcher contract.
SBK-GEM uses the controller’s Java major version as the minimum remote Java
version. -javadir <home> optionally identifies a preferred remote JDK. Otherwise
GEM asks the remote Java agent to validate Java discovered from PATH.
If the preferred or PATH Java is absent or older than the controller Java, GEM
copies Java separately as one cached tar through Apache MINA SCP. In default
mode, -fullcopy false, GEM generates or reuses a runtime-only image with
the controller JDK’s jlink; the selected
module and option contract comes from Gradle’s
gradle/sbk-compact-java-runtime.properties and the packaged
worker-runtime/java-runtime.properties descriptor. Java has its own content
identity and reuse marker, so an unchanged image is not copied again.
The SBK archive likewise has an independent content identity and is transferred
only when its exact content is absent.
Use -fullcopy true when remote deployment specifically requires the
complete controller JDK and complete SBK package.
For each remote launch, the agent starts io.sbk.main.SbkMain directly with
the selected JDK and the verified SBK pathing/main JARs. The deployed shell
launcher is not executed. A newer remote JDK is accepted because it can execute
SBK classes built for the controller’s Java release. The agent passes the
verified home to remote SBK as both SBK_JAVA_HOME and JAVA_HOME, marks its
source as SBK_GEM_REMOTE_JDK, and invokes the absolute bin/java path.
Before a multi-host run:
SbkGemMain delegates to SbkGem.GrpcLogger and SBM port arguments. After SSH authentication, each node receives the
numeric controller callback address selected by its SSH route; an explicit -localhost value overrides this.-totalrecords and -totalthroughput when requested, creating node-specific -records and -throughput argument lists.SbmBenchmark and SbkGemBenchmark.SbkGemBenchmark establishes SSH sessions, enforces homogeneous platform
compatibility, and verifies or atomically deploys the exact SBK/Java runtime bundle.SBK-GEM treats remote execution and SBM client registration as one distributed result. The final summary on the GEM/SBM host reports the expected node count, successful and failed node counts, the maximum number of SBK clients registered with SBM, and one terminal status for every configured host:
SUCCESS – the remote SBK process completed with exit code zero.EXIT_FAILURE – remote SBK started but returned a non-zero exit code.SSH_ERROR – SSH connection, authentication, command startup, or transport failed.TIMEOUT – remote execution exceeded its configured deadline.CANCELLED – the asynchronous remote operation was cancelled.NOT_COMPLETED – no terminal result was available for the configured host.If any host fails, or fewer SBK clients register with SBM than expected, GEM
labels the distributed run FAILED or INCOMPLETE, logs that its performance
results are invalid for comparison, aborts any clients waiting at the
coordinated-start barrier, and exits non-zero. One failed future cannot hide the
outcomes of the other nodes: GEM waits for and reports every configured host.
The registration barrier also has a separate deadline, so a remote process that
never reaches SBM cannot leave the other clients waiting indefinitely. Configure
sbmRegistrationTimeoutSeconds in gem.properties for slow JVM or storage-driver
startup; its default is 120 seconds. The independent remoteTimeoutSeconds
setting remains the timeout for individual SSH control operations.
Remote driver initialization is deliberately outside SBM’s reporting clock. This two-phase ready-then-release barrier prevents a short benchmark from running before SBM is ready. The remote SBK processes and SBM use independent periodic windows, so the first aggregate result can arrive within two reporting intervals after the prepared-client barrier; subsequent aggregate windows use the normal configured interval.
For diagnostics, GEM retains only the most recent 256 KiB of each remote process’s stdout and stderr. Failed hosts include these bounded tails in the SBM-host log. This preserves the relevant exception and shutdown messages without allowing a noisy remote process to consume unbounded GEM heap.
| Class | Responsibility |
|---|---|
io.gem.main.SbkGemMain |
Executable entry point |
io.gem.api.impl.SbkGem |
Discovery, argument parsing, remote-command construction |
SbkGemBenchmark |
Remote sessions and embedded-SBM lifecycle |
SshSession |
One SSH connection/session abstraction |
SshUtils |
SSH and file-transfer helpers |
ConnectionConfig |
Remote connection model |
SbkRuntimeBundle |
Content-addressed SBK archive creation, identity, and local cache management |
ManagedJavaRuntime |
Controller-JDK identity, archive, transfer, extraction, and reuse planning |
RemoteAgentFiles |
Atomic installation of the packaged Java agent through SFTP |
RemoteAgent and RemoteAgentProtocol |
Bounded typed requests and responses exchanged with the remote Java agent |
SbkGemRemoteAgentMain |
Remote probe, SBK activation/verification, runtime leases, cleanup, and process launch |
RemoteTargetPlan |
Deduplication of physical deployment work shared by logical nodes |
GemPrometheusLogger |
GEM/SBM aggregate metrics output |
GemWebLogger |
GEM adapter for the embedded SBM local live web console |
Select -out GemPrometheusLogger to publish the embedded SBM aggregate at
http://<sbk-gem-host>:9719/metrics. Remote SBK processes continue using GrpcLogger; register only the coordinator
endpoint in the separately deployed SBK Dashboard. The endpoint uses
component="sbm" because the embedded SBM owns aggregation and metrics. See the
PrometheusLogger and SBK Dashboard guide for dashboard setup, Compose networking,
registration, tags, retention, and troubleshooting.
Select -out GemWebLogger for dependency-free aggregate graphs. The web console uses plain HTTP and listens on all
IPv4 interfaces at port 9720 by default. At benchmark start and completion, SBK-GEM prints run URLs for localhost,
loopback, hostname, and every usable host IPv4 address. Remote SBK processes still use GrpcLogger; the embedded SBM publishes the
combined cluster result to the Local Web Console. Web console
options are listed by sbk-gem -out GemWebLogger -help and are forwarded only to the local SBM logger. A running
web console is reused even while other SBK, SBM, or SBK-GEM WebLogger benchmarks are active; every aggregate has an
independent run URL and browser selection. After a run, graphs remain available while a browser is connected; the unused web console exits
after the configured idle timeout, which defaults to one minute. Remote browser access requires suitable routing and
firewall rules and does not automatically use SBK-GEM’s SSH connections. Because the service has no authentication
or TLS, use a trusted network or create a separate tunnel when needed.
See the WebLogger guide for web console options, concurrent-run and shutdown behavior, network security, and complete SBK, SBM, and SBK-GEM examples.
Diagnose distributed failures by boundary:
| Symptom | Check |
|---|---|
| Cannot connect | DNS, route, SSH port, account, SSH_AUTH_SOCK, identity-file permissions, and known_hosts |
| Authentication rejected | Remote authorized_keys, requested user, agent contents, configured identity files, or optional password |
| Host key rejected | Missing or changed entry in the SBK-GEM user’s ~/.ssh/known_hosts; verify the key out of band before updating it |
| Copy/install fails | Remote permissions, disk space, remote scp/tar support, archive-digest or file-manifest diagnostic |
| Remote Java failure | Homogeneous OS, Java 25 compatibility, executable java and javac |
| Driver not found | Same distribution on all hosts, both driver registration files, pathing JAR |
| No aggregate records | Remote command includes GrpcLogger; callback host/port reachable |
| Some clients disappear | Remote stderr, SBM connection metrics, firewall/NAT timeouts |
| Different results by host | Distribution hash, JVM flags, CPU/network topology, backend locality |
Save the sanitized connection topology, full forwarded SBK arguments, local and remote commit/version, Java versions, driver SDK version, embedded SBM settings, host clock status, and backend configuration. Treat aggregate throughput as a sum across load generators and verify that the target—not the network or SBM host—is the intended bottleneck.