Skip to content

Benchmark CLI

q2mm-benchmark runs force-field fitting experiments and saves their outcomes. Use it when you want a reproducible record of the requested settings, the actual calculation, and why a candidate was accepted, rejected, skipped, or errored. It uses the same numerical q2mm.optimize implementation as Python applications; it is not a separate optimizer.

Commands are generated by Pydantic-settings CliApp from the shared component settings and benchmark command models. The examples below describe the current interface, not a rerun of previously published measurements. The old flat argparse interface has been removed.

Inspect before executing

q2mm-benchmark --help
q2mm-benchmark single --help
q2mm-benchmark list
q2mm-benchmark single --system ch3f --show-config
q2mm-benchmark matrix --system ch3f --show-config

Help, list, and --show-config do not construct engines, initialize devices, load scientific datasets, evaluate objectives, or write results. list checks lightweight dependency/plugin descriptors. --show-config is a normal model JSON dump of requested policy, not verified runtime provenance. For a matrix, it includes the actual catalog policies; empty backend/form selectors remain unresolved until execution. Batch's empty system selector means all registered systems at execution time.

Only the explicit device-probe command initializes devices for inspection:

q2mm-benchmark preflight

Optional dependencies can be absent. A failed installed device probe returns failure with diagnostics; an unexpected exception is not converted into success.

Single and batch settings

single requires one --system. batch accepts repeated --system values, or defaults to all registered systems. Both use the same actual BenchmarkSettings/RunProfile models, with a JAX analytical objective and SciPy L-BFGS-B default. Single-stage is the single-run workflow; batch defaults to Method E2 and disables diagnostic analysis unless --analyze is supplied.

# Bounded CPU example: a candidate record, not a convergence proof.
JAX_PLATFORMS=cpu q2mm-benchmark single --system ch3f \
  --functional-form harmonic --optimizer.maxiter 0 --n-evals 0 \
  --no-analyze --no-promote --output results/ch3f

# Inspect a selected batch without loading its datasets.
q2mm-benchmark batch --system rh-enamide --system heck-relay \
  --optimizer.ftol 1e-12 --optimizer.fc-fraction 0.05 \
  --optimizer.eq-fraction 0.05 --ratio-tol null --show-config

SciPy can take one step even with maxiter=0; that cap does not promise zero objective evaluations or a converged result. Every executed candidate records the actual solver result. Publication optimization remains subject to the methodology and completeness blockers.

Use dotted component fields, not flat projected knobs:

# Switch optimizer families: only Optax fields survive.
q2mm-benchmark single --system ch3f --optimizer.kind optax \
  --optimizer.optimizer adam --optimizer.max-steps 20 \
  --optimizer.learning-rate 0.001 --no-optimizer.verbose --show-config

# Explicit finite differences belong to the Python objective executor.
q2mm-benchmark single --system ch3f \
  --objective '{"executor":"python","gradient_mode":"finite_difference","fd_step":0.02}' \
  --optimizer.maxiter 2 --workflow.name single-stage --show-config

Component JSON uses Python field names (underscores); generated dotted flags use kebab-case. A full typed component replaces its declared default, while partial dotted/dictionary settings use native nested-default updates. For different optimizer families, provide --optimizer.kind; SciPy-only fields with kind=optax are errors. In a same-family update, unspecified preset values remain in force. fd_step must be positive for explicit finite differences and must be null for scalar-only or analytical objectives.

Use null to clear optional values and positive/negative boolean flags to set booleans explicitly, for example --optimizer.fc-fraction null, --optimizer.ftol 0, --optimizer.verbose, --no-optimizer.verbose, or --no-promote. Zero is retained where the underlying settings support it. Use numeric tokens for numeric flags; JSON booleans are not numeric settings. Unknown fields, nonfinite values, and invalid component combinations are diagnosed rather than silently defaulted.

Backend keys are open to installed plugins. Factory options are passed exactly to that backend as recursively serialized JSON dictionaries and lists; immutable settings internals are not passed to the factory. Explicit loading is authoritative: a valid configured executable path can work even when the catalog's generic PATH probe says unavailable. Actual missing dependencies produce skipped candidates; configuration, factory, and runtime-contract failures produce error candidates. Inspection still uses only cheap probes and never attempts configured construction.

The JAX objective requires the jax backend. For other backends, explicitly select the Python objective as shown below; choosing a backend never silently changes the executor. Contradictory single/batch settings fail validation before engine construction, even with --show-config.

q2mm-benchmark single --system ch3f --backend.key openmm \
  --backend.options '{"platform_name":"CPU"}' \
  --objective '{"executor":"python","gradient_mode":"none"}' --show-config

q2mm-benchmark single --system rh-enamide \
  --data-roots 'rh_enamide=/data/rh-enamide' \
  --data-roots 'mm3_base=/data/mm3_base.fld' --show-config

Lists and dictionaries use native repeated, comma-separated, or JSON parsing. Quote a KEY=PATH value containing spaces, for example --data-roots 'supporting_info="/data/supporting info"'. Valid root keys are ch3f, rh_enamide, supporting_info, and mm3_base. No tolerance/backend environment variables, dotenv files, or automatic config files supply settings. Only the existing documented external data-root fallback remains, resolved by its loader owner at execution, not by inspection.

Catalog matrices

Matrix requires one system. Repeated --backends, --forms, and --presets select the Cartesian product; omissions mean all cheaply available MM backends, the system's default forms, and all catalog presets, respectively. Matrices execute the exact typed catalog policies. They do not translate a shared flat iteration/tolerance knob into different algorithm families. For a customized solver policy, use single or batch.

The entire requested grid must pass the same shared settings validation before any candidate executes. A non-JAX backend paired with a JAX objective is incoherent: the whole command returns 2 with a diagnostic, even if earlier selections were valid. No engine factories, scientific data loads, candidates, or output directories are created. Defaults are not silently narrowed and executors are never rewritten. If all available backends plus all presets do not form a coherent grid, explicitly narrow --backends or --presets; for example, use --backends jax with JAX presets, or Python-executor presets for other backends. Explicit incoherent selections also fail --show-config; unresolved automatic selectors remain unverified during inspection. Runtime unavailability or unsupported functional forms of a valid profile still produce the runner's ordinary persisted skipped candidates.

q2mm-benchmark matrix --system ch3f --backends jax \
  --forms harmonic --forms mm3 \
  --presets scipy-lbfgsb-jax --presets optax-adam-cosine \
  --output results/ch3f

# A backend selector can also carry its actual BackendSettings.
q2mm-benchmark matrix --system ch3f \
  --backends '{"key":"openmm","options":{"platform_name":"CPU"}}' \
  --presets scipy-lbfgsb --forms harmonic --output results/ch3f

--functional-form is for single/batch; matrix uses --forms. Use --starting-point published to retain publication OPT values, or the default qfuerza for Hessian-derived initialization.

Results and exit status

q2mm-benchmark load results/ch3f

Every candidate is saved under <output>/candidates/. Only accepted candidates are promoted to <output>/accepted/ and <output>/forcefields/; --no-promote disables promotion, not candidate persistence.

main(argv) and the installed console script use the same statuses:

Status Meaning
0 Inspection/load succeeded, or the runner's outcome is successful. A skip-only run is successful but does not prove any optimization occurred.
1 Execution/load/probe failed, any candidate errored, or optimizations ran and none was accepted.
2 CLI syntax or settings validation failed.

The requested profile schema is explicitly benchmark-profile-v2. Its identity projection names the loader selections and exact typed execution components; presentation, labels, and private input-origin details do not define identity. Scientific problem fingerprints still use their existing explicit projections. Historical persisted candidates remain readable byte-for-byte as written: loading does not migrate their schema, invent provenance, or recompute their IDs.

Python callers use the same typed profile directly, not a scalar-knob adapter:

from pathlib import Path
from q2mm.benchmarks.profiles import RunProfile, catalog_settings
from q2mm.benchmarks.runner import run_profiles
from q2mm.settings import ScipySettings

policy = catalog_settings("scipy-lbfgsb-fd")
profile = RunProfile(
    system="ch3f",
    objective=policy.objective,
    optimizer=ScipySettings(maxiter=0, verbose=False),
    n_evals=0,
)
outcome = run_profiles([profile], output_dir=Path("results/ch3f"), analyze=False, promote=False)

For publication caller policies, recommended_publication_profile() selects real typed profile defaults. A partial dictionary such as recommended_publication_profile("heck-relay", optimizer={"maxiter": 10}) retains that policy's ftol=1e-12, force-constant fraction 0.05, and equilibrium fraction 0.05; ordinary QFUERZA publication policies retain force-constant fraction 0.20 instead. Published-start profiles retain ftol=1e-8 and no fractional bounds. Passing a complete ScipySettings or another typed optimizer component replaces the selected component, including its bounds, rather than merging it. These are caller policy defaults, not a change to the publication objective or evidence of a new scientific run.