Skip to content

qcstore

A store for quantum chemistry results. Every run of every calculation is kept as plain JSON you can read with cat and jq, filed under the calculation's key, and nothing is ever overwritten or deleted. Many processes can write to one store at once, on a local disk or NFS, without locks, and a writer killed mid-write leaves nothing a reader trusts.

qcstore depends only on qcdata and the Python standard library.

Install

qcstore is in alpha. Install it pinned with qcdata (--prerelease=allow lets the resolver take pre-releases):

uv pip install --prerelease=allow "qcdata==0.24.0a6" "qcstore==0.1.0a4"

To cache calculations that qccompute runs, as at the end of the next section, add the runner and its external programs:

uv pip install --prerelease=allow "qcdata==0.24.0a6" "qccompute[qccodec]==0.23.0a5" "qccodec==0.22.0a4" "qcstore==0.1.0a4"

Five minutes

A store files each calculation under its key, qcdata.calculation_key(inp): the input alone. Each run also keeps the numerics identity of every program that produced it, so results of different code stay apart while the key stays the same across releases. The runner sets the identities on every output it returns, execution.identities: it asks each program for a string that changes whenever its results may change, and put stores them with the run. For a Python library that is its version ({"tblite": "0.8.0"}); for an external executable, its version and the sha256 of the file ({"xtb": "6.7.1 sha256:9c1f…"}); for a program with numerics epochs, the epoch. qccompute.Runner().identities(inp) shows them for an input.

from qcdata import ProgramInput, ProgramOutput, SinglePointData, Structure
from qcstore import Store

water = Structure(
    symbols=["O", "H", "H"],
    geometry=[[0.0, 0.0, 0.0], [0.0, 1.43, 1.11], [0.0, -1.43, 1.11]],  # bohr
)
inp = ProgramInput(
    program="tblite", calctype="energy", structure=water, model={"method": "GFN2-xTB"}
)
output = ProgramOutput(
    input_data=inp,
    success=True,
    logs="SCF converged in 9 iterations\n",
    results=SinglePointData(
        provenance={"program": "tblite", "program_version": "0.8.0"}, energy=-5.07
    ),
    execution={"identities": {"tblite": "0.8.0"}},  # what the runner sets
)

store = Store("./store")
run = store.put(output, tags=["demo"], meta={"by": "readme"})

key = store.key(inp)
tblite = {"tblite": "0.8.0"}
store.get(key, tblite)             # only a run of tblite 0.8.0
store.get(key, {"tblite": "0.9.0"}) # None: never run with 0.9.0
store.get(key)                     # the latest run, whatever code made it...
store.get(key).execution.identities  # ...and which code that was
store.blob(key, "logs")            # b"SCF converged in 9 iterations\n"
store.get(key).execution.store_run # "k1:35a1.../20261117T...": read from this store

get(key) without identities returns the latest run (the latest success, else the latest failure) whatever code made it, so two builds of one program can both be behind a key. Ask with identities, get(key, identities), to get a run of that code, or read the returned output's execution.identities to see which code you got.

A new release that gives the same results need not rerun anything: carry the runs to the new identities, with the evidence. The evidence is stored once, however many runs it covers, and asserts that each run's whole output is what the new code would produce.

newer = {"tblite": "0.9.0"}
store.carry([(key, run)], newer, {"benchmark": "S22", "max_abs_diff": 0.0})
store.get(key, newer)              # the 0.8.0 run, valid for 0.9.0 too

A wrong result is withdrawn, not deleted:

store.withdraw(key, run, "wrong geometry")
store.get(key)                     # None: no other run
for r in store.runs(key):
    print(r.id, r.success, r.identities, r.withdrawn.reason)

Questions across calculations go to an optional SQLite index, kept up to date from the store's journals:

index = store.index()
index.query(tag="demo", program="tblite")
index.query(identities=newer)      # every run valid under tblite 0.9.0
index.sql(
    "SELECT formula, COUNT(DISTINCT key) AS calculations FROM runs GROUP BY formula"
)

With qccompute, a store caches calculations: qccompute.compute(inp, store=store) returns a stored result when one from the same code exists, and stores what it runs. Either way the output names its run, execution.store_run, so a result can be cited exactly and read back later, even after a newer run of the same calculation has become the preferred one:

out = qccompute.compute(inp, store=store)
cited = out.execution.store_run    # "k1:35a1.../20261117T..."
store.get(key, run=cited)          # that run, preferred or not, withdrawn or not

Two processes that miss the same key at once each compute it, and the store keeps both runs: duplicate work, never a wrong result.

On disk

store/
  records/k1/3f/3f9a.../               one directory per calculation, k1:3f9a...
    current.json                       the preferred run, overall and per identity set
    tags/demo                          an empty file per tag
    20261117T101500.123456Z-7c1e9a0b/  one directory per run, never changed
      record.json                      meta and the output, identities included
      blobs/output.logs                logs, files and large arrays, named by JSON path
      carries/9c1f...                  an empty marker per carry of this run
      withdrawn.json                   who withdrew it, when, and why
  carries/9c1f....json                 a carry: "also valid for tblite 0.9.0", the evidence
  journals/<writer>/<batch>.jsonl      what each writer changed, for the index
  index.sqlite                         the optional index, rebuildable at any time

python -m qcstore verify ./store checks every record: keys, blob digests, pointers.

Documentation

qcstore.docs.afh.org/next documents the alpha: the API, and the design: the layout, how writers agree without locks, and what a crash can and cannot leave behind. Until qcstore's first final release, the site root shows the same pages.