Skip to content

Store

qcstore.Store

Store(uri: str | PathLike[str] | Backend)

Every run of every calculation, under its key.

Example
store = Store("./store")
run = store.put(output, tags=["screen"])  # output.execution.identities
store.get(store.key(inp), {"tblite": "0.8.0"})

Attributes:

Name Type Description
backend Backend

Where the files live.

Parameters:

Name Type Description Default
uri str | PathLike[str] | Backend

A directory (a path or a file:// URI), local or on NFS, made on the first write; or a Backend.

required

Raises:

Type Description
ValueError

A URI scheme without a backend.

Source code in src/qcstore/store.py
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
def __init__(self, uri: str | os.PathLike[str] | Backend) -> None:
    """Open a store.

    Args:
        uri: A directory (a path or a ``file://`` URI), local or on NFS, made on the
            first write; or a `Backend`.

    Raises:
        ValueError: A URI scheme without a backend.
    """
    if isinstance(uri, str | os.PathLike):
        text = os.fspath(uri)
        if "://" in text:
            parsed = urlparse(text)
            if parsed.scheme != "file":
                raise ValueError(f"No qcstore backend for {parsed.scheme}:// URIs.")
            text = unquote(parsed.path)
        self.backend: Backend = LocalBackend(text)
    else:
        self.backend = uri
    self._journal = _Journal(self.backend)

blob

blob(key: str, path: str, run: str | None = None) -> bytes

Read one log or file of a run, without loading anything else.

Parameters:

Name Type Description Default
key str

The calculation's key.

required
path str

The blob's path in the output: "logs", "results.files.orca.gbw", "results.frames.3.logs", "input_data.files.input.xyz".

required
run str | None

The run; None for the preferred one.

None

Returns:

Type Description
bytes

The bytes (text is UTF-8).

Raises:

Type Description
KeyError

No such run or blob.

Source code in src/qcstore/store.py
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
def blob(self, key: str, path: str, run: str | None = None) -> bytes:
    """Read one log or file of a run, without loading anything else.

    Args:
        key: The calculation's key.
        path: The blob's path in the output: ``"logs"``, ``"results.files.orca.gbw"``,
            ``"results.frames.3.logs"``, ``"input_data.files.input.xyz"``.
        run: The run; None for the preferred one.

    Returns:
        The bytes (text is UTF-8).

    Raises:
        KeyError: No such run or blob.
    """
    run = self._run_id(key, run)
    try:
        return self.backend.get(f"{_run_dir(key, run)}/blobs/{_blob_name(path)}")
    except FileNotFoundError:
        raise KeyError(f"Run {run} has no blob {path!r}.") from None

carry

carry(
    runs: Iterable[tuple[str, str]],
    identities: Identities,
    evidence: Mapping[str, Any],
) -> str

Declare runs valid for other identities too, citing the evidence once.

A carry asserts that each run's whole output, record included, is what the code of the new identities would produce for its input, within the calculation's accuracy. Carry after a release that changes no results, with evidence that it does not (a benchmark rerun); a release that changes the shape of record cannot be carried. Lookups under the new identities then reuse the runs. Nothing is copied: the carry is one file, carries/<id>.json, and each run gets an empty marker naming it.

Parameters:

Name Type Description Default
runs Iterable[tuple[str, str]]

(key, run) pairs: from runs, iter, or an index query.

required
identities Identities

The identities the runs are also valid for.

required
evidence Mapping[str, Any]

JSON: what was compared, and how closely it agreed.

required

Returns:

Type Description
str

The carry's id: the sha256 of its file.

Raises:

Type Description
KeyError

A run that is not in the store; nothing is carried then.

ValueError

No runs, or evidence is not JSON.

TypeError

identities are not strings to strings.

Source code in src/qcstore/store.py
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
def carry(
    self,
    runs: Iterable[tuple[str, str]],
    identities: Identities,
    evidence: Mapping[str, Any],
) -> str:
    """Declare runs valid for other identities too, citing the evidence once.

    A carry asserts that each run's whole output, ``record`` included, is what the
    code of the new identities would produce for its input, within the
    calculation's accuracy. Carry after a release that changes no results, with
    evidence that it does not (a benchmark rerun); a release that changes the shape
    of ``record`` cannot be carried. Lookups under the new identities then reuse
    the runs. Nothing is copied: the carry is one file, ``carries/<id>.json``, and
    each run gets an empty marker naming it.

    Args:
        runs: ``(key, run)`` pairs: from `runs`, `iter`, or an index query.
        identities: The identities the runs are also valid for.
        evidence: JSON: what was compared, and how closely it agreed.

    Returns:
        The carry's id: the sha256 of its file.

    Raises:
        KeyError: A run that is not in the store; nothing is carried then.
        ValueError: No runs, or ``evidence`` is not JSON.
        TypeError: ``identities`` are not strings to strings.
    """
    identities = dict(sorted(_check_strings(identities).items()))
    by_key: dict[str, list[str]] = {}
    for key, run in runs:
        by_key.setdefault(key, []).append(self._run_id(key, run))
    if not by_key:
        raise ValueError("Name at least one (key, run) to carry.")
    document = _dumps(
        {
            "identities": identities,
            "evidence": _json_object(evidence, "evidence"),
            "by": _user(),
            "time": _now(),
        }
    )
    carry = _digest(document)
    _put_once(self.backend, f"carries/{carry}.json", document)
    for key, key_runs in by_key.items():
        with self._journal.op("carry", key, runs=key_runs, carry=carry):
            for run in dict.fromkeys(key_runs):
                _put_once(
                    self.backend, f"{_run_dir(key, run)}/carries/{carry}", b""
                )
            self._update_pointer(key)
    return carry

copy_to

copy_to(
    other: Store, keys: Iterable[str] | None = None
) -> int

Copy runs into another store, unchanged: ids, times and contents.

Runs the other store has are skipped, but their withdrawals and carries are copied, as are the keys' tags, so the other store prefers the same runs.

Parameters:

Name Type Description Default
other Store

The destination.

required
keys Iterable[str] | None

The calculations to copy; None for all.

None

Returns:

Type Description
int

The number of runs copied.

Source code in src/qcstore/store.py
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
def copy_to(self, other: Store, keys: Iterable[str] | None = None) -> int:
    """Copy runs into another store, unchanged: ids, times and contents.

    Runs the other store has are skipped, but their withdrawals and carries are
    copied, as are the keys' tags, so the other store prefers the same runs.

    Args:
        other: The destination.
        keys: The calculations to copy; None for all.

    Returns:
        The number of runs copied.
    """
    copied = 0
    for key in self.iter() if keys is None else keys:
        runs = self._run_ids(_check_key(key))
        if not runs:
            continue
        tags = self.backend.list(f"{_key_dir(key)}/tags")
        with other._journal.op("copy", key, runs=runs, tags=tags):
            for run in runs:
                copied += other._copy_run(self.backend, key, run)
            for tag in tags:
                _put_once(other.backend, f"{_key_dir(key)}/tags/{tag}", b"")
            other._update_pointer(key)
    return copied

get

get(
    key: str,
    identities: Identities | None = None,
    *,
    run: str | None = None,
    blobs: bool = True,
) -> Output | None

Return the preferred run's output, or one named run's; None if there is none.

The preferred run is the latest success not withdrawn, else the latest run not withdrawn. Without identities that is the latest run whatever code made it: the returned output's execution.identities says which. With identities, only runs produced by exactly those identities, or carried to them, count. A carried run keeps the identities that produced it.

With run, that run is returned whether it is preferred or not, withdrawn or not: a citation of a run (an output's execution.store_run) reads back exactly what was cited. runs says whether it was withdrawn.

Parameters:

Name Type Description Default
key str

The calculation's key.

required
identities Identities | None

The numerics identity of each program the input names, as the caller would run them now; None for any.

None
run str | None

A run of the key: its id, or "<key>/<run>" as in execution.store_run. Not with identities.

None
blobs bool

Load logs and files. False leaves them out (logs None, files empty) and reads only the record and its arrays.

True

Returns:

Type Description
Output | None

The output as stored, with execution.store_run set to "<key>/<run>":

Output | None

where it was read from. None if there is none (or no such run).

Raises:

Type Description
ValueError

key is not a key; run is not a run id, or names another key; or both run and identities are given.

Source code in src/qcstore/store.py
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
def get(
    self,
    key: str,
    identities: Identities | None = None,
    *,
    run: str | None = None,
    blobs: bool = True,
) -> Output | None:
    """Return the preferred run's output, or one named run's; None if there is none.

    The preferred run is the latest success not withdrawn, else the latest run not
    withdrawn. Without ``identities`` that is the latest run whatever code made
    it: the returned output's ``execution.identities`` says which. With
    ``identities``, only runs produced by exactly those identities, or carried to
    them, count. A carried run keeps the identities that produced it.

    With ``run``, that run is returned whether it is preferred or not, withdrawn
    or not: a citation of a run (an output's ``execution.store_run``) reads back
    exactly what was cited. `runs` says whether it was withdrawn.

    Args:
        key: The calculation's key.
        identities: The numerics identity of each program the input names, as the
            caller would run them now; None for any.
        run: A run of the key: its id, or ``"<key>/<run>"`` as in
            ``execution.store_run``. Not with ``identities``.
        blobs: Load logs and files. False leaves them out (``logs`` None, ``files``
            empty) and reads only the record and its arrays.

    Returns:
        The output as stored, with ``execution.store_run`` set to ``"<key>/<run>"``:
        where it was read from. None if there is none (or no such run).

    Raises:
        ValueError: ``key`` is not a key; ``run`` is not a run id, or names
            another key; or both ``run`` and ``identities`` are given.
    """
    _check_key(key)
    if run is None:
        run = self._preferred(key, identities)
    else:
        if identities is not None:
            raise ValueError("Pass identities or a run, not both.")
        run = _check_run(key, run)
        if not self.backend.exists(f"{_run_dir(key, run)}/record.json"):
            return None
    return None if run is None else self._load(key, run, blobs)

index

index(path: str | PathLike[str] | None = None) -> Index

Open the store's SQLite index, brought up to date from the journals.

Parameters:

Name Type Description Default
path str | PathLike[str] | None

The index file; by default index.sqlite in a local store's directory. SQLite needs a local disk: for a store on NFS, pass a path on a local disk.

None

Returns:

Type Description
Index

The index.

Source code in src/qcstore/store.py
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
def index(self, path: str | os.PathLike[str] | None = None) -> Index:
    """Open the store's SQLite index, brought up to date from the journals.

    Args:
        path: The index file; by default ``index.sqlite`` in a local store's
            directory. SQLite needs a local disk: for a store on NFS, pass a path on
            a local disk.

    Returns:
        The index.
    """
    from .index import Index

    if path is None:
        if not isinstance(self.backend, LocalBackend):
            raise ValueError("Pass the index's path: the store is not local.")
        path = self.backend.root / "index.sqlite"
    return Index(self, Path(path))

iter

iter() -> Iterator[str]

Every key with at least one complete run, in key order.

Source code in src/qcstore/store.py
441
442
443
444
445
def iter(self) -> Iterator[str]:
    """Every key with at least one complete run, in key order."""
    for key in self._keys():
        if self._has_runs(key):
            yield key

key

key(inp: Input) -> str

The key a calculation is filed under: qcdata's calculation_key.

Source code in src/qcstore/store.py
216
217
218
def key(self, inp: Input) -> str:
    """The key a calculation is filed under: qcdata's `calculation_key`."""
    return _calculation_key(inp)

put

put(
    output: Output,
    *,
    tags: Iterable[str] = (),
    meta: Mapping[str, Any] | None = None,
) -> str

Add a run under the key of output.input_data.

The run is filed under the identities of the code that produced it, output.execution.identities ({program: numerics}), which the runner sets on every output, and is found again under exactly these. They are stored with the output. execution.store_run says where an output was read from, so it is never stored, at any depth.

Parameters:

Name Type Description Default
output Output

The calculation's output, successful or not, with its identities.

required
tags Iterable[str]

Tags for the key.

()
meta Mapping[str, Any] | None

JSON metadata stored with the run.

None

Returns:

Type Description
str

The new run's id.

Raises:

Type Description
ValueError

output has no identities; or it is a session frame (record.warm_start true): a warm start can reach another solution than the input alone, so it is not that input's result. Or a tag is invalid, or meta is not JSON.

Source code in src/qcstore/store.py
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
def put(
    self,
    output: Output,
    *,
    tags: Iterable[str] = (),
    meta: Mapping[str, Any] | None = None,
) -> str:
    """Add a run under the key of ``output.input_data``.

    The run is filed under the identities of the code that produced it,
    ``output.execution.identities`` (``{program: numerics}``), which the runner
    sets on every output, and is found again under exactly these. They are stored
    with the output. ``execution.store_run`` says where an output was read from,
    so it is never stored, at any depth.

    Args:
        output: The calculation's output, successful or not, with its identities.
        tags: Tags for the key.
        meta: JSON metadata stored with the run.

    Returns:
        The new run's id.

    Raises:
        ValueError: ``output`` has no identities; or it is a session frame
            (``record.warm_start`` true): a warm start can reach another solution
            than the input alone, so it is not that input's result. Or a tag is
            invalid, or ``meta`` is not JSON.
    """
    if not output.execution.identities:
        raise ValueError(
            "This output does not say which code produced it: "
            "output.execution.identities is empty. qccompute's runner sets it; "
            "a store keeps only results whose code it can name."
        )
    if output.record.get("warm_start") is True:
        raise ValueError(
            "This output started warm from a session's history (record.warm_start "
            "is true), so it may not be the result of its input alone; a store "
            "keeps only calculations run from their input."
        )
    inp = output.input_data
    key = self.key(inp)
    tags = _check_tags(tags)
    meta = _json_object(meta or {}, "meta")
    run = _new_run_id()
    files = _run_files(key, run, output, meta)
    with self._journal.op("put", key, run=run, tags=tags):
        self._write_run(key, run, files)
        for tag in tags:
            _put_once(self.backend, f"{_key_dir(key)}/tags/{tag}", b"")
        self._update_pointer(key)
    return run

runs

runs(key: str) -> list[Run]

Every complete run of a calculation, oldest first.

Raises:

Type Description
ValueError

key is not a key.

Source code in src/qcstore/store.py
322
323
324
325
326
327
328
def runs(self, key: str) -> list[Run]:
    """Every complete run of a calculation, oldest first.

    Raises:
        ValueError: ``key`` is not a key.
    """
    return [_run(f) for f in self._read_runs(_check_key(key))]

tag

tag(key: str, *tags: str) -> None

Add tags to a calculation; tags already there stay.

Raises:

Type Description
KeyError

The store has no run of this calculation.

ValueError

A tag is not 1-64 of a-z0-9._- without a dot at either end, or is a name Windows reserves.

Source code in src/qcstore/store.py
419
420
421
422
423
424
425
426
427
428
429
430
431
432
def tag(self, key: str, *tags: str) -> None:
    """Add tags to a calculation; tags already there stay.

    Raises:
        KeyError: The store has no run of this calculation.
        ValueError: A tag is not 1-64 of ``a-z0-9._-`` without a dot at either
            end, or is a name Windows reserves.
    """
    checked = _check_tags(tags)
    if not self._has_runs(_check_key(key)):
        raise KeyError(f"No run of {key} in this store.")
    with self._journal.op("tag", key, tags=checked):
        for tag in checked:
            _put_once(self.backend, f"{_key_dir(key)}/tags/{tag}", b"")

untag

untag(key: str, *tags: str) -> None

Remove tags from a calculation; tags not there are ignored.

Source code in src/qcstore/store.py
434
435
436
437
438
439
def untag(self, key: str, *tags: str) -> None:
    """Remove tags from a calculation; tags not there are ignored."""
    checked = _check_tags(tags)
    with self._journal.op("untag", _check_key(key), tags=checked):
        for tag in checked:
            self.backend.delete(f"{_key_dir(key)}/tags/{tag}")

withdraw

withdraw(key: str, run: str, reason: str) -> None

Withdraw a run: it stops being preferred, and nothing is deleted.

Withdrawing a withdrawn run keeps the first withdrawal.

Raises:

Type Description
KeyError

No such run.

Source code in src/qcstore/store.py
351
352
353
354
355
356
357
358
359
360
361
362
363
def withdraw(self, key: str, run: str, reason: str) -> None:
    """Withdraw a run: it stops being preferred, and nothing is deleted.

    Withdrawing a withdrawn run keeps the first withdrawal.

    Raises:
        KeyError: No such run.
    """
    run = self._run_id(key, run)
    marker = _dumps({"by": _user(), "time": _now(), "reason": reason})
    with self._journal.op("withdraw", key, run=run):
        _put_once(self.backend, f"{_run_dir(key, run)}/withdrawn.json", marker)
        self._update_pointer(key)

qcstore.Run dataclass

Run(
    id: str,
    time: datetime,
    success: bool,
    error_kind: str | None,
    identities: dict[str, str],
    carries: list[Carry],
    withdrawn: Withdrawal | None,
    meta: dict[str, Any],
    program_version: str | None,
    host: str | None,
    cpu_model: str | None,
    devices: list[str],
    libc: str | None,
)

One run of a calculation, as Store.runs lists it.

Attributes:

Name Type Description
id str

The run id: its UTC time and a random suffix.

time datetime

When it was stored (UTC).

success bool

Whether the calculation succeeded.

error_kind str | None

The kind of failure, for a failed run.

identities dict[str, str]

The numerics identity of each program that produced it.

carries list[Carry]

The other identities it was carried to.

withdrawn Withdrawal | None

Why it was withdrawn, if it was.

meta dict[str, Any]

What the writer stored with it.

program_version str | None

The program version its output reports.

host str | None

The machine it ran on, as its output reports.

cpu_model str | None

The machine's CPU model, as its output reports.

devices list[str]

The GPU devices it ran with, as its output reports.

libc str | None

The C library version it ran with, as its output reports.

qcstore.Withdrawal dataclass

Withdrawal(by: str, time: datetime, reason: str)

Why a run stopped being preferred.

Attributes:

Name Type Description
by str

The user who withdrew it.

time datetime

When (UTC).

reason str

Why.

qcstore.Carry dataclass

Carry(
    id: str,
    identities: dict[str, str],
    evidence: dict[str, Any],
    by: str,
    time: datetime,
)

Runs declared valid for other identities: new code, same results.

Attributes:

Name Type Description
id str

The sha256 of the carry's file, carries/<id>.json.

identities dict[str, str]

The identities the runs are also valid for, {program: numerics}.

evidence dict[str, Any]

Why: what was compared, and how closely it agreed.

by str

The user who carried them.

time datetime

When (UTC).