Skip to content

Verify

python -m qcstore verify URI [--repair]

Check a store's files: records complete and consistent, blobs intact, pointers right.

Report dataclass

Report(
    keys: int = 0,
    runs: int = 0,
    incomplete: list[str] = list(),
    problems: list[str] = list(),
    repaired: list[str] = list(),
)

What verify found.

Attributes:

Name Type Description
keys int

Calculations checked.

runs int

Complete runs checked.

incomplete list[str]

Runs without record.json: being written, or their writer died. Readers ignore them; they are not problems.

problems list[str]

Everything wrong, one message each.

repaired list[str]

Pointers rewritten (with repair).

verify

verify(store: Store, *, repair: bool = False) -> Report

Check every run of every calculation in a store.

For each complete run: its record names its key and run, its output loads (every blob and array file matching its sha256), the key recomputed from its input is the key it is filed under, its output names its identities, and each of its carries' files matches its sha256. For each key: current.json names the runs the rules prefer.

Parameters:

Name Type Description Default
store Store

The store.

required
repair bool

Rewrite each current.json that is missing or wrong (a writer died between storing a run and updating the pointer). Nothing else is changed.

False

Returns:

Type Description
Report

The report.

Source code in src/qcstore/verify.py
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def verify(store: Store, *, repair: bool = False) -> Report:
    """Check every run of every calculation in a store.

    For each complete run: its record names its key and run, its output loads (every
    blob and array file matching its sha256), the key recomputed from its input is the
    key it is filed under, its output names its identities, and each of its carries'
    files matches its sha256. For each
    key: ``current.json`` names the runs the rules prefer.

    Args:
        store: The store.
        repair: Rewrite each ``current.json`` that is missing or wrong (a writer died
            between storing a run and updating the pointer). Nothing else is changed.

    Returns:
        The report.
    """
    report = Report()
    strays: list[str] = []
    for key in store._keys(strays):
        report.keys += 1
        directory = _key_dir(key)
        for name in store.backend.list(directory):
            if _RUN.fullmatch(name) and not store.backend.exists(
                f"{directory}/{name}/record.json"
            ):
                report.incomplete.append(f"{directory}/{name}")
        try:
            runs = store._read_runs(key)
        except Exception as e:
            report.problems.append(f"{directory}: unreadable: {e}")
            continue
        for run in runs:
            report.runs += 1
            where = _run_dir(key, run.id)
            try:
                if (run.record["key"], run.record["run"]) != (key, run.id):
                    raise ValueError(
                        f"the record names {run.record['key']} {run.record['run']}"
                    )
                output = store._load(key, run.id, blobs=True)
                if store.key(output.input_data) != key:
                    raise ValueError("its input's key is not the key it is filed under")
                if not output.execution.identities:
                    raise ValueError("its output has no identities")
            except Exception as e:
                report.problems.append(f"{where}: {e}")
        if not runs:
            continue
        stored = store._pointer_now(key)
        if stored != _pointer(runs):
            if repair:
                store._update_pointer(key)
                report.repaired.append(f"{directory}/current.json")
            else:
                state = "missing" if stored is None else "stale"
                report.problems.append(
                    f"{directory}/current.json: {state}; `verify --repair` rewrites it"
                )
    report.problems += [
        f"{path}: not part of the layout (records/k1/<2 hex>/<64 hex>)"
        for path in strays
    ]
    return report