Skip to content

Provenance

Every label image patchworks writes records how it was made: the segmentation function and its bound settings, tiling, overlap, stitching, level, channel, codec, the input, the library versions (including the segmentation tool's: Cellpose, PlantSeg, torch, ...) and a UTC timestamp. It sits in the label group's attrs under "patchworks" (for write_to= stores, on the labels array), so it travels with the data.

The cluster workflow stores its whole effective config (everything but the email settings), plus what was only decided at run time under resolved (tiles segmented, the empty threshold, the intensity range and thresholds measured for the image, the size filter in voxels, the object count, and for Cellpose the defaults it filled in) and the seam report ratios under seams.

from patchworks import read_provenance

rec = read_provenance("scan.zarr/labels/cilia_labels")
rec["settings"]["custom"]                # the method's settings
rec["settings"]["resolved"]              # tiles, empty threshold, object count
rec["seams"]                             # seam vs interior orphan rate per axis

patchworks.read_provenance(store: Any, component: str | None = None) -> dict | None

The provenance record of a label group (or array), if it has one.

Parameters:

Name Type Description Default
store (str, Path, Group or Array)

A label group path such as "scan.zarr/labels/cells", or an opened zarr node.

 required
component str

Array inside store to read instead (e.g. write_to stores keep it on the "labels" array).

 None

Returns:

Type Description
dict or None

The record written by :func:provenance, or None.
Source code in src/patchworks/_provenance.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def read_provenance(store: Any, component: str | None = None) -> dict | None:
    """The provenance record of a label group (or array), if it has one.

    Parameters
    ----------
    store : str, Path, zarr.Group or zarr.Array
        A label group path such as ``"scan.zarr/labels/cells"``, or an
        opened zarr node.
    component : str, optional
        Array inside *store* to read instead (e.g. ``write_to`` stores keep
        it on the ``"labels"`` array).

    Returns
    -------
    dict or None
        The record written by :func:`provenance`, or None.
    """
    from ._io import open_group_any

    node = store
    if isinstance(store, str) or hasattr(store, "__fspath__"):
        node = open_group_any(str(store))
    if component is not None:
        node = node[component]
    return dict(node.attrs).get(PROVENANCE_KEY)

patchworks.provenance(**settings: Any) -> dict[str, Any]

A JSON-safe record of a run: versions, time, and its settings.

Anything not JSON-serialisable (a function, a path object) is stored as its string form, so the record can always be written as zarr attrs.

Examples:

>>> rec = provenance(tile_shape=(16, 1024, 1024), stitch="iou")
>>> rec["settings"]["stitch"], "patchworks" in rec["versions"]
('iou', True)
Source code in src/patchworks/_provenance.py 
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def provenance(**settings: Any) -> dict[str, Any]:
    """A JSON-safe record of a run: versions, time, and its *settings*.

    Anything not JSON-serialisable (a function, a path object) is stored as
    its string form, so the record can always be written as zarr attrs.

    Examples
    --------
    >>> rec = provenance(tile_shape=(16, 1024, 1024), stitch="iou")
    >>> rec["settings"]["stitch"], "patchworks" in rec["versions"]
    ('iou', True)
    """
    record: dict[str, Any] = {
        "created": _dt.datetime.now(_dt.timezone.utc).isoformat(
            timespec="seconds"
        ),
        "versions": {
            name: _version(name)
            for name in (
                "patchworks",
                "zarr",
                "dask",
                "numpy",
                "scipy",
                "scikit-image",
                # The segmentation tools, where installed: the same settings
                # can segment differently from one of their versions to the
                # next.
                "cellpose",
                "plantseg",
                "plant-seg",
                "careamics",
                "torch",
                "pycudadecon",
            )
        },
        "python": platform.python_version(),
        "settings": settings,
    }
    record["versions"] = {k: v for k, v in record["versions"].items() if v}
    # A JSON round trip turns anything unserialisable into its string form.
    return json.loads(json.dumps(record, default=str))