Snapshots and cache¶
Save and reopen a snapshot¶
from osteosarc import Dataset
data = Dataset.sync("baseline")
print(data.id)
data = Dataset.open("baseline")
A snapshot records metadata URLs and SHA256 receipts. sync creates it;
open verifies the saved files and defaults to offline operation. Reusing a
name with sync reopens it without refreshing the sources.
To permit additional downloads, use Dataset.open("baseline", offline=False).
An uncached request while offline raises OfflineError; changed cached bytes
raise IntegrityError.
Choose a cache directory¶
from osteosarc import Cache
cache = Cache(".cache/my-analysis")
Pass cache=cache to Dataset.sync or Dataset.open. Otherwise the cache
location is chosen in this order:
| Setting | Location |
|---|---|
OSTEOSARC_CACHE |
An isolated cache for this package |
OPENVAX_DATA_CACHE |
A shared OpenVax cache |
| macOS default | ~/Library/Caches/openvax |
| Linux default | $XDG_CACHE_HOME/openvax, or ~/.cache/openvax |
Downloaded objects live at objects/sha256/<sha256><original suffixes>, so
other OpenVax tools can reuse them. Osteosarc's snapshots, receipts, and
extracted reads live under osteosarc/ within that root.
Refresh metadata¶
new_data = Dataset.sync("follow-up", refresh=True)
print(new_data.receipts()["bucket"].sha256)
Use a new name: existing snapshots cannot be overwritten. A file's first full download is bound to its snapshot, so refreshing the same URL elsewhere cannot replace those bytes. A snapshot cannot recover historical contents of a file that was never downloaded.
The CLI's sync --source-revision <commit> pins GitLab source resources to a
full commit. Site-served files have no equivalent versioning.
Import a file you already downloaded¶
from osteosarc import SNAPSHOT_SOURCES, digest
# This example uses an existing snapshot's table; substitute your file and URL.
old_path = data.source_path("vafs")
receipt = Cache().import_file(
old_path, SNAPSHOT_SOURCES["vafs"], sha256=digest(old_path)
)
print(receipt.sha256, receipt.size)
An import records the import time. Retain your old manifest if the original acquisition date matters.
Download behavior¶
Datacache downloads and verifies files, retries transient failures, and publishes complete files atomically. Osteosarc adds per-URL locks, immutable snapshot receipts, and MD5 checks where the source supplies them. Existing 0.1.0 snapshots and cached reads remain usable without conversion.
HTTP headers are checked before and after downloads when the server supports HEAD. Regional BAM/CRAM access uses SAMtools; its indexes use datacache and its extracted BAMs have separate checksum receipts.
Interrupted whole-file downloads restart from the beginning; byte-range resume is not supported. Cached objects keep their original suffixes so format-specific readers can recognize them.
Read extraction has a separate cache keyed by its complete request. Importing the Python package performs no downloads.