Samples and files¶
Known overlapping RNA evidence¶
The reprocessed Tempus ALMY2X4KMV and KCVBE1UI1P products overlap almost completely in the audited vaccine regions: 18,971 shared read names and 40,684 matching core alignment records out of 40,696 per file, including base qualities. They must not be counted as independent RNA evidence. This is a regional observation, not proof that the full libraries are identical or that either biological sample label is correct. See the provenance investigation.
Each file.evidence_overlaps lists reviewed relationships for its exact product
URL. A selection's files.overlap_groups reports pairs present together, and
files.require_no_known_overlaps() raises before they can be used as independent
evidence. To keep one explicitly, call
files.without_known_overlaps(prefer=[chosen_url]). The returned selection records
every exclusion in files.source["excluded_evidence_overlaps"]; original labels,
sample claims and receipts remain unchanged. No listed overlap means no reviewed
overlap is known, not proof of independence.
The published FASTQ catalogue already identifies KCVBE1UI1P as a duplicate delivery
of ALMY2X4KMV RNA. Corresponding mate objects also have matching lengths and multipart
ETags (server metadata, not whole-file SHA-256 hashes). The overlap registry preserves
that source receipt and the object metadata. The reviewed October metadata now
labels the Tempus tumor products T0 and records the matched normal separately as
blood_2024-09-21. Historical snapshots retain the tempus-file-labels warning.
The updated labels do not make the overlapping RNA products independent; the
specimen-provenance investigation remains open.
Catalogue¶
- Samples are what was collected: tumor biopsies and resections at time points T0 to T3, organoids grown from the T1 and T2 tumors, and blood draws.
- Files are what's in the public S3 bucket: aligned reads (BAMs), raw reads (FASTQs), variant calls, expression tables, scans and more, nearly 400,000 in all.
Each sample lists its BAMs and FASTQ locations. A location can be a directory or a filename prefix. Most files belong to no sample: imaging, pathology slides, analyses and references. The examples open your most recent snapshot; see get started.
Samples¶
osteosarc samples # every sample, and what was sequenced
osteosarc samples T1_tumor # one sample's files, with commands to get them
osteosarc samples --files # every sample's BAMs and FASTQ folders
osteosarc samples --tissue blood --assay cite-seq # filter by time point, tissue, assay or platform
The same in Python, where each line shows a table in a notebook or REPL:
from osteosarc import Dataset
data = Dataset.open()
data.samples
data.samples.select(timepoint="T0", tissue="tumor")
data.samples.select(assay="scrna-seq", platform="ont")
The single-cell Oxford Nanopore samples in the 2026-10-07 snapshot:
3 tumor samples
sample timepoint date where sequencing BAMs FASTQ locations
-------- --------- ---------- ----- ------------------------------------------ ---- ---------------
T1_tumor T1 2024-06-06 UCLA rna-seq, wes, wgs, scrna-seq (ont, pacbio) 12 10
T2_tumor T2 2025-01-28 UCLA rna-seq, wgs, scrna-seq (ont), cite-seq 8 7
T3_tumor T3 2025-04-17 MSKCC scrna-seq (ont), cite-seq 2 5
One sequencing run can have several BAMs, such as a vendor's and a reprocessed one.
data.samples["T1_tumor"] shows one sample: where it came from, each BAM with its
assay, provider, size and whether you've downloaded it, each FASTQ folder with its
library, FASTQ-only file count and size, and the calls that fetch them. Filename
prefixes include the CeGaT, regenerated Natera and UCLA deliveries. The Tempus
DNA/FastQ/T and DNA/FastQ/N row labels resolve to the full tumor and normal
filename prefixes within their shared directory; they are not actual folders.
sample.fastq_folders retains these published location labels, while the displayed
locations and download commands use the resolved prefixes.
What the fields mean¶
| Field | Example | Meaning |
|---|---|---|
| Sample ID | T0_tumor | A sample, or a fraction of one |
| Time point | T0 | When it was collected; tumor and blood from one visit share it |
| Tissue | tumor | tumor, blood or organoid |
| Assay | rna-seq | What was sequenced, and whether bulk or single-cell |
| Platform | ont | The sequencing technology, when the site says |
| Provider | BostonGene | Who produced the data |
T0_tumor is the primary tumor resection on 2022-12-16. T0_blood is a historical grouping of normal products, not proof that all blood was drawn that day: its current notes identify the BostonGene draw as 2023-01-27, while Natera normal collection details remain unconfirmed. The Personalis normal is recorded separately as blood_2024-06-11. T1_tumor and T1_organoid share a time point but not a tissue. T2_organoid comes from the 2025-01-28 UCLA biopsy; only its WGS is assigned to the sample because the delivered RNA contains substantial contamination from another person. The WGS notes also flag mouse reads and required preprocessing. T3_tumor_CD45neg is a CD45-negative fraction of T3_tumor. Later blood draws are named by date, such as blood_2025-06-26, and have no time point. A snapshot's date is when you downloaded the website's metadata, not a collection date.
In Python, a sample's fields are attributes:
t1 = data.samples["T1_tumor"]
print(t1.id, t1.timepoint, t1.date, t1.tissue, t1.site, t1.description)
print(t1.assays, t1.providers)
print(t1.bams[:2], t1.fastq_folders[:2])
A sample's disagreements list dates or sites that other sources give differently, and its corrections name the fixes that changed it.
Where sequencing comes from¶
A sample's sequencing combines the site's sample registry with its FASTQ table, which records each location's assay. The eleven 2026 blood entries in the October 7 snapshot have no assays in the registry, but their FASTQ folders show single-cell gene expression, T-cell receptor and CITE-seq libraries.
The FASTQ table also supplies each matched file's assay and provider for file filters. Its library-specific labels take precedence over a broader data-page directory, such as CeGaT RNA delivered inside a WES folder; conflicting claims about an exact file remain visible.
The site's labels map to osteosarc's names like this; using a site label as a filter gives an error that names the right one:
| Site label | Assay | Library | Data |
|---|---|---|---|
| RNA, bulk RNA | rna-seq | Bulk RNA sequencing | |
| WES | wes | Whole-exome DNA sequencing | |
| WGS | wgs | Whole-genome DNA sequencing | |
| Panel | panel | Targeted DNA panel (Tempus xT) | |
| scRNA, scRNA_GEX | scrna-seq | gene expression | Single-cell RNA sequencing |
| scRNA_TCR, scRNA_TCRgd, scRNA_BCR | scrna-seq | αβ TCR, γδ TCR, BCR | T- and B-cell receptor libraries from the same cells |
| scRNA_ONT | scrna-seq, platform ont | long reads | Single-cell Oxford Nanopore RNA sequencing |
| PacBio | scrna-seq, platform pacbio | long reads | Single-cell PacBio RNA sequencing |
| CITE | cite-seq | antibody tags | The antibody-tag library of a CITE-seq capture |
Single cells aren't samples: their barcodes are inside the single-cell files.
A sample's files¶
A sample's files are its BAMs and everything in its FASTQ folders:
t0 = data.samples["T0_tumor"]
for file in t0.files.select(kind="alignment", assay="rna-seq"):
print(file.key, file.size)
T1_tumor has both bulk and single-cell RNA, so pick one by assay, and long reads by platform:
t1 = data.samples["T1_tumor"]
bulk = t1.files.select(kind="alignment", assay="rna-seq")
single_cell = t1.files.select(kind="alignment", assay="scrna-seq")
print(len(bulk), len(single_cell), len(single_cell.select(platform="ont")))
Many files have no platform label; the T0 alignments, for instance, don't. Filtering by platform illumina leaves them out, so filter by assay instead.
Find any file¶
osteosarc files with no filters summarizes the bucket: how many files of each
kind, their size, and the largest folders. Filters list files:
osteosarc files
osteosarc files --kind alignment --assay wgs
osteosarc files --sample T1_tumor --kind alignment
osteosarc files --prefix hudson_lab/PBMC_scRNAseq/FASTQ/Jan2026/
Each file is listed with its complete key, which is what osteosarc download and
osteosarc reads take, and marked local if you've downloaded it. In Python:
ont = data.files.select(kind="alignment", platform="ont")
vcfs = data.files.select(kind="variants", format="vcf")
scans = data.files.select(kind="image", format="dcm")
pvac = data.files.select(prefix="neoantigen_prediction/pvactools/", format="tsv")
source = data.file("rna-seq/reprocessed/BG003082/BG003082.Aligned.sortedByCoord.out.md.bam")
print(source.size, source.samples)
The kinds are alignment, reads, variants, expression, annotation, table, reference, index, image and other. A prefix or text in the key finds files whatever their labels say.
When the site's sources disagree about a file's sample, filters skip it;
include_conflicts=True matches any of their values. A file's claims show what each
source says.
Get a file¶
Most of the time you want a region of a BAM, not the whole thing: reads streams just those reads into a small local BAM. To download a whole file:
osteosarc download rna-seq/reprocessed/BG003082/BG003082.Aligned.sortedByCoord.out.md.bam --to .
osteosarc downloads
osteosarc files --downloaded
--to puts the file, and its index, in a folder under its own name, as a read-only
hard link to the cached copy where it can, so it isn't stored twice. downloads
lists what you have and where. In Python, data.download(file, to="."),
data.local_path(file) and data.downloads() do the same. For a whole FASTQ folder
the AWS command line is faster; osteosarc samples SAMPLE prints the command.
Read a table¶
The site's read counts, one row per variant and BAM:
counts = data.vafs.select(gene="SMC5")
print(counts.rows[:2])
Other tables download when you parse them: the site's own (snv_top, dna_fusions, rna_fusions) or any CSV or TSV in the bucket.
data = Dataset.open(offline=False)
top = data.parse("snv_top")
print(top.columns, top.rows[:2])
Values stay as text, so "0", "NA" and an empty cell stay different; to_dataframe()
turns a table into a pandas data frame.
Look for newer files¶
A snapshot uses the file listing the website published. To list a folder of the bucket as it is now, without changing your snapshot:
from osteosarc import Cache
from osteosarc.discovery import list_bucket
listing = list_bucket(Cache(), "ONT/", refresh=True)
print(len(listing["files"]))
osteosarc sync --refresh saves a new snapshot with the website's latest metadata.