Analysis flow¶

Here, we’ll track typical data transformations like subsetting that occur during analysis.

If exploring more generally, read this first: Project flow.

Setup¶

Install the lamindb Python package:

pip install 'lamindb[jupyter,bionty]'

# a lamindb instance containing Bionty schema
!lamin init --storage ./analysis-usecase --schema bionty

import lamindb as ln
import bionty as bt
from lamin_utils import logger

bt.settings.auto_save_parents = False

💡 connected lamindb: testuser1/analysis-usecase

Register an initial dataset¶

Here we register an initial artifact with a pipeline script register_example_file.py.

!python analysis-flow-scripts/register_example_file.py

Show code cell output Hide code cell output

💡 connected lamindb: testuser1/analysis-usecase

💡 saved: Transform(uid='K4wsS5DTYdFp6K79', version='0', name='register_example_file.py', key='register_example_file.py', type='script', created_by_id=1, updated_at='2024-07-21 16:21:53 UTC')
💡 saved: Run(uid='omrC1yUxtPYfPPdFa7qA', transform_id=1, created_by_id=1)

✅ added 3 records with Feature.name for columns: 'cell_type', 'tissue', 'disease'

❗ 1 non-validated categories are not saved in Feature.name: ['cell_type_id']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns

✅ added 99 records from public with Gene.ensembl_gene_id for var_index: 'ENSG00000000003', 'ENSG00000000005', 'ENSG00000000419', 'ENSG00000000457', 'ENSG00000000460', 'ENSG00000000938', 'ENSG00000000971', 'ENSG00000001036', 'ENSG00000001084', 'ENSG00000001167', 'ENSG00000001460', 'ENSG00000001461', 'ENSG00000001497', 'ENSG00000001561', 'ENSG00000001617', 'ENSG00000001626', 'ENSG00000001629', 'ENSG00000001630', 'ENSG00000001631', 'ENSG00000002016', ...
💡 saving labels for 'cell_type'

✅ added 3 records from public with CellType.name for cell_type: 'T cell', 'hematopoietic stem cell', 'hepatocyte'
❗ 1 non-validated categories are not saved in CellType.name: ['my new cell type']!
      → to lookup categories, use lookup().cell_type
      → to save, run .add_new_from('cell_type')
💡 saving labels for 'tissue'

✅ added 4 records from public with Tissue.name for tissue: 'kidney', 'liver', 'heart', 'brain'
💡 saving labels for 'disease'

✅ added 4 records from public with Disease.name for disease: 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'

✅ added 1 record with CellType.name for cell_type: 'my new cell type'

✅ var_index is validated against Gene.ensembl_gene_id
✅ cell_type is validated against CellType.name
✅ tissue is validated against Tissue.name
✅ disease is validated against Disease.name

💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/zgPU2UcjPOu2Cl6LCJor.h5ad')
✅ storing artifact 'zgPU2UcjPOu2Cl6LCJor' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/zgPU2UcjPOu2Cl6LCJor.h5ad'
💡 parsing feature names of X stored in slot 'var'
✅    99 terms (100.00%) are validated for ensembl_gene_id
✅    linked: FeatureSet(uid='f6LNyphifmGoejy3R9LA', n=99, dtype='float', registry='bionty.Gene', hash='-frOq7J0bik-J7Ad9DX7', created_by_id=1, run_id=1)
💡 parsing feature names of slot 'obs'
✅    3 terms (75.00%) are validated for name
❗    1 term (25.00%) is not validated for name: cell_type_id
✅    linked: FeatureSet(uid='qVjducIcegcLy5Cr08Mj', n=3, registry='Feature', hash='GugUReFIYGK7zNz0eMnZ', created_by_id=1, run_id=1)

✅ saved 2 feature sets for slots: 'var','obs'

Pull the registered dataset, apply a transformation, and register the result¶

Track the current notebook:

ln.settings.transform.stem_uid = "eNef4Arw8nNM"
ln.settings.transform.version = "0"
ln.track()

💡 notebook imports: bionty==0.44.2 lamin_utils==0.13.2 lamindb==0.74.1

💡 saved: Transform(uid='eNef4Arw8nNM6K79', version='0', name='Analysis flow', key='analysis-flow', type='notebook', created_by_id=1, updated_at='2024-07-21 16:22:28 UTC')

💡 saved: Run(uid='Y46Jo9DuOhj8m1FvmYp0', transform_id=2, created_by_id=1)

Run(uid='Y46Jo9DuOhj8m1FvmYp0', started_at='2024-07-21 16:22:28 UTC', is_consecutive=True, transform_id=2, created_by_id=1)

artifact = ln.Artifact.filter(description="anndata with obs").one()
artifact.describe()

Artifact(uid='zgPU2UcjPOu2Cl6LCJor', description='anndata with obs', suffix='.h5ad', type='dataset', accessor='AnnData', size=46992, hash='IJORtcQUSS11QBqD-nTD0A', hash_type='md5', n_observations=40, visibility=1, key_is_virtual=True, updated_at='2024-07-21 16:22:27 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase'
    .transform = 'register_example_file.py'
    .run = '2024-07-21 16:21:53 UTC'
  Labels
    .tissues = 'kidney', 'liver', 'heart', 'brain'
    .cell_types = 'T cell', 'hematopoietic stem cell', 'hepatocyte', 'my new cell type'
    .diseases = 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell', 'hepatocyte', 'my new cell type'
    'disease' = 'chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease'
    'tissue' = 'kidney', 'liver', 'heart', 'brain'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'tissue', 'disease'

Get a backed AnnData object¶

adata = artifact.open()
adata

AnnDataAccessor object with n_obs × n_vars = 40 × 100
  constructed for the AnnData object zgPU2UcjPOu2Cl6LCJor.h5ad
    obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
    var: ['_index']

Subset dataset to specific cell types and diseases¶

cell_types = artifact.cell_types.all().lookup(return_field="name")
diseases = artifact.diseases.all().lookup(return_field="name")

Create the subset:

subset_obs = adata.obs.cell_type.isin(
    [cell_types.t_cell, cell_types.hematopoietic_stem_cell]
) & (adata.obs.disease.isin([diseases.liver_lymphoma, diseases.chronic_kidney_disease]))

adata_subset = adata[subset_obs]
adata_subset

AnnDataAccessorSubset object with n_obs × n_vars = 20 × 100
  obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
  var: ['_index']

adata_subset.obs[["cell_type", "disease"]].value_counts()

cell_type                disease               
T cell                   chronic kidney disease    10
hematopoietic stem cell  liver lymphoma            10
Name: count, dtype: int64

Register the subsetted AnnData:

curate = ln.Curate.from_anndata(
    adata_subset.to_memory(), 
    var_index=bt.Gene.ensembl_gene_id, 
    categoricals={
        "cell_type": bt.CellType.name, 
        "disease": bt.Disease.name, 
        "tissue": bt.Tissue.name,
    },
    organism="human"
)

curate.validate()

artifact = curate.save_artifact(description="anndata with obs subset")

artifact.describe()

Artifact(uid='odR6XTR5rNECtEqsZeVl', description='anndata with obs subset', suffix='.h5ad', type='dataset', accessor='AnnData', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', hash_type='md5', n_observations=20, visibility=1, key_is_virtual=True, updated_at='2024-07-21 16:22:29 UTC')
  Provenance
    .created_by = 'testuser1'
    .storage = '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase'
    .transform = 'Analysis flow'
    .run = '2024-07-21 16:22:28 UTC'
  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'
  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
    'tissue' = 'kidney', 'liver'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'tissue', 'disease'

Examine data flow¶

Query a subsetted .h5ad artifact containing “hematopoietic stem cell” and “T cell”:

cell_types = bt.CellType.lookup()

my_subset = ln.Artifact.filter(
    suffix=".h5ad",
    description__endswith="subset",
    cell_types__in=[
        cell_types.hematopoietic_stem_cell,
        cell_types.t_cell,
    ],
).first()

my_subset

Artifact(uid='odR6XTR5rNECtEqsZeVl', description='anndata with obs subset', suffix='.h5ad', type='dataset', accessor='AnnData', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', hash_type='md5', n_observations=20, visibility=1, key_is_virtual=True, created_by_id=1, storage_id=1, transform_id=2, run_id=2, updated_at='2024-07-21 16:22:29 UTC')

Common questions that might arise are:

What is the history of this artifact?
Which features and labels are associated with it?
Which notebook analyzed and registered this artifact?
By whom?
And which artifact is its parent?

Let’s answer this using LaminDB:

print("--> What is the history of this artifact?\n")
artifact.view_lineage()

print("\n\n--> Which features and labels are associated with it?\n")
logger.print(artifact.features)
logger.print(artifact.labels)

print("\n\n--> Which notebook analyzed and registered this artifact\n")
logger.print(artifact.transform)

print("\n\n--> By whom\n")
logger.print(artifact.created_by)

print("\n\n--> And which artifact is its parent\n")
display(artifact.run.input_artifacts.df())

--> What is the history of this artifact?

_images/90bb2f59f0e899eb1a89a153f844635ec305cf4a8794a3f5138447b1374600e3.svg

--> Which features and labels are associated with it?

  Features
    'cell_type' = 'T cell', 'hematopoietic stem cell'
    'disease' = 'chronic kidney disease', 'liver lymphoma'
    'tissue' = 'kidney', 'liver'
  Feature sets
    'var' = 'TSPAN6', 'TNMD', 'DPM1', 'SCYL3', 'FIRRM', 'FGR', 'CFH', 'FUCA2', 'GCLC', 'NFYA', 'STPG1', 'NIPAL3', 'LAS1L', 'ENPP4', 'SEMA3F', 'CFTR', 'ANKIB1', 'CYP51A1', 'KRIT1', 'RAD52'
    'obs' = 'cell_type', 'tissue', 'disease'

  Labels
    .tissues = 'kidney', 'liver'
    .cell_types = 'T cell', 'hematopoietic stem cell'
    .diseases = 'chronic kidney disease', 'liver lymphoma'

--> Which notebook analyzed and registered this artifact

Transform(uid='eNef4Arw8nNM6K79', version='0', name='Analysis flow', key='analysis-flow', type='notebook', created_by_id=1, updated_at='2024-07-21 16:22:28 UTC')

--> By whom

User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at='2024-07-21 16:21:50 UTC')

--> And which artifact is its parent

	uid	version	description	key	suffix	type	accessor	size	hash	hash_type	n_objects	n_observations	visibility	key_is_virtual	storage_id	transform_id	run_id	created_by_id	updated_at
id
1	zgPU2UcjPOu2Cl6LCJor	None	anndata with obs	None	.h5ad	dataset	AnnData	46992	IJORtcQUSS11QBqD-nTD0A	md5	None	40	1	True	1	1	1	1	2024-07-21 16:22:27.992715+00:00

Show code cell content Hide code cell content

!lamin delete --force analysis-usecase
!rm -r ./analysis-usecase

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamin_cli/__main__.py", line 105, in delete
    return delete(instance, force=force)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/_delete.py", line 98, in delete
    n_objects = check_storage_is_empty(
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/core/upath.py", line 779, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb contains 4 objects ('_is_initialized' ignored) - delete them prior to deleting the instance
['/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/SGzI9PupwUO6t8JrjHMz.py', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/_is_initialized', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/odR6XTR5rNECtEqsZeVl.h5ad', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/rRovtPt7S9qBVxpwD3Xm.txt', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/zgPU2UcjPOu2Cl6LCJor.h5ad']