Parameters

The pipeline supports many types of parameters for customization to your compute environment and data. These can all be specified on the command line using the standard syntax --argument="value" or --argument "value". You can also use any option supported by Nextflow itself. Note that certain arguments (i.e. those interpreted by Nextflow) use a single dash instead of two.

Environment Variables

You can export variables into your environment before calling the pipeline, or set them on the same line like this:

SINGULARITY_TMPDIR=/opt/tmp ./examples/demo_small.sh /opt/demo_small

Variable	Default	Description
TMPDIR	/tmp	Directory used for temporary files by certain processes.
SINGULARITY_TMPDIR	/tmp	Directory where Docker images are downloaded and converted to Singularity Image Format. Needs to be large enough to accommodate several GB, so moving it out of /tmp is sometimes necessary.

Data

Describe your input data and where pipeline results should be saved

Parameter	Description	Help Text	Type
data_manifest	Name or path to the file manifest for downloading input data. Default: segmentation	If specified, the data in the manifest is downloaded into --data_dir before the pipeline begins. Valid values are any base filename found in the data-sets directory (e.g. “demo_small”, “demo_medium”) or any absolute path which points to a manifest file. By default this just downloads the segmentation model.	string
verify_md5	Verify MD5 sum for all downloads. Default: true	This can be disabled to save time, but it’s not recommended.	string
shared_work_dir	Shared working directory accessible by all nodes. Typically something like /fsx/username/pipeline	Setting this parameter will automatically configure data_dir, output_dir, segmentation_model_dir, spark_work_dir, and singularity_cache_dir. You can override any of them in the hidden settings. When running on a system like AWS Batch, you should set this to an FSx for Lustre filesystem, and the final_output_dir to a Fuse-mounted S3 bucket. This will cause all processing to happen on high-performance disk, and the outputs will only be copied to slower S3 at the very last step.
string
data_dir	Path to the directory containing the input CZI/MVL acquisition files. If shared_work_dir is defined, this defaults to $shared_work_dir/inputs.	If shared_work_dir is defined, this is automatically set to $shared_work_dir/inputs.	string
segmentation_model_dir	Path to the directory containing the machine learning model for segmentation.	If shared_work_dir is defined, this is automatically set to $shared_work_dir/inputs/model/starfinity. It is assumed that either the model is already there, or it will be downloaded and unzipped according to the data_manifest. Otherwise it defaults to ${projectDir}/external-modules/segmentation/model/starfinity, which is normally configured by setup.sh.	string
output_dir	Path to the directory containing pipeline outputs. If shared_work_dir is defined, this defaults to $shared_work_dir/outputs.	If shared_work_dir is defined, this is automatically set to $shared_work_dir/outputs.	string
publish_dir	Optional publishing directory where results should be copied when the pipeline is successfully completed. Typically a Fusion mount path like /fusion/s3/bucket-name.	This is useful for getting data off of FSx and onto something externally accessible like S3.	string
acq_names	Names of acquisition rounds to process. These should match the names of the CZI/MVL files found in the data_dir.	e.g. LHA3_R3_small,LHA3_R5_small if you have files called LHA3_R3_small.czi and LHA3_R5_small.czi	string
ref_acq	Name of the acquisition round to use as the fixed reference.	e.g. LHA3_R3_small	string
channels	List of channel names to process.	Channel names are specified in the format “c[channel_number]”, where the channel_number is 0-indexed.	string
dapi_channel	Name of the DAPI channel.	The DAPI channel is used as a reference channel for registration, segmentation, and spot extraction.	string
bleed_channel	Channel (other than DAPI) that needs bleedthrough correction.		string

Stitching

Stitching options

Parameter	Description	Help Text	Type
stitching_output	Output directory for stitching results. Default: stitching	This directory path is relative to output_dir	string
spark_work_dir	Path to directory containing Spark working files and logs during stitching. Default: $shared_work_dir/spark or $workDir/spark	The Spark configuration is written here by the pipeline before launching the Spark cluster. The Spark workers write their logs back here, and it is also used to communicate the master IP address to all workers. Therefore, this must be a shared directory accessible to both the head node and all worker nodes. On AWS, Fuse-mounted S3 will not work here due to write buffering. It’s best to use FSx, but EBS will also work, as long as its mounted on all the EC2 nodes.	string
spark_local_dir	Path to directory that Spark will uses for local temporary files. Default: /tmp	This path does not need to be shared among workers, and does not need to be accessible to the head node. Usually, /tmp will do.	string
stitching_czi_pattern	A suffix pattern that is applied to acq_names when creating CZI names e.g. “_V%02d”		string
stitching_ref	Index of the channel used for stitching, e.g. ‘c1’ or ‘1’. You can also specify ‘all’ to use all of the channels. Default: the dapi_channel	If this is not defined it defaults to dapi_channel	string
resolution	Voxel resolution in all 3 dimensions. Default: 0.23,0.23,0.42	This is a comma-delimited tuple as x,y,z.	string
axis	Axis mapping for the objective->pixel coordinates conversion. Default: -x,y,z	Comma-separated axis specification with optional flips.	string
stitching_block_size	Block size to use when converting CZI to n5 before stitching. Default: 128,128,64		string
flatfield_correction	Apply flatfield correction before stitching? Default: true		boolean
retile_z_size	Block size (in Z dimension) when retiling after stitching. Default: 64	This must be smaller than the number of Z slices in the data.	integer
with_fillBackground	Use fillBackground option when running fuse step. Default: true	Turning this off may help process certain types of data that error otherwise.	boolean
stitching_mode	Rematching mode (‘full’ or ‘incremental’). Default: incremental		string
stitching_padding	Padding for the overlap regions. Default: 0,0,0		string
stitching_blur_sigma	Sigma value of the gaussian blur preapplied to the images before stitching. Default: 2		integer
workers	Number of Spark workers to use for stitching one acquisition. Default: 4		integer
worker_cores	Number of cores allocated to each Spark worker. Default: 4		integer
gb_per_core	Size of memory (in GB) that is allocated for each core of a Spark worker. Default: 4	The total memory usage for stitching one acquisition will be workers worker_cores gb_per_core.	integer
driver_memory	Amount of memory to allocate for the Spark driver. Default: 15g		string
wait_for_spark_timeout_seconds	Number of seconds to wait for Spark cluster to start. Default: 3600		integer
sleep_between_timeout_checks_seconds	Number of seconds to sleep between timeout checks. Default: 2		integer
stitching_app	Path to the JAR file containing the stitching application. Default: /app/app.jar		string

Registration

Options for the registration algorithm (Bigstream)

Parameter	Description	Help Text	Type
registration_output	Output directory for registration results. Default: registration	This path is relative to output_dir.	string
aff_scale	The scale level for affine alignments. Default: s3		string
def_scale	The scale level for deformable alignments. Default: s2		string
spots_cc_radius	Default: 8		integer
spots_spot_number	Default: 2000		integer
ransac_cc_cutoff	Default: 0.9		number
ransac_dist_threshold	Default: 2.5		number
deform_iterations	Default: 500x200x25x1		string
deform_auto_mask	Default: 0		string
registration_xy_stride	The number of voxels along x/y for registration tiling. Default: 256	Must be power of 2.	integer
registration_xy_overlap	Tile overlap on x/y axes	Defaults to registration_xy_stride/8 when not specified.	integer
registration_z_stride	The number of voxels along z for registration tiling. Default: 256	Must be power of 2.	integer
registration_z_overlap	Tile overlap on Z axes	Defaults to registration_z_stride/8 when not specified.	integer
ransac_cpus	Number of CPU cores for RANSAC. Default: 1		integer
ransac_memory	Amount of memory for RANSAC. Default: 1 G		string
spots_cpus	Number of CPU cores for Spots step of registration. Default: 1		string
spots_memory	Amount of memory for Spots step of registration. Default: 2 G		string
interpolate_cpus	Number of CPU cores for Interpolate step of registration. Default: 1		integer
interpolate_memory	Amount of memory for Interpolate step of registration. Default: 1 G		string
coarse_spots_cpus	Number of CPU cores for Coarse Spots step of registration. Default: 1		integer
coarse_spots_memory	Amount of memory for Coarse Spots step of registration. Default: 2 G		string
aff_scale_transform_cpus	Number of CPU cores for Affine Scale Transform step of registration. Default: 1		integer
aff_scale_transform_memory	Amount of memory for Affine Scale Transform step of registration. Default: 15 G		string
def_scale_transform_cpus	Number of CPU cores for deformable scale registration. Default: 8		integer
def_scale_transform_memory	Amount of memory for Deformable Scale Transform step of registration. Default: 80 G		string
deform_cpus	Number of CPU cores for Deform step of registration. Default: 1		integer
deform_memory	Amount of memory for Deform step of registration. Default: 10 G		string
registration_stitch_cpus	Number of CPU cores for Stitch step of registration. Default: 2		integer
registration_stitch_memory	Amount of memory for Stitch step of registration. Default: 20 G		string
registration_transform_cpus	Number of CPU cores for final Transform step of registration. Default: 12		integer
registration_transform_memory	Amount of memory for final Transform step of registration. Default: 80 G		string

Cell Segmentation

Options for the cell segmentation algorithm (Starfinity)

Parameter	Description	Help Text	Type
segmentation_output	Output directory for segmentation results. Default: segmentation	This path is relative to output_dir.	string
segmentation_scale	Imagery scale to use for segmentation. Default: s2		string
segmentation_cpus	Number of CPU cores for segmentation. Default: 3		integer
segmentation_memory	Amount of memory for segmentation. Default: 45 G		string

Spot Extraction

Options for spot extraction

Parameter	Description	Help Text	Type
spot_extraction_output	Output directory for spot extraction results. Default: spots	This path is relative to output_dir.	string
spot_extraction_scale	Scale of imagery to use for spot extraction. Default: s0		string

Spot Extraction: Airlocalize

Options for the AirLocalize spot extraction algorithm

Parameter	Description	Help Text	Type
airlocalize_xy_stride	The number of voxels along x/y for registration tiling. Default: 1024	Must be power of 2. Increasing this requires increasing airlocalize_memory.	integer
airlocalize_xy_overlap	Tile overlap on x/y axes	Defaults to 5% of airlocalize_xy_stride	integer
airlocalize_z_stride	The number of voxels along Z for registration tiling. Default: 512	Must be a power of 2. Increasing this requires increasing airlocalize_memory.	integer
airlocalize_z_overlap	Tile overlap on z axes	Defaults to 5% of airlocalize_z_stride	integer
default_airlocalize_params	Path to the default AirLocalize parameter file. Default: /app/airlocalize/params/air_localize_default_params.txt	By default, this points to default parameters inside the container	string
per_channel_air_localize_params	Comma-delimited paths to alternative AirLocalize parameter files, one per channel.	If you have 4 channels, and you are extracting spots from c0, c1, and c3, this parameter should look like this: /path/to/params_c0.txt,/path/to/params_c1.txt,,/path/to/params_c3.txt. Note the double comma to denote the empty file for c2, which should not be processed.	string
airlocalize_cpus	Number of CPU cores to allocate for each AirLocalize job. Default: 1		integer
airlocalize_memory	Amount of RAM to allocate to each AirLocalize job. Needs to be increased when increasing strides. Default: 2 G		integer

Spot Extraction: RS-FISH

Options for the RS-FISH spot extraction algorithm

Parameter	Description	Help Text	Type
use_rsfish	Use RS-FISH instead of AirLocalize for Spot Extraction. Default: false		boolean
rsfish_min	Minimal intensity of the image. Default: 0		integer
rsfish_max	Maximal intensity of the image. Default: 4096		integer
rsfish_anisotropy	The anisotropy factor. Default: 0.7	Scaling of z relative to xy. Can be determined using the RS-FISH anisotropy plugin in Fiji.	number
rsfish_sigma	Sigma value for Difference-of-Gaussian (DoG) calculation. Default 1.5		number
rsfish_threshold	Threshold value for Difference-of-Gaussian (DoG) calculation. Default: 0.007		number
rsfish_params	Any other parameters to pass to the RS-FISH algorithm.	Complete parameter documentation for RS-FISH is available here.	string
rsfish_workers	Number of Spark workers to use for RS-FISH spot detection. Default: 4		integer
rsfish_worker_cores	Number of cores allocated to each RS-FISH Spark worker. Default: 4		integer
rsfish_gb_per_core	Size of memory (in GB) that is allocated for each core of a RS-FISH Spark worker. Default: 4	The total memory usage for one acquisition will be workers worker_cores gb_per_core.	integer
rsfish_driver_cores	Number of cores allocated for the RS-FISH Spark driver. Default: 1		string
rsfish_driver_memory	Amount of memory to allocate for the RS-FISH Spark driver. Default: 15g		string

Per channel RS-FISH Parameters

The following parameters can be set per channel: rsfish_min, rsfish_max, rsfish_anisotropy, rsfish_sigma, rsfish_threshold. For that simply prefix the corresponding parameter with --per_channel. and set the values using a comma delimited list. The values will be associated with the corresponding channel based on their position, i.e. first value will be associated with the first channel, second with the second channel, etc. If a value is missing or empty the parameter value for the channel will be set to the default from the parameter with the same name (presented above).

For example if the command like is: --channels c0,c1,c2,c3 --sigma 1.7 --per_channel.sigma "1.2,,1.4

channel c0 will use sigma 1.2

channel c1 will use the default sigma 1.7 (because of the empty value)

channel c2 will use sigma 1.4

channel c3 will use the default sigma 1.7 (because of the missing value - sigma values list is shorter then the channels list)

Spot Warping

Options for warping detected spots to registration

Parameter	Description	Help Text	Type
warp_spots_cpus	Number of CPU cores to use for warp spots. Default: 2		integer
warp_spots_memory	Amount of memory for warp spots. Default: 30 G		string

Intensity Measurement

Options for extracting quantified measurements of spot intensities

Parameter	Description	Help Text	Type
measure_intensities_output	Output directory for intensities. Default: intensities	This path is relative to output_dir.	string
measure_intensities_cpus	Number of CPU cores to use for intensity measurement. Default: 1		integer
measure_intensities_memory	Amount of memory for intensity measurement. Default: 8 G		string

Spot Assignment

Options for mapping spot counts to segmented cells

Parameter	Description	Help Text	Type
assign_spots_output	Output directory for spot assignments. Default: assignments	This path is relative to output_dir.	string
assign_spots_cpus	Number of CPU cores to use for spot assignment. Default: 1		integer
assign_spots_memory	Amount of memory for spot assignment. Default: 5 G		string

Container Options

Customize the Docker containers used for each pipeline step

Parameter	Description	Help Text	Type
mfrepo	Docker registry/repository to use for containers. Default: janeliascicomp	By default, the pipeline uses containers built as part of this project and deployed to DockerHub. You can rebuild the containers and deploy them to your own Registry and specify it here.	string
spark_container_repo	Docker container repo for stitching. Default: <mfrepo>		string
spark_container_name	Docker container name for stitching. Default: stitching		string
spark_container_version	Docker container version for stitching. Default: 1.0.0		string
registration_container	Docker container for running registration and warp_spots. Default: <mfrepo>/registration:1.2.0		string
segmentation_container	Docker container for running segmentation. Default: <mfrepo>/segmentation:1.0.0		string
airlocalize_container	Docker container for running spot extraction. Default: <mfrepo>/airlocalize:1.0.2		string
spots_assignment_container	Docker container for running intensity measurement and spot assignment. Default: <mfrepo>/spot_assignment:1.2.0		string

Other Options

Other global options affecting all pipelines stages

Parameter	Description	Help Text	Type
skip	Comma-delimited list of steps to skip, e.g. stitching,registration.	Valid values: stitching,spot_extraction,segmentation,registration,warp_spots,measure_intensities,assign_spots	string
singularity_cache_dir	Shared directory where Singularity containers are cached. Default: $shared_work_dir/singularity_cache or $HOME/.singularity_cache		string
singularity_user	User to use for running Singularity containers. Default: $USER	This is automatically set to ec2-user when using the ‘tower’ profile	string
runtime_opts	Runtime options for the container engine being used (e.g. Singularity or Docker).	Runtime options for Singularity must include mounts for any directory paths you are using. You can also pass the –nv flag here to make use of NVIDIA GPU resources. For example, --nv -B /your/data/dir -B /your/output/dir
string
lsf_opts	Options for LSF cluster at Janelia, when using the lsf profile.		string