reveal.js

<style>

body {
  background: #001;
  background-color: #001; 
}

.reveal h4 {
  text-transform: none;
}

.bullets {
	width:100%;
	border:0px;
}

.bullets tr td{
	border:0px;
	padding:0px;
	vertical-align: middle;
}

.bullets img {
	padding-right:50px;
}

.rowshow {
	width:100%;
	text-align:left;
	vertical-align: middle;
	margin-top: 50%;
}

.rowshow .image {
	vertical-align: middle;
	margin-right: 80px;
	float:left;
}

.rowshow .image img {
	margin-top:50%
}

.rowshow .text {
	margin-top:50%
}

.reveal {
  font-family: 'Source Sans Pro', Helvetica, sans-serif;
  font-size: 36px;
  font-weight: normal;
  color: #fff; }

.reveal h1, .reveal h2{
    text-transform: none;
}

.reveal h5 {
    text-transform: none;
    border-bottom: 1px solid #AAAAFF;
    font-weight: 900;
    color: #AAAAFF;
    margin-bottom:1px;
    margin-top:8px;
}

.reveal ul li {
  margin-bottom:18px;
}

em {
  color: #aaffaa;
}

.reveal .slides .header{
  position:absolute;
  top: -50%;
  left: -50%;
}
.reveal .slides .footer{
  position:absolute;
  bottom: 0%;
  color:green;
  left: -50%;
}
.reveal .well{
  color:white;
  min-height:20px;padding:19px;margin-bottom:20px;background-color:#444;
  border:1px solid #ccc;
  font-size:80%;

}
  .reveal .well em {
    color: #aaffaa;
  }

.col3 {
    float: left;
    width: 33.33333333%;
 }

.col2 {
   float: left;
   width: 50%;
   margin-bottom:60px;
}

.reveal .small{
   font-size:.7em;
 }

.reveal .footnote{
   text-align:right;
 }

.reveal .title {
    text-align:left;
}
.well blockquote{border-color:#ddd;border-color:rgba(0,0,0,.15)}
.well-lg{padding:24px;border-radius:6px}
.well-sm{padding:9px;border-radius:3px}

.reveal .bullet {
  vertical-align:middle;
  padding-right:5px;
}

.reveal .bullet img {
  vertical-align: middle;
}

.reveal ul li {
  margin-bottom: 0px;
}

</style>

<section id="title" data-background="/images/presentations/bg.svg.png" data-transition-speed="slow">
<h2>Building computational analysis pipelines</h2>

Nathan Sheffield, PhD

<br>
<div class="bullet"><img src="/images/logo/University_of_Virginia_Rotunda_logo.svg" height="50"><img src="/images/logo/University_of_Virginia_logo_white.svg" height="55">
<img src="/images/logo/UVA_CPHG_dark.svg" height="75"><img src="/images/logo/logo_databio_long.svg" height="55"></div>
<span style="font-size:0.6em"><a href="http://www.databio.org/slides">www.databio.org/slides</a></span>
<br>

</section>

---

# Talk outline

1. What is a pipeline?
2. Interactive computing to shell scripts
4. Bioinformatics pipeline frameworks
5. Modular pipeline engineering

---

# What is a pipeline?

> A series of commands to run on input data
> to produce some result

```text
$ bowtie2 ...
$ trimmomatic ...
$ mac2s ...
```

<div class="fragment">
<div class="col2">workflow</div>
<div class="col2">pipeline</div>
</div>

<div class="fragment">
Often considered synonyms. To me:

'workflow' emphasizes *reproducibility*  
'pipeline' emphasizes *reusability*  
</div>
---

# reusability and reproducibility

*reproducibility*: you can re-run the analysis with the same inputs to get the same outputs.
		
*reusability*: you can re-use the analysis with different inputs to get compatible (but different) outputs.

---
# Analysis spectrum

Is one inherently *best*? no.  
Each has pros and cons and applies best to different needs.

</div>

---

<table class="bullets">
<tr>
	<td width="135"><img src="/images/characters/jordan_smile.svg" align="left" width="125" /></td>
	<td>Jordan got data from the sequencer today.</td>
</tr>
</table>

<table class="bullets fragment" style="border:0px solid black; border-collapse: collapse;">
<tr>
	<td width="135"><img src="/images/icons/console.svg" align="left" width="125" /></td>
	<td>He sits down at the terminal to process it.
<span>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">bowtie2 sample.fastq ...
trimmomatic ...
macs2 output.fastq ...</code></pre></figure>
</span>
</td>
</tr>
</table>

</section>

<section>
Two weeks later...
</section>
<section>
<table class="bullets">
<tr>
	<td width="135"><img src="/images/characters/jordan_smile.svg" width="125" /></td>
	<td>Jordan got data from the sequencer today.</td>
</tr>
</table>
<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/icons/console.svg" align="left" width="125" /></td>
	<td>He sits down at the terminal to process it.</td>
</tr>
</table>
<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/characters/jordan_confused.svg" align="left" width="125" /></td>
	<td>Hmm... what did I do last time?</td>
</tr>
</table>
</section>

<table class="bullets">
<tr>
	<td width="135"><img src="/images/characters/jordan_idea.svg" width="125" /></td>
	<td>Jordan has an idea.</td>
</tr>
</table>

<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/icons/app.svg" width="125" /></td>
	<td>He sits down at the terminal to <span style="text-decoration: line-through;">process</span> <b>script</b> it.</td>
</tr>
</table>

</section>

<section>
Two weeks later...
</section>

<section>
<table class="bullets">
<tr>
	<td width="135"><img src="/images/characters/jordan_smile.svg" width="125" /></td>
	<td>Jordan got data from the sequencer today.</td>
</tr>
</table>
<table class="bullets">
<tr>
	<td width="135"><img src="/images/icons/file_shell_script.svg" width="125" /></td>
	<td>He gets the script running...</td>
</tr>
</table>
<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/characters/server-crash.svg" width="125" /></td>
	<td>The server crashes. It would be nice if the script could pick up where it left off...</td>
</tr>
</table>
</section>

<section>
<table class="bullets">
<tr>
	<td width="135"><img src="/images/characters/jordan_smile.svg" width="125" /></td>
	<td>Now Jordan has 500 samples for a time series experiment.</td>
</tr>
</table>
<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/icons/refresh.svg" width="125" /></td>
	<td>He starts writing some looping functions to handle cluster submission.</td>
</tr>
</table>

<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/icons/clock.svg" width="125" /></td>
	<td>This is going to take awhile...</td>
</tr>
</table>

</section>

<section>
<table class="bullets">
<tr>
	<td width="135"><img src="/images/characters/jordan_smile.svg" width="125" /></td>
	<td>In the meantime, Jordan generates other samples requiring slightly different parameters.</td>
</tr>
</table>
<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/icons/file_shell_script2.svg" width="125" /></td>
	<td>No problem, I'll just duplicate this script...</td>
</tr>
</table>
<table class="bullets fragment">
<tr>
	<td width="135"><img src="/images/characters/lily_open.svg" width="125" /></td>
	<td>Stop! There is a better way...</td>
</tr>
</table>
</section>

---

# Challenges with interactive computing
- low reproducibility
- low reusability
- no restartability
- no maintainability

---

# Challenges with shell pipelines
- only some of reproducibility solved
- only some of reusability solved
- still no restartability
- failed steps do not halt the entire pipeline
- difficult to scale to 500 samples
- two pipelines running simultaneously may interfere
- tracking which version was used with which samples
- memory use is left unmonitored and unchecked
- difficult to maintain
- no record of the output produced by default

---
# Analysis spectrum

---

# Pipeline frameworks

https://github.com/pditommaso/awesome-pipeline

---

# Common frameworks used in bioinformatics

1. Snakemake
2. Nextflow
3. CWL

---

# Snakemake

```text
SAMPLES = ["A", "B"]

rule all:
    input:
        "plots/quals.svg"

rule bwa_map:
    input:
        "data/genome.fa",
        "data/samples/{sample}.fastq"
    output:
        "mapped_reads/{sample}.bam"
    shell:
        "bwa mem {input} | samtools view -Sb - > {output}"

rule samtools_sort:
    input:
        "mapped_reads/{sample}.bam"
    output:
        "sorted_reads/{sample}.bam"
    shell:
        "samtools sort -T sorted_reads/{wildcards.sample} "
        "-O bam {input} > {output}"

rule samtools_index:
    input:
        "sorted_reads/{sample}.bam"
    output:
        "sorted_reads/{sample}.bam.bai"
    shell:
        "samtools index {input}"

rule bcftools_call:
    input:
        fa="data/genome.fa",
        bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
        bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
    output:
        "calls/all.vcf"
    shell:
        "bcftools mpileup -f {input.fa} {input.bam} | "
        "bcftools call -mv - > {output}"

rule plot_quals:
    input:
        "calls/all.vcf"
    output:
        "plots/quals.svg"
    script:
        "scripts/plot-quals.py"

```

---

# Nextflow

```text
#!/usr/bin/env nextflow
params.in = "$HOME/sample.fa"
sequences = file(params.in)
SPLIT = (System.properties['os.name'] == 'Mac OS X' ? 'gcsplit' : 'csplit')

process splitSequences {
    input:
    file 'input.fa' from sequences
    output:
    file 'seq_*' into records
    """
    $SPLIT input.fa '%^>%' '/^>/' '{*}' -f seq_
    """
}

process reverse {
    input:
    file x from records
    output:
    stdout result
    """
    cat $x | rev
    """
}
result.subscribe { println it }
```

---
# Common Workflow Language (CWL)

```text
cwlVersion: v1.2

# What type of CWL process we have in this document.
class: CommandLineTool
# This CommandLineTool executes the linux "echo" command-line tool.
baseCommand: echo

# The inputs for this process.
inputs:
  message:
    type: string
    # A default value that can be overridden, e.g. --message "Hola mundo"
    default: "Hello World"
    # Bind this message value as an argument to "echo".
    inputBinding:
      position: 1
outputs: []
```

---

# Challenges with pipeline frameworks
- learning curve
- evaluation: how do you pick?
- lock-in
- lack of interoperability
<li class="fragment">all-or-none adoption required</li>

---

# Modular pipeline engineering

---

<section>
<h4>Most pipelines require individual metadata organization</h4>
<div class="col2">
<img src="/images/presentations/pep/data_input-new_white.svg" width="375">
</div>
<div class="col2 fragment">
<img src="/images/presentations/pep/data_input-rev_white.svg" width="375">
</div>
</section>

<section>
<h3>What if?</h3>
<div class="col2">
<img src="/images/presentations/pep/data_input2.svg" width="325">
</div>
<div class="col2">
<img src="/images/presentations/pep/data_input2_rev.svg" width="325">
</div>
<div class="fragment">
Why is this hard to do?
<br>Because of <i>microwave syndrome</i>....
</div>
</section>

<section>
<h4>Microwave syndrome</h4>
<div>
<img src="/images/web/IFB_17PM-MEC1.png" height="180">
<img src="/images/web/LG_LMV2031SB.png" height="180">
<img src="/images/web/panasonic_NN-CT585SBPQ.png" height="180">
</div>

<div class="well fragment">In user interface design, prioritizing easy access to integrated functions over their individual components. </div>
</section>

<section>
<h3>The UNIX philosophy</h3>
<div class="col2">
<img src="/images/web/unix_book.jpg" height="450">
</div>
<div class="col2">
<div class="well"><span style="color:#ffb; font-weight:bold">[T]he power of a system comes more from the relationships among programs than from the programs themselves.</span><br><br>
<span style="font-size: 0.8em">Many UNIX programs do quite trivial tasks in isolation, but, combined with other programs, become general and useful tools.</span><br/><br/>
<span class="small">- Kernighan and Pike, The UNIX Programming Environment (1983, p. viii)</span>
</div>
</div>
</section>

---

# Ecosystem of modules for pipeline building

- [PEP](https://pep.databio.org) -- standardizing representation of sample metadata
- [geofetch](https://geofetch.databio.org) -- retrieving data from GEO
- [pypiper](https://pypiper.databio.org) -- building simple, sequential Python pipelines
- [pipestat](https://pipestat.databio.org) -- reporting and storage of pipeline results
- [refgenie](https://refgenie.databio.org) -- access to reference genome resources
- [looper](https://looper.databio.org) -- processing multi-sample projects, job submission
- [divvy](https://looper.databio.org) -- configuring compute cluster resources
- [bulker](https://bulker.io) -- distributing independent, containerized computing environments

---

<div class="col2">
<h3>Problem</h3>
<img src="/images/presentations/pep/data_input-new_white.svg" width="375">
</div>

<div class="col2 fragment">

<h3>Solution</h3>
<img src="/images/presentations/pep/data_input_plug.svg" width="375">
</div>

---

<section>
<h4>PEP: A standard format for project metadata</h4>
<img src="/images/presentations/pep/pep_center_white.svg" width="700">
</section>

---

<div class="bullet">
<h2><img src="/images/presentations/pep/pep_logo.svg" width="70">PEP format</h2>
</div>
<div class="bullet">
<img src="/images/icons/file.svg" width="30"> project_config.yaml
</div>

```yaml
pep_version: 2.0
sample_table: /path/to/samples.csv
... (project-level metadata)
```

<hr>
<div class="bullet">
<img src="/images/icons/file.svg" width="30"> samples.csv
</div>

```text
sample_name, protocol, organism, data_source
frog_0h, RNA-seq, frog, /path/to/frog0.gz
frog_1h, RNA-seq, frog, /path/to/frog1.gz
frog_2h, RNA-seq, frog, /path/to/frog2.gz
frog_3h, RNA-seq, frog, /path/to/frog3.gz
```

---

<section>
<div>
<img src="/images/logo/logo_pypiper.svg" width="250">
<h3>Pypiper</h3>
</div>

Builds a pipeline for a single sample.

<div class="small">
GitHub: <a href="http://github.com/databio/pypiper">http://github.com/epigen/pypiper</a><br>
Documentation: <a href="https://pypiper.databio.org">https://pypiper.databio.org</a><br>
</div>
</section>

<section>
<h3>Pypiper features</h3>
	<div class="col2" style="text-align: left">
	<span class="bullet"><img src="/images/icons/yinyang.svg" width="50" class="bullet">Simplicity</span><br>
	<span class="bullet"><img src="/images/icons/refresh.svg" width="50" class="bullet">Restartability</span><br>
	<span class="bullet"><img src="/images/icons/lock.svg" width="50" class="bullet">File integrity lock</span><br>
	<span class="bullet"><img src="/images/icons/increase.svg" width="50" class="bullet">Memory monitoring</span><br>
	</div><div class="col2" style="text-align: left">
	<span class="bullet"><img src="/images/icons/magnify.svg" width="50" class="bullet">Job monitoring</span><br>
	<span class="bullet"><img src="/images/icons/redx.svg" width="50" class="bullet">Robust error handling</span><br>
	<span class="bullet"><img src="/images/icons/file_log.svg" width="50" class="bullet">Automatic logging</span><br>
	<span class="bullet"><img src="/images/icons/printer.svg" width="50" class="bullet">Easy result reports</span><br>
	<span class="bullet"><img src="/images/icons/collate.svg" width="50" class="bullet">Collate input files</span><br>
	</div>
</section>

<section>
<span class="bullet"><img src="/images/icons/yinyang.svg" width="50" class="bullet">Simplicity</span><br>
Bash script:

```bash
shuf -i 1-500000000 -n 10000000 > outfile.txt
```

Pypiper script:

```python
pm.run("shuf -i 1-500000000 -n 10000000 > outfile.txt")
```

Using pypiper is as easy as writing a shell script. <br><br>Additional <i>options</i> provide power <i>on demand</i>.
</section>

<section>
<span class="title"><img src="/images/icons/refresh.svg" width="50" class="bullet">Restartability</span><br clear="all"/>

```python
target = os.path.join(outfolder, "outfile.txt")  # output file
cmd = "shuf -i 1-500000000 -n 10000000 > " + target
pm.run(command, target)
```

Commands (optionally) only run if target does not already exist.
<p>Pipeline will thus pick up where it left off.</p>
</section>

<section>
<span class="bullet"><img src="/images/icons/lock.svg" width="50" class="bullet">File integrity lock</span><br>
<b>Lock files</b> ensure commands only run if the target is unlocked:

<ul>
<li>pipelines will not proceed with incomplete files</li>
<li>multiple pipelines can create/use the same files</li>
</ul>
</section>

<section>
<span class="bullet"><img src="/images/icons/magnify.svg" width="50" class="bullet">Job monitoring</span><br>Pypiper uses a flag system to track status<br>
<span class="bullet"><img src="/images/icons/flag_green.svg" width="50" class="bullet"> Job running   
<img src="/images/icons/flag_checker.svg" width="50" class="bullet"> Job completed  
<img src="/images/icons/flag_red.svg" width="50" class="bullet"> Job failed</span>
<br><br>
Summarizing jobs is easy: just count the flags<br>
<img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_red.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><br><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_green.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_green.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><img src="/images/icons/flag_checker.svg" width="30"><br>
<span class="bullet"><img src="/images/icons/flag_green.svg" width="50" class="bullet"> = 2   
<img src="/images/icons/flag_checker.svg" width="50" class="bullet"> = 17 
<img src="/images/icons/flag_red.svg" width="50" class="bullet"> =1</span>
</section>

<section>
<span class="bullet"><img src="/images/icons/redx.svg" width="50" class="bullet">Robust error handling</span><br>
	<img src="/images/presentations/pypiper/fail_pipeline.svg" height="400"><br>
If a process fails, the pipeline fails.
</section>

<section>
<span class="bullet"><img src="/images/icons/file_log.svg" width="50" class="bullet">Automatic logging</span><br>
<img src="/images/presentations/pypiper/log_split.svg" height="400"><br>
Output is automatically split to screen and file.
</section>

<section>
<span class="bullet"><img src="/images/icons/printer.svg" width="50" class="bullet">Easy result reports</span><br>

```python
reads = count_reads(unaligned_file)
aligned = count_reads(aligned_file)
pm.report_result("aligned_reads", aligned)
pm.report_result("alignment_rate", aligned/reads)
```

Output:

```tsv
aligned_reads	2526232
alignment_rate	0.64234
```

</section>

<section>
Example pipeline

```python
import pypiper, os
outfolder = "pipeline_output/"  # folder for results
pm = pypiper.PipelineManager(name="shuf", outfolder)

target = os.path.join(outfolder, "outfile.txt")  # output file
command = "shuf -i 1-500000000 -n 10000000 > " + target
pm.run(command, target)

pm.stop_pipeline()
```

</section>

---

<section>
<div>
<img src="/images/logo/logo_looper.svg" width="150">
<h3>Looper</h3>

Deploys pipelines across samples by connecting<br>
samples to any command-line tool

<div class="small">
<a href="https://looper.databio.org">https://looper.databio.org</a><br>
</div>
</div>
</section>

<!-- <section>
<h3>Parallelism Philosophy</h3>
<img src="/images/concepts/parallel_sequential.svg" width="100%"><br>
<div class="fragment">
<div class="col3" style="background-color:#211">by process
<img src="/images/concepts/parallel_process.svg" width="300">
</div>
<div class="col3" style="background-color:#112">by sample
<img src="/images/concepts/parallel_sample.svg" width="300">
</div>
<div class="col3" style="background-color:#121">by dependence
<img src="/images/concepts/parallel_dependency.svg" width="300">
</div>
</div>

</section>

-->

<section>
<span class="bullet">pipeline_interface.yaml</span><br>

```yaml
pipeline_name: count_lines
pipeline_type: sample
var_templates:
  pipeline: {looper.piface_dir}/count_lines.sh
command_template: {pipeline.var_templates.pipeline} {sample.file}
```

Tells looper how to run an arbitrary command.

</section>

<section>
	<h3>Looper features</h3>
	<div class="col2" style="text-align: left">
		<span class="bullet"><img src="/images/icons/input-mouse.svg" width="50" class="bullet">Single-input runs</span><br>
		<span class="bullet"><img src="/images/icons/flexible.svg" width="50" class="bullet">Flexible pipelines</span><br>
		<span class="bullet"><img src="/images/icons/piechart.svg" width="50" class="bullet">Flexible resources</span><br>
	</div>
    <div class="col2" style="text-align: left">
		<span class="bullet"><img src="/images/icons/computer.svg" width="50" class="bullet">Flexible compute</span><br>
		<span class="bullet"><img src="/images/icons/flag_checker.svg" width="50" class="bullet">Job status-aware</span><br>
	</div>
</section>

<section>
<span class="bullet"><img src="/images/icons/input-mouse.svg" width="50" class="bullet">Single-input runs</span><br>

Run your entire project with one line:

```bash
looper run
```

</section>

<section>
<span class="bullet"><img src="/images/icons/piechart.svg" width="50" class="bullet">Flexible resources</span><br>

```yaml
pipeline_key:
  name: pipeline_name
  arguments:
    "--option" : value
  resources:
    default:
      file_size: "0"
      cores: "2"
      mem: "6000"
      time: "01:00:00"
    large_input:
      file_size: "2000"
      cores: "4"
      mem: "12000"
      time: "08:00:00"
```

Resources can vary by input file size
</section>

<section>
	<span class="bullet"><img src="/images/icons/computer.svg" width="50" class="bullet">Flexible compute</span><br>

```yaml
compute:
  slurm:
    submission_template: templates/slurm_template.sub
    submission_command: sbatch
  localhost:
    submission_template: templates/localhost_template.sub
    submission_command: sh
```

<hr>
Adjust compute package on-the-fly:

```yaml
> looper run --compute localhost
```

</section>
<section>
<span class="bullet"><img src="/images/icons/flag_checker.svg" width="50" class="bullet">Job status-aware</span><br>

Looper only submits jobs for samples not already flagged as running, completed, or failed.

```bash
looper check project_config.yaml
```

```bash
looper summarize project_config.yaml
```
</section>

---

# Looper CLI usage

```console
version: 1.5.0
usage: looper [-h] [--version] [--logfile LOGFILE] [--dbg] [--silent]
              [--verbosity V] [--logdev]
              {run,rerun,runp,table,report,destroy,check,clean,inspect,init,init-piface}
              ...

looper - A project job submission engine and project manager.

positional arguments:
  {run,rerun,runp,table,report,destroy,check,clean,inspect,init,init-piface}
    run                 Run or submit sample jobs.
    rerun               Resubmit sample jobs with failed flags.
    runp                Run or submit project jobs.
    table               Write summary stats table for project samples.
    report              Create browsable HTML report of project results.
    destroy             Remove output files of the project.
    check               Check flag status of current runs.
    clean               Run clean scripts of already processed jobs.
    inspect             Print information about a project.
    init                Initialize looper dotfile.
    init-piface         Initialize generic pipeline interface.
```

---

# Looper run CLI usage

```console
usage: looper run [-h] [-i] [-d] [-t S] [-x S] [-y S] [-f] [--divvy DIVCFG] [-p P] [-s S]

[-c K [K ...]] [-u X] [-n N] [--looper-config LOOPER_CONFIG]

[-S YAML [YAML ...]] [-P YAML [YAML ...]] [-l N] [-k N]
                  [--sel-attr ATTR] [--sel-excl [E ...] | --sel-incl [I ...]]
                  [-a A [A ...]]
                  [config_file]

Run or submit sample jobs.

positional arguments:
  config_file                        Project configuration file (YAML) or pephub registry
                                     path.

options:
  -h, --help                         show this help message and exit
  -i, --ignore-flags                 Ignore run status flags? Default=False
  -d, --dry-run                      Don't actually submit the jobs. Default=False
  -t S, --time-delay S               Time delay in seconds between job submissions
  -x S, --command-extra S            String to append to every command
  -y S, --command-extra-override S   Same as command-extra, but overrides values in PEP
  -f, --skip-file-checks             Do not perform input file checks
  -u X, --lump X                     Total input file size (GB) to batch into one job
  -n N, --lumpn N                    Number of commands to batch into one job

--looper-config LOOPER_CONFIG      Looper configuration file (YAML)

-S YAML [YAML ...], --sample-pipeline-interfaces YAML [YAML ...]
                                     Path to looper sample config file
  -P YAML [YAML ...], --project-pipeline-interfaces YAML [YAML ...]
                                     Path to looper project config file
  -a A [A ...], --amend A [A ...]    List of amendments to activate

divvy arguments:
  Configure divvy to change computing settings

--divvy DIVCFG                     Path to divvy configuration file. Default=$DIVCFG env
                                     variable. Currently: not set
  -p P, --package P                  Name of computing resource package to use
  -s S, --settings S                 Path to a YAML settings file with compute settings
  -c K [K ...], --compute K [K ...]  List of key-value pairs (k1=v1)

sample selection arguments:
  Specify samples to include or exclude based on sample attribute values

-l N, --limit N                    Limit to n samples
  -k N, --skip N                     Skip samples by numerical index
  --sel-attr ATTR                    Attribute for sample exclusion OR inclusion
  --sel-excl [E ...]                 Exclude samples with these values
  --sel-incl [I ...]                 Include only samples with these values
```

---

Standardized pipeline result reporting.<br>

<div class="small">
<a href="https://pipestat.databio.org">https://pipestat.databio.org</a><br>
</div>
</div>
</section>

---

---

# Advantages of the modular approach
- individual components may be used independently
- improved interoperability *across workflows*
- flexibility to fit different computing needs
- reduced lock-in; swap components as needed
- writing your own components is easier

---

# Moving to the next level

## If you're just getting started:
Script everything! (No interactive analysis allowed).

</div>

---

# Moving to the next level

## If you're already scripting everything:
Use a framework instead of just scripts

---

# Moving to the next level

## If you're already using a framework:
What do you think of modular pipelines?

---

<section id="acknowledgements" data-background="/images/presentations/bg.svg.png">
<h2>Thank You</h2>

<br clear="all"/>
<span class="small bullet"><img src="/images/logo/github_bug_black.svg" height="20" class="bullet"><a href="http://github.com/nsheff">nsheff</a></span> ·
<span class="small bullet"><img src="/images/icons/web.svg" height="25" class="bullet"><a href="http://databio.org">databio.org</a></span> ·
<span class="small bullet"><img src="/images/icons/letter.svg" height="25" class="bullet"><a href="mailto:nsheffield@virginia.edu">nsheffield@virginia.edu</a></span><br>

</section>