references/workflow-languages.md
Workflow Languages and Imports
Choose a Workflow Path
| Source | DNAnexus path |
|---|---|
| Existing apps/applets assembled visually or by API | Native workflow |
| WDL draft-2, 1.0, or 1.1 | dxCompiler |
| CWL 1.2 | dxCompiler |
| Nextflow source folder/repository | dx build --nextflow |
| One Python/Bash program | Applet/app, not a workflow |
Keep the source workflow, compiler/importer version, container digests, input JSON, and generated executable IDs together as provenance.
Native Workflows
Native workflows connect app/applet stages. Build from a dxworkflow.json:
dx build --workflow "path/to/workflow-source" \
--destination "project-xxxx:/workflows/my-workflow"
Inspect inputs:
dx run "workflow-xxxx" -h
Run:
dx run "workflow-xxxx" \
--input-json-file "inputs.json" \
--destination "project-xxxx:/runs/run-001" \
--cost-limit 50
This returns an analysis-... ID. Each stage runs as a job or nested analysis.
Use native workflows when:
- Components already exist as maintained DNAnexus apps/applets.
- Users need UI-editable stage connections.
- The pipeline is small enough to maintain as a native workflow definition.
Promote stable inputs/outputs to workflow-level fields. Keep project-qualified default links only when the referenced data is deliberately shared and accessible in every intended context.
WDL and CWL with dxCompiler
Current baseline: dxCompiler 2.18.0.
Supported language versions at that baseline:
- WDL draft-2
- WDL 1.0
- WDL 1.1
- CWL 1.2
WDL 2.0/development support is not production-ready. CWL 1.0 and 1.1 must be upgraded to 1.2 before compilation.
Setup
Prerequisites:
- Current
dxtoolkit and authenticated project context - dxCompiler 2.18.0 JAR from the official GitHub release
- Java 8 or 11
- Docker only if using the documented containerized compiler workflow
Pin the JAR release and verify release checksums or provenance before use. Do not use an unversioned download URL in production automation.
Validate WDL
dxCompiler uses strict WDL parsing. Validate and lint with the matching wdlTools release before compilation.
Then compile:
java -jar "dxCompiler-2.18.0.jar" compile \
"workflow.wdl" \
-project "project-xxxx" \
-folder "/workflows" \
-inputs "inputs.json"
The compile command can create a DNAnexus-formatted input JSON. Run the
compiled workflow with dx run and the generated input file:
dx run "workflow-xxxx" \
--input-json-file "inputs.dx.json" \
--destination "project-xxxx:/runs/run-002" \
--cost-limit 50
Run java -jar dxCompiler-2.18.0.jar help for the exact options supported by
that pinned release.
Authenticated WDL imports
dxCompiler 2.17.0 and later support per-domain bearer tokens for protected HTTP(S) WDL
imports through DXCOMPILER_WDL_IMPORT_BEARER_TOKENS.
This variable is highly sensitive:
- Inject only through a secret manager.
- Scope each token to the required import domain.
- Never print or log the variable.
- Never accept an arbitrary import host while credentials are active.
- Remove the variable after compilation.
- Do not embed it in workflow source, inputs, or extras.
Prefer immutable source references and checksum imported WDL dependencies.
Prepare CWL
dxCompiler expects one packed CWL 1.2 document.
- Upgrade older CWL to 1.2.
- Pack imports into one compound document.
- Validate with
cwltool --validate. - Compile the packed file.
cwltool --validate "workflow.cwl.json"
java -jar "dxCompiler-2.18.0.jar" compile \
"workflow.cwl.json" \
-project "project-xxxx" \
-folder "/workflows"
Pin cwltool, the upgrader/packer, and dxCompiler in reproducible
environments.
Known constraints
Review the current dxCompiler README and release notes before migration. At the documented baseline, relevant limitations include:
- WDL task/workflow names must be unique across the import tree.
- Some permissive type conversions accepted by other WDL engines are rejected.
- CWL 1.0/1.1 is not compiled directly.
- Calling native DNAnexus apps/applets from CWL through
dxniis not supported. - Some CWL requirements and global workflow publication remain unsupported.
Do not promise byte-identical behavior to Cromwell or another CWL runner without conformance tests.
Nextflow
Creating a DNAnexus app/applet from Nextflow source requires a DNAnexus license.
Current dx-toolkit supports Nextflow engine versions 25.10 and 24.10;
25.10 is the current default and 24.10 is retained only as an unmaintained
compatibility option.
Repository import
dx build --nextflow \
--repository "https://github.com/nextflow-io/hello" \
--repository-tag "<pinned-tag-or-commit>" \
--destination "project-xxxx:/applets/hello"
Pin a repository tag or commit. Do not import a floating default branch for a production pipeline.
For a private repository, --git-credentials accepts a DNAnexus file. Use a
short-lived, read-only credential and store it in a project whose membership
has been reviewed. Anyone with sufficient project access may be able to read
that file.
Local source
dx build --nextflow "path/to/pipeline" \
--nextflow-version "25.10" \
--destination "project-xxxx:/applets/my-pipeline"
An nf-core-style layout is encouraged but not required.
Inspect and run
dx run "applet-xxxx" -h
The generated executable exposes fields for Nextflow run options, configuration files, and a YAML/JSON parameter file. Use the generated help rather than assuming field names.
dx run "applet-xxxx" \
--input-json-file "nextflow-inputs.json" \
--destination "project-xxxx:/runs/run-003" \
--cost-limit 100
The execution is a head job supervising process subjobs. Monitor both:
dx watch "job-head"
dx watch "job-child"
dx find jobs --root-execution "job-head"
Check current CLI help if a filter name differs; dx-toolkit evolves.
Offline execution
With dx-toolkit 0.415.0, rebuilt Nextflow applets honor
nextflow_run_opts="-offline" and automatically enable offline mode when the
job has no outbound network access. -offline=false requests online mode; it
does not grant network permission. Pre-stage required plugins, pipeline
resources, containers, and input data so offline execution does not need a
registry or remote download. Rebuild older applets to incorporate the updated
template, then test a small run under the intended network policy.
Containers
DNAnexus Nextflow execution supports Docker as the container runtime. Pin containers by immutable digest.
Options include:
- Pulling from an approved registry
--cache-dockerto cache image tarballs in the selected project--docker-secretsfor a DNAnexus file containing private registry credentials
Registry credentials are readable data objects to users with sufficient project access. Use pull-only, short-lived credentials and review membership.
External image pulls require network access. Caching images improves reproducibility and reduces registry dependence.
Resume and cache
Nextflow pipeline applets support resume/cache-preservation inputs. Current platform guidance limits preserved sessions in a project to 20. Clean old work directories and cache sessions deliberately to control storage costs.
Only one job can resume and preserve the same session at a time. Do not launch concurrent writes to one preserved session.
Head job and process resources
The head job orchestrates the pipeline; process subjobs perform scientific work. Size them separately:
- Choose a stable head-job instance that can manage the graph.
- Use Nextflow
cpus,memory, anddiskdirectives for processes. - Confirm the generated executor maps requirements as expected.
- Keep queue size within platform limits (current documented maximum: 1000).
Advanced work directory
DNAnexus documents using an S3 bucket as a Nextflow work directory through OIDC. This requires a reviewed cloud trust policy and narrow bucket permissions. Do not embed AWS credentials; use the DNAnexus job OIDC provider and least-privilege IAM role.
Portability Test Plan
For any imported workflow:
- Validate source with its native tooling.
- Pin compiler/importer and container versions.
- Compile/import into a test project.
- Run a small non-sensitive fixture.
- Compare outputs and checksums to the reference runner.
- Compare scatter behavior, retries, and resource mapping.
- Test failed-task reporting and resume.
- Confirm output locations and metadata.
- Record execution cost and wall time.
- Test in every intended cloud/region.
Scientific equivalence matters more than successful compilation. Validate reference builds, coordinate systems, sort orders, floating-point tolerances, and tool versions.
Security Checklist
- No workflow source or inputs contain credentials.
- Imported URLs are trusted and immutable.
- Protected-import tokens are domain-scoped and not logged.
- Container images are pinned and provenance-checked.
- Private repository/registry files are in tightly controlled projects.
- Network access is limited to required hosts.
- External work-directory roles are least privilege.
- PHI does not leave approved projects/TREs.
- Cost limits, timeouts, and concurrency are bounded.
- Generated apps/applets are reviewed before publication.