Sequential workflows

Step 1: Add more service metadata

In this tutorial, we are going to create a workflow that downloads a .zip file from an HTTP server and then extracts it. For this, we will use the wget and unzip programs. Similar to the first tutorial, we need to add service metadata for these services to the conf/services/services.yaml file.

First, add the following metadata for the wget service:

- id: download
  name: Download
  description: Download a file from a HTTP server
  path: wget
  runtime: other
 
  parameters:
    - id: output_file
      name: Output file
      description: The target file to which to download
      type: output
      cardinality: 1..1
      label: -O
      fileSuffix: .zip
      dataType: file
 
    - id: url
      name: URL
      description: The URL of the file to download
      type: input
      cardinality: 1..1
      dataType: string

Similar to the cp service from the first tutorial, this service has two parameters. However, in this case, the output parameter has been specified first and has a label -O. Steep will respect the order of the parameters and the label when calling wget later during workflow execution and generate a command line in the following form:

wget -O <output_file> <url>

The output parameter also has a fileSuffix. Specifying this will make sure Steep generates an output filename with the correct extension (in this case .zip) later during workflow execution.

Next, we need to add the metadata for the unzip service:

- id: extract
  name: Extract
  description: Extract a ZIP archive
  path: unzip
  runtime: other
 
  parameters:
    - id: input_file
      name: Input file
      description: The archive to extract
      type: input
      cardinality: 1..1
      dataType: file
 
    - id: destination_directory
      name: Destination
      description: >-
        The destination directory to which the archive
        should be extracted
      type: output
      cardinality: 1..1
      label: -d
      dataType: directory

In this case, the output parameter has a label -d and is a directory. Steep is able to automatically create output directories during workflow execution.

Step 2: Create the workflow

Create a new file download.yaml and paste the following workflow into it:

api: 4.7.0
actions:
  - type: execute
    service: download
    inputs:
      - id: url
        value: https://github.com/steep-wms/steep-wms.github.io/archive/refs/heads/master.zip
    outputs:
      - id: output_file
        var: o1
 
  - type: execute
    service: extract
    inputs:
      - id: input_file
        var: o1
    outputs:
      - id: destination_directory
        var: output_directory
        store: true

The workflow consists of two execute actions calling the download and the extract service. It downloads the source code of this website from GitHub and extracts it into Steep’s output directory.

Steep is able to automatically detect the dependency between the two actions based on their inputs and outputs. The output of the download action is stored in the variable o1, which is in turn used as input for the extract action. This tells Steep to execute the download action first, wait for it to finish, and then execute the extract action.

Step 3: Submit the workflow

Run the following command on your terminal to submit the new workflow to Steep:

curl -X POST http://localhost:8080/workflows --data-binary @download.yaml

Similar to the first tutorial, Steep will return a workflow ID.

Wait until the workflow has finished. If it was successful, you will find the extracted archive in Steep’s output directory:

ls /tmp/steep/out/[WORKFLOW ID]

Since we did not specify the store flag for the download action, the downloaded .zip file will be stored in Steep’s temporary directory, which is /tmp by default:

ls /tmp/[WORKFLOW ID]

Again, replace [WORKFLOW ID] with the ID returned by Steep.