Scale Your Data Workflows to New Heights

Steep is a cloud-based workflow management system that orchestrates your microservices to process big data in a resource- and cost-efficient manner.

Get started
actions:
  - type: execute
    service: tile
    inputs:
      - id: input_file
        value: dop20_32_475_5524_1_he.jpg
      - id: num_tiles
        value: 5
    outputs:
      - id: output_directory
        var: tiles
 
  - type: for
    enumerator: i
    input: tiles
    actions:
      - type: execute
        service: segment
        inputs:
          - id: input_file
            var: i
        outputs:
          - id: output_file
            var: mask_image
            store: true

Declarative and data-driven workflows

This example splits an aerial image into 5×5 tiles and then applies an AI-based image segmentation service to each of them.

Steep automatically detects dependencies between the services and executes them in parallel if possible.

TileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTileTile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Tile
Large image tileLarge image tile mask

Microservices

Steep is able to orchestrate microservices with arbitrary interfaces and written in any programming language.

Service metadata

Every service has a unique ID, a name, and a description. The path attribute points to the service’s executable or Docker image, and the runtime and the parameters specify how it is executed.

Capability-based scheduling

Services may require certain capabilities. During scheduling, Steep matches these capabilities with those of available agents in the distributed environment and assigns the services accordingly.

- id: segment
  name: Image segmentation
  description: >-
    Segment geospatial data with the Segment Anything
    Model (SAM)
  path: segment.py
  runtime: other
 
  requiredCapabilities:
    - python
    - gpu
 
  parameters:
    - id: input_file
      name: Input image
      description: The input image to segment
      type: input
      cardinality: 1..1
      dataType: string
 
    - id: output_file
      name: Output image
      description: Output image with masks
      type: output
      cardinality: 1..1
      dataType: string
      fileSuffix: .jpg
- id: ml-gpu
  flavor: p2.xlarge
  imageName: ami-050096f31d010b533  # Ubuntu 22.04
  availabilityZone: eu-central-1
  blockDeviceSizeGb: 50
  blockDeviceVolumeType: gp3  # SSD
  maxVMs: 10
 
  creation:
    retries:
      maxAttempts: 5
      delay: 10s
      exponentialBackoff: 2
 
  provisioningScripts:
    - conf/setups/cuda.sh
    - conf/setups/python.sh
    - conf/setups/pip-samgeo.sh
    - conf/setups/docker.sh
    - conf/setups/steep.sh
 
  providedCapabilities:
    - docker
    - python
    - gpu
kind: Deployment
apiVersion: apps/v1
metadata:
  name: steep
spec:
  # cluster with 3 nodes enabling automatic
  # failover and zero downtime deployments
  replicas: 3
 
  selector:
    matchLabels:
      app: steep
  template:
    metadata:
      labels:
        app: steep
    spec:
      containers:
        - name: steep
          image: steep/steep
          env:
            - name: STEEP_HTTP_HOST
              value: "0.0.0.0"
          ports:
            - containerPort: 8080

Cloud support

Deploy Steep to Kubernetes and automatically manage your cloud infrastructure based on resource demand.

Docker & Kubernetes

Steep has first-level support for Docker. It can run in a container and orchestrate other containers. It can also be deployed to a Kubernetes cluster.

Connectors

Steep can connect to various cloud providers, such as AWS and OpenStack. It creates and provisions virtual machines on demand and destroys them when they are no longer needed.

Web-based user interface

Web user interface

Submit workflows, monitor their progress, and review Steep’s configuration.

But wait, there’s more!

See what other features Steep has to offer.

Automatic failover

Workflows or individual services can automatically be retried in case of an error, and crashed workflows can be resumed without information loss.

Cyclic workflows

Submit cyclic workflow graphs without a priori runtime knowledge. Workflows are converted incrementally and on-demand to so-called process chains.

Languages and runtimes

Custom runtimes enable your team to develop microservices in various programming languages and with different frameworks. This allows you to focus on the task at hand.

Interfaces and extensibility

Use the REST-like HTTP interface in your applications to submit workflows or to fetch information during execution. Write your own plugins to customize Steep.

Production-ready

Steep is very reliable and has been used in production for many years to execute workflows from various domains. The source code has a very high test coverage.

Open-Source

Steep is free and open-source. It is released under the Apache License, Version 2.0. The code can be found on GitHub. We will be more than happy to accept your contributions!