Shell

A shell task type’s example and dive into information of PyDolphinScheduler.

Example

with ProcessDefinition(
    name="tutorial",
    schedule="0 0 0 * * ? *",
    start_time="2021-01-01",
    tenant="tenant_exists",
) as pd:
    # [end workflow_declare]
    # [start task_declare]
    task_parent = Shell(name="task_parent", command="echo hello pydolphinscheduler")
    task_child_one = Shell(name="task_child_one", command="echo 'child one'")
    task_child_two = Shell(name="task_child_two", command="echo 'child two'")
    task_union = Shell(name="task_union", command="echo union")
    # [end task_declare]

    # [start task_relation_declare]
    task_group = [task_child_one, task_child_two]
    task_parent.set_downstream(task_group)

    task_union << task_group

Dive Into

Task shell.

class pydolphinscheduler.tasks.shell.Shell(name: str, command: str, *args, **kwargs)

Bases: Task

Task shell object, declare behavior for shell task to dolphinscheduler.

Parameters:

• name – A unique, meaningful string for the shell task.

• command –

  One or more command want to run in this task.

  It could be simply command:

  Shell(name=..., command="echo task shell")
  

  or maybe same commands trying to do complex task:

  command = '''echo task shell step 1;
  echo task shell step 2;
  echo task shell step 3
  '''
  
  Shell(name=..., command=command)
  

_downstream_task_codes: Set[int]

_task_custom_attr: set = {'raw_script'}

_task_relation: Set[TaskRelation]

_upstream_task_codes: Set[int]

YAML file example

# Define the workflow
workflow:
  name: "Shell"
  release_state: "offline"
  run: true

# Define the tasks under the process
tasks:
  - name: task_parent
    task_type: Shell
    command: |
      echo hello pydolphinscheduler 
      echo run task parent

  - name: task_child_one
    task_type: Shell
    deps: [task_parent]
    command: echo "child one"

  - name: task_child_two
    task_type: Shell
    deps: [task_parent]
    command: echo "child two"