Upload and Use Multiple Resources

Resource center help us manager resources in a centralized way, easy to change and distribute them to all the workers. for more detail you can see resources files.

In this section we will show you how to upload and use multiple resources which is more common in production environment and in the real word.

Overview

from pydolphinscheduler.core import Workflow
from pydolphinscheduler.core.resource import Resource
from pydolphinscheduler.tasks import Shell

dependence = "dependence.py"
main = "main.py"

with Workflow(
    name="multi_resources_example",
    # [start create_new_resources]
    resource_list=[
        Resource(
            name=dependence,
            content="from datetime import datetime\nnow = datetime.now()",
        ),
        Resource(name=main, content="from dependence import now\nprint(now)"),
    ],
    # [end create_new_resources]
) as workflow:
    # [start use_exists_resources]
    task_use_resource = Shell(
        name="use-resource",
        command=f"python {main}",
        resource_list=[
            dependence,
            main,
        ],
    )
    # [end use_exists_resources]

    workflow.run()

In this example, we will upload two python files to resource center and use them in one single task, the python files are named dependence.py and main.py. And main.py use dependence.py as a dependency which will use a variable now declared in dependence.py. So in task shell could call python main.py to get all things done.

Upload Resources

The module Resource need to be import firstly.

from pydolphinscheduler.core.resource import Resource

Then we need to create two resources object and assign them to resource_list of the workflow. All content of resources should assign to content attribute of the resource object. Please know that we import variable now from dependence.py in main.py.

    resource_list=[
        Resource(
            name=dependence,
            content="from datetime import datetime\nnow = datetime.now()",
        ),
        Resource(name=main, content="from dependence import now\nprint(now)"),
    ],

Use Resources

Same as using single resource, all we need is to assign them to resource_list attribute of the task and then call the main file to run our task. In this example, we call python main.py which will use dependence.py as a dependency.

    task_use_resource = Shell(
        name="use-resource",
        command=f"python {main}",
        resource_list=[
            dependence,
            main,
        ],
    )

After run the workflow, will execute main.py and print the current datetime. You can see the result like this:

2022-11-29 16:16:51.952742