PyDolphinScheduler
PyDolphinScheduler is Python API for Apache DolphinScheduler, which allow you definition your workflow by Python code, aka workflow-as-codes.
History
We seperated PyDolphinScheduler code base from Apache dolphinscheduler code base into independent repository at Nov 7, 2022. Now the code base is in Apache dolphinscheduler-sdk-python and all issue and pull requests should be submitted to this repository.
The reason why we seperated is for the following reasons:
Clear responsibility: The code base of PyDolphinScheduler is independent of DolphinScheduler, it has its own users and developers, CI and test cases, even the language is different.
Independent release cycle: PyDolphinScheduler is a SDK, it should be released independently of DolphinScheduler. before we seperated, we have to release PyDolphinScheduler every time when DolphinScheduler. This brings two main problems:
DolphinScheduler release cycle is a little long for PyDolphinScheduler, because PyDolphinScheduler is a new subproject for DolphinScheduler, it is not stable enough, we have many features and enhancement want to add, so we have to release DolphinScheduler more frequently.
We have to release PyDolphinScheduler every time when DolphinScheduler, even if there is no change in PyDolphinScheduler. In the past, we release both of them at the same time, and some of DolphinScheduler bugfix version like 2.0.5, 2.0.6, 2.0.7 only fix some bugs in DolphinScheduler, but not in PyDolphinScheduler, is it unnecessary to release PyDolphinScheduler but we have to because the rule is we release them both at the same time.
Should use the same version as DolphinScheduler do, due to the same release cycle, we have to have the same version as DolphinScheduler, so you may see there are some functional adder in DolphinScheduler bugfix version, it may be a little strange to PyDolphinScheduler users.
Faster CI Run: PyDolphinScheduler has its own CI, it is not necessary to run DolphinScheduler CI when we only change PyDolphinScheduler code. And DolphinScheduler CI is more complete, and some of CI is required to merge pull requests, Separating PyDolphinScheduler from DolphinScheduler can make CI run faster in both two repositories.
For more details, please refer to seperated mail thread discuss.
Version
At Nov 7, 2022 we seperated PyDolphinScheduler. And before that, PyDolphinScheduler have the same version with DolphinScheduler. So the version of PyDolphinScheduler is the same as DolphinScheduler, for example PyDolphinScheduler 2.0.5 is matched with DolphinScheduler 2.0.5. For more details about why we seperated PyDolphinScheduler, please refer to History.
And after we seperated, we will release PyDolphinScheduler independently, and the version of PyDolphinScheduler will be different with DolphinScheduler, and some of PyDolphinScheduler version will not match multiple DolphinScheduler version because the code do not change in PyDolphinScheduler, and we will release PyDolphinScheduler but support multiple DolphinScheduler version due to we do not change PythonGateway code.
We will release PyDolphinScheduler version 4.0.0 as the first version after we seperated, so please be ware of when you use PyDolphinScheduler above version 4.0.0.
DolphinScheduler Version |
PyDolphinScheduler Version |
Note |
|---|---|---|
2.0.5 |
2.0.5 |
before we seperated |
above 2.0.6 and prior 3.0.0 |
2.0.6 |
before we seperated |
3.0.0 |
3.0.0 |
before we seperated |
above 3.0.1 and prior 3.1.0 |
3.0.1 |
before we seperated |
above 3.1.0 and prior 3.1.2 |
3.1.0 |
before we seperated |
Content
You could go and find how to install the project. Or if you want to see simply example then go and see Tutorial for more detail.