PyDolphinScheduler
PyDolphinScheduler is Python API for Apache DolphinScheduler, which allow you define your workflow by Python code, aka workflow-as-codes.
History
We seperated PyDolphinScheduler code base from Apache dolphinscheduler code base into independent repository at Nov 7, 2022. Now the code base is in Apache dolphinscheduler-sdk-python and all issue and pull requests should be submitted to this repository.
The reason why we seperated is for the following reasons:
Clear responsibility: The code base of PyDolphinScheduler is independent of DolphinScheduler, it has its own users and developers, CI and test cases, even the language is different.
Independent release cycle: PyDolphinScheduler is a SDK, it should be released independently of DolphinScheduler. before we seperated, we have to release PyDolphinScheduler every time when DolphinScheduler released. This brings two main problems:
DolphinScheduler release cycle is a little long for PyDolphinScheduler, because PyDolphinScheduler is a new subproject for DolphinScheduler, it is not stable enough, we have many features and enhancement want to add, so we have to release DolphinScheduler more frequently.
We have to release PyDolphinScheduler every time when DolphinScheduler, even if there is no change in PyDolphinScheduler. In the past, we release both of them at the same time, and some of DolphinScheduler bugfix version like 2.0.5, 2.0.6, 2.0.7 only fix some bugs in DolphinScheduler, but not in PyDolphinScheduler, is it unnecessary to release PyDolphinScheduler but we have to because the rule is we release them both at the same time.
Should use the same version as DolphinScheduler do, due to the same release cycle, we have to have the same version as DolphinScheduler, so you may see there are some functional adder in DolphinScheduler bugfix version, it may be a little strange to PyDolphinScheduler users.
Faster CI Run: PyDolphinScheduler has its own CI, it is not necessary to run DolphinScheduler CI when we only change PyDolphinScheduler code. And DolphinScheduler CI is more complete, and some of CI is required to merge pull requests, Separating PyDolphinScheduler from DolphinScheduler can make CI run faster in both two repositories.
For more details, please refer to seperated mail thread discuss.
Version
At Nov 7, 2022 we seperated PyDolphinScheduler. And before that, PyDolphinScheduler have the same version with DolphinScheduler. So the version of PyDolphinScheduler is the same as DolphinScheduler, for example PyDolphinScheduler 2.0.5 is matched with DolphinScheduler 2.0.5. For more details about why we seperated PyDolphinScheduler, please refer to History.
After being separated from the DolphinScheduler repository, PyDolphinScheduler will be released independently, and the version of PyDolphinScheduler will not correspond to the version of DolphinScheduler, which means some of the PyDolphinScheduler versions will correspond to multiple DolphinScheduler versions when there is no change in the PythonGateway code. While part of the code of PyDolphinScheduler will only correspond to one version of DolphinScheduler, if the code of the next version of PythonGateway has changed.
We will release PyDolphinScheduler version 4.0.0 as the first version after we seperated, so please be ware of when you use PyDolphinScheduler above version 4.0.0.
DolphinScheduler Version |
PyDolphinScheduler Version |
Note |
|---|---|---|
2.0.5 |
2.0.5 |
before we seperated |
>=2.0.6, <3.0.0 |
2.0.6 |
before we seperated |
3.0.0 |
3.0.0 |
before we seperated |
>=3.0.1, <3.1.0 |
3.0.1 |
before we seperated |
>=3.1.0, <3.1.2 |
3.1.0 |
before we seperated |
>=3.1.2, <3.1.4 |
>=4.0.0, <4.0.3 |
|
>=3.1.5, <3.2.0 |
>=4.0.4, <4.1.0 |
Content
You could go and find how to install the project. Or if you want to see simply example then go and see Tutorial for more detail.