DolphinScheduler development

Software Requests

Before setting up the DolphinScheduler development environment, please make sure you have installed the software as below:

Git: DolphinScheduler version control system
JDK: DolphinScheduler backend language
Maven: Java Package Management System
Node: DolphinScheduler frontend language

Clone Git Repository

Download the git repository through your git management tool, here we use git-core as an example

mkdir dolphinscheduler
cd dolphinscheduler
git clone git@github.com:apache/dolphinscheduler.git

Notice

There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode

Standalone mode: Recommended，more convenient to build development environment, it can cover most scenes.
Normal mode: Separate server master, worker, api, logger, which can cover more test environments than standalone, and it is more like production environment in real life.

DolphinScheduler Standalone Quick Start

Note: Standalone server only for development and debugging, cause it use H2 Database, Zookeeper Testing Server which may not stable in production If you want to test plugin, you can modify plugin.bind in StandaloneServer class or modify the configuration file by yourself. Standalone is only supported in DolphinScheduler 1.3.9 and later versions

Git Branch Choose

Use different Git branch to develop different codes

If you want to develop based on a binary package, switch git branch to specific release branch, for example, if you want to develop base on 1.3.9, you should choose branch 1.3.9-release.
If you want to develop the latest code, choose branch branch dev.

Start backend server

Compile backend code

mvn install -DskipTests

Find the class org.apache.dolphinscheduler.server.StandaloneServer in Intellij IDEA and clikc run main function to startup.

Start frontend server

Install frontend dependencies and run it

cd dolphinscheduler-ui
npm install
npm run start

The browser access address http://localhost:12345/dolphinscheduler can login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123

DolphinScheduler Normal Mode

Prepare

zookeeper

Download ZooKeeper, and extract it.

Create directory zkData and zkLog
Go to the zookeeper installation directory, copy configure file zoo_sample.cfg to conf/zoo.cfg, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper
```
# We use path /data/zookeeper/data and /data/zookeeper/datalog here as example
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/datalog
```
Run ./bin/zkServer.sh in terminal by command ./bin/zkServer.sh start.

Database

The DolphinScheduler's metadata is stored in relational database. Currently supported MySQL and Postgresql. We use MySQL as an example. Start the database and create a new database named dolphinscheduler as DolphinScheduler metabase

After creating the new database, run the sql file under dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql directly in MySQL to complete the database initialization

Start Backend Server

Following steps will guide how to start the DolphinScheduler backend service

Backend Start Prepare

Open project: Use IDE open the project, here we use Intellij IDEA as an example, after opening it will take a while for Intellij IDEA to complete the dependent download
Plugin installation(Only required for 2.0 or later): Compile plugin by command mvn -U clean install -Dmaven.test.skip=true

Note: ${VERSION} needs to be manually modified according to the current version, regarding ***.plugin.binding, maven.local.repository does not need to be modified.
- alert plugin config (alert.properties)
```
 alert.plugin.dir=./dolphinscheduler-dist/target/dolphinscheduler-dist-${VERSION}/lib/plugin/alert	
```
- registry plugin config (registry.properties)
```
 registry.plugin.dir=./dolphinscheduler-dist/target/dolphinscheduler-dist-${VERSION}/lib/plugin/registry/zookeeper	
```
- task plugin config (worker.properties)
```
   task.plugin.dir=./dolphinscheduler-dist/target/dolphinscheduler-dist-${VERSION}/lib/plugin/task	
```

File change

If you use MySQL as your metadata database, you need to modify dolphinscheduler/pom.xml and change the scope of the mysql-connector-java dependency to compile. This step is not necessary to use PostgreSQL
Modify database configuration, modify the database configuration in the dolphinscheduler/dolphinscheduler-dao/datasource.properties

# We here use MySQL with database, username, password named dolphinscheduler as an example
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true
spring.datasource.username=dolphinscheduler
spring.datasource.password=dolphinscheduler

Log level: add a line <appender-ref ref="STDOUT"/> to the following configuration to enable the log to be displayed on the command line

dolphinscheduler-server/src/main/resources/logback-worker.xml

dolphinscheduler-server/src/main/resources/logback-master.xml

dolphinscheduler-api/src/main/resources/logback-api.xml

here we add the result after modify as below:
```
<root level="INFO">
+  <appender-ref ref="STDOUT"/>
  <appender-ref ref="APILOGFILE"/>
  <appender-ref ref="SKYWALKING-LOG"/>
</root>
```

Note: Only DolphinScheduler 2.0 and later versions need to inatall plugin before start server. It not need before version 2.0.

Server start

There are three necessary server we have to start, including MasterServer，WorkerServer，ApiApplicationServer, and a optional server you could start if you need, named LoggerServer.

MasterServer：Execute function main in the class org.apache.dolphinscheduler.server.master.MasterServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false
WorkerServer：Execute function main in the class org.apache.dolphinscheduler.server.worker.WorkerServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false
ApiApplicationServer：Execute function main in the class org.apache.dolphinscheduler.api.ApiApplicationServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-api.xml -Dspring.profiles.active=api. After it started, you could find Open API documentation in http://localhost:12345/dolphinscheduler/doc.html
LoggerServer：Optional server, only start if you need，Execute function main in the class org.apache.dolphinscheduler.server.log.LoggerServer by Intellij IDEA

Start Frontend Server

Install frontend dependencies and run it

cd dolphinscheduler-ui
npm install
npm run start

The browser access address http://localhost:12345/dolphinscheduler can login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123