Configuration

Preface

This document explains the DolphinScheduler application configurations according to DolphinScheduler-1.3.x versions.

Directory Structure

Currently, all the configuration files are under [conf ] directory. Check the following simplified DolphinScheduler installation directories to have a direct view about the position of [conf] directory and configuration files it has. This document only describes DolphinScheduler configurations and other topics are not going into.

[Note: the DolphinScheduler (hereinafter called the ‘DS’) .]

├── LICENSE
│
├── NOTICE
│
├── licenses                                    directory of licenses
│
├── bin                                         directory of DolphinScheduler application commands, configrations scripts 
│   ├── dolphinscheduler-daemon.sh              script to start or shut down DolphinScheduler application
│   ├── env                                     directory of scripts to load environment variables
│   │   ├── dolphinscheduler_env.sh             script to export environment variables [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...] when you start or stop service using script `dolphinscheduler-daemon.sh`
│   │   └── install_env.sh                      script to export environment variables for DolphinScheduler installation when you use scripts `install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`
│   ├── install.sh                              script to auto-setup services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│   ├── remove-zk-node.sh                       script to cleanup ZooKeeper caches 
│   ├── scp-hosts.sh                            script to copy installation files to target hosts 
│   ├── start-all.sh                            script to start all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│   ├── status-all.sh                           script to check the status of all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│   └── stop-all.sh                             script to shut down all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│
├── alert-server                                directory of DolphinScheduler alert-server commands, configrations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler alert-server
│   ├── conf
│   │   ├── application.yaml                    configurations of alert-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc. 
│   │   ├── dolphinscheduler_env.sh             script to load environment variables for alert-server
│   │   └── logback-spring.xml                  configurations of alert-service log
│   └── libs                                    directory of alert-server libs
│
├── api-server                                  directory of DolphinScheduler api-server commands, configrations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler api-server
│   ├── conf
│   │   ├── application.yaml                    configurations of api-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh             script to load environment variables for api-server
│   │   └── logback-spring.xml                  configurations of api-service log
│   ├── libs                                    directory of api-server libs
│   └── ui                                      directory of api-server related front-end web resources 
│
├── master-server                               directory of DolphinScheduler master-server commands, configrations scripts and libs
│   ├── bin                                
│   │   └── start.sh                            script to start DolphinScheduler master-server
│   ├── conf
│   │   ├── application.yaml                    configurations of master-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh             script to load environment variables for master-server
│   │   └── logback-spring.xml                  configurations of master-service log
│   └── libs                                    directory of master-server libs
│
├── standalone-server                           directory of DolphinScheduler standalone-server commands, configrations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler standalone-server
│   ├── conf
│   │   ├── application.yaml                    configurations of standalone-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh             script to load environment variables for standalone-server
│   │   ├── logback-spring.xml                  configurations of standalone-service log
│   │   └── sql                                 .sql files to create or upgrade DolphinScheduler metadata
│   ├── libs                                    directory of standalone-server libs
│   └── ui                                      directory of standalone-server related front-end web resources
│       
├── tools                                       directory of DolphinScheduler metadata tools commands, configrations scripts and libs
│   ├── bin
│   │   └── upgrade-schema.sh                   script to initialize or upgrade DolphinScheduler metadata
│   ├── conf
│   │   ├── application.yaml                    configurations of tools
│   │   └── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                    directory of tool libs
│   └── sql                                     .sql files to create or upgrade DolphinScheduler metadata
│     
├── worker-server                               directory of DolphinScheduler worker-server commands, configrations scripts and libs
│       ├── bin
│       │   └── start.sh                        script to start DolphinScheduler worker-server
│       ├── conf
│       │   ├── application.yaml                configurations of worker-server
│       │   ├── common.properties               configurations of common-service like storage, credentials, etc.
│       │   ├── dolphinscheduler_env.sh         script to load environment variables for worker-server
│       │   └── logback-spring.xml              configurations of worker-service log
│       └── libs                                directory of worker-server libs
│
└── ui                                          directory of front-end web resources

Configurations in Details

dolphinscheduler-daemon.sh [startup or shutdown DS application]

dolphinscheduler-daemon.sh is responsible for DS startup and shutdown. Essentially, start-all.sh or stop-all.sh startup and shutdown the cluster via dolphinscheduler-daemon.sh. Currently, DS just makes a basic config, remember to config further JVM options based on your practical situation of resources.

Default simplified parameters are:

export DOLPHINSCHEDULER_OPTS="
-server 
-Xmx16g 
-Xms1g 
-Xss512k 
-XX:+UseConcMarkSweepGC 
-XX:+CMSParallelRemarkEnabled 
-XX:+UseFastAccessorMethods 
-XX:+UseCMSInitiatingOccupancyOnly 
-XX:CMSInitiatingOccupancyFraction=70
"

"-XX:DisableExplicitGC" is not recommended due to may lead to memory link (DS dependent on Netty to communicate).

datasource.properties [datasource config properties]

DS uses Druid to manage database connections and default simplified configs are:

zookeeper.properties [zookeeper config properties]

common.properties [hadoop、s3、yarn config properties]

Currently, common.properties mainly configures Hadoop,s3a related configurations.

application-api.properties [API-service log config]

master.properties [master-service log config]

worker.properties [worker-service log config]

alert.properties [alert-service log config]

quartz.properties [quartz config properties]

This part describes quartz configs and configure them based on your practical situation and resources.

install_config.conf [DS environment variables configuration script[install or start DS]]

install_config.conf is a bit complicated and is mainly used in the following two places.

• DS Cluster Auto Installation.

System will load configs in the install_config.conf and auto-configure files below, based on the file content when executing 'install.sh'. Files such as dolphinscheduler-daemon.sh, datasource.properties, zookeeper.properties, common.properties, application-api.properties, master.properties, worker.properties, alert.properties, quartz.properties, etc.

• Startup and Shutdown DS Cluster.

The system will load masters, workers, alert-server, API-servers and other parameters inside the file to startup or shutdown DS cluster.

File Content

# Note:  please escape the character if the file contains special characters such as `.*[]^${}\+?|()@#&`.
#   eg: `[` escape to `\[`

# Database type (DS currently only supports PostgreSQL and MySQL)
dbtype="mysql"

# Database url and port
dbhost="192.168.xx.xx:3306"

# Database name
dbname="dolphinscheduler"


# Database username
username="xx"

# Database password
password="xx"

# ZooKeeper url
zkQuorum="192.168.xx.xx:2181,192.168.xx.xx:2181,192.168.xx.xx:2181"

# DS installation path, such as '/data1_1T/dolphinscheduler'
installPath="/data1_1T/dolphinscheduler"

# Deployment user
# Note: Deployment user needs 'sudo' privilege and has rights to operate HDFS.
#     Root directory must be created by the same user if using HDFS, otherwise permission related issues will be raised.
deployUser="dolphinscheduler"


# Followings are alert-service configs
# Mail server host
mailServerHost="smtp.exmail.qq.com"

# Mail server port
mailServerPort="25"

# Mail sender
mailSender="xxxxxxxxxx"

# Mail user
mailUser="xxxxxxxxxx"

# Mail password
mailPassword="xxxxxxxxxx"

# Whether mail supports TLS
starttlsEnable="true"

# Whether mail supports SSL. Note: starttlsEnable and sslEnable cannot both set true.
sslEnable="false"

# Mail server host, same as mailServerHost
sslTrust="smtp.exmail.qq.com"

# Specify which resource upload function to use for resources storage, such as sql files. And supported options are HDFS, S3 and NONE. HDFS for upload to HDFS and NONE for not using this function.
resourceStorageType="NONE"

# if S3, write S3 address. HA, for example: s3a://dolphinscheduler，
# Note: s3 make sure to create the root directory /dolphinscheduler
defaultFS="hdfs://mycluster:8020"

# If parameter 'resourceStorageType' is S3, following configs are needed:
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"

# If ResourceManager supports HA, then input master and standby node IP or hostname, eg: '192.168.xx.xx,192.168.xx.xx'. Or else ResourceManager run in standalone mode, please set yarnHaIps="" and "" for not using yarn.
yarnHaIps="192.168.xx.xx,192.168.xx.xx"


# If ResourceManager runs in standalone, then set ResourceManager node ip or hostname, or else remain default.
singleYarnIp="yarnIp1"

# Storage path when using HDFS/S3
resourceUploadPath="/dolphinscheduler"


# HDFS/S3 root user
hdfsRootUser="hdfs"

# Followings are Kerberos configs

# Specify Kerberos enable or not
kerberosStartUp="false"

# Kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"

# Keytab username
keytabUserName="hdfs-mycluster@ESZ.COM"

# Username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"


# API-service port
apiServerPort="12345"


# All hosts deploy DS
ips="ds1,ds2,ds3,ds4,ds5"

# Ssh port, default 22
sshPort="22"

# Master service hosts
masters="ds1,ds2"

# All hosts deploy worker service
# Note: Each worker needs to set a worker group name and default name is "default"
workers="ds1:default,ds2:default,ds3:default,ds4:default,ds5:default"

#  Host deploy alert-service
alertServer="ds3"

# Host deploy API-service
apiServers="ds1"

dolphinscheduler_env.sh [load environment variables configs]

When using shell to commit tasks, DolphinScheduler will export environment variables from bin/env/dolphinscheduler_env.sh. The mainly configuration including JAVA_HOME, mata database, registry center, and task configuration.

# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/soft/java}

# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_DRIVER_CLASS_NAME
export SPRING_DATASOURCE_URL
export SPRING_DATASOURCE_USERNAME
export SPRING_DATASOURCE_PASSWORD

# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}

# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}

# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/soft/hadoop}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
export SPARK_HOME1=${SPARK_HOME1:-/opt/soft/spark1}
export SPARK_HOME2=${SPARK_HOME2:-/opt/soft/spark2}
export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python}
export HIVE_HOME=${HIVE_HOME:-/opt/soft/hive}
export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}

export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH

Services logback configs