Standalone Deployment

1、Install basic softwares (please install required softwares by yourself)

  • PostgreSQL (8.2.15+) or MySQL (5.7) : Choose One, JDBC Driver 5.1.47+ is required if MySQL is used
  • JDK (1.8+) : Required. Double-check configure JAVA_HOME and PATH environment variables in /etc/profile
  • ZooKeeper (3.4.6+) :Required
  • Hadoop (2.6+) or MinIO :Optional. If you need resource function, for Standalone Deployment you can choose a local directory as the upload destination (this does not need Hadoop deployed). Of course, you can also choose to upload to Hadoop or MinIO.
 Tips:DolphinScheduler itself does not rely on Hadoop, Hive, Spark, only use their clients to run corresponding task.

2、Download the binary tar.gz package.

  • Please download the latest version installation package to the server deployment directory. For example, use /opt/dolphinscheduler as the installation and deployment directory. Download address: Download, download package, move to deployment directory and unzip it.
# Create the deployment directory. Please do not choose a high-privilege directory such as /root or /home.
mkdir -p /opt/dolphinscheduler;
cd /opt/dolphinscheduler;

# unzip
tar -zxvf apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz -C /opt/dolphinscheduler;

# rename
mv apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin  dolphinscheduler-bin

3、Create deployment user and assign directory operation permissions

  • Create a deployment user, and be sure to configure sudo secret-free. Here take the creation of a dolphinscheduler user as example.
# To create a user, you need to log in as root and set the deployment user name.
useradd dolphinscheduler;

# Set the user password, please modify it yourself.
echo "dolphinscheduler123" | passwd --stdin dolphinscheduler

# Configure sudo secret-free
echo 'dolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers
sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers

# Modify the directory permissions so that the deployment user has operation permissions on the dolphinscheduler-bin directory
chown -R dolphinscheduler:dolphinscheduler dolphinscheduler-bin
 - Because the task execution is based on 'sudo -u {linux-user}' to switch among different Linux users to implement multi-tenant job running, so the deployment user must have sudo permissions and is secret-free. If beginner learners don’t understand, you can ignore this point for now.
 - Please comment out line "Default requiretty", if it present in "/etc/sudoers" file. 
 - If you need to use resource upload, you need to assign user the permission to operate the local file system, HDFS or MinIO.

4、SSH secret-free configuration

  • Switch to the deployment user and configure SSH local secret-free login

    su dolphinscheduler;
    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    cat ~/.ssh/ >> ~/.ssh/authorized_keys
    chmod 600 ~/.ssh/authorized_keys

​ Note: If configure successed, the dolphinscheduler user does not need to enter a password when executing the command ssh localhost.

5、Database initialization

  • Log in to the database, the default database type is PostgreSQL. If you choose MySQL, you need to add the mysql-connector-java driver package to the lib directory of DolphinScheduler.
mysql -uroot -p
  • After log into the database command line window, execute the database initialization command and set the user and password.

Note: {user} and {password} need to be replaced with a specific database username and password.

   mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
   mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
   mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}';
   mysql> flush privileges;
  • Create tables and import basic data

    • Modify the following configuration in under the conf directory.
      vi conf/
    • If you choose Mysql, please comment out the relevant configuration of PostgreSQL (vice versa), you also need to manually add the [[mysql-connector-java driver jar] (] package to lib directory, and then configure the database connection information correctly.
      # mysql
      spring.datasource.url=jdbc:mysql://xxx:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true     # Replace the correct IP address
      spring.datasource.username=xxx						# replace the correct {username} value
      spring.datasource.password=xxx						# replace the correct {password} value
    sh script/

Note: If you execute the above script and report "/bin/java: No such file or directory" error, please configure JAVA_HOME and PATH variables in /etc/profile.

6、Modify runtime parameters.

  • Modify the environment variable in file under 'conf/env' directory (take the relevant software installed under '/opt/soft' as example)

        export HADOOP_HOME=/opt/soft/hadoop
        export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
        #export SPARK_HOME1=/opt/soft/spark1
        export SPARK_HOME2=/opt/soft/spark2
        export PYTHON_HOME=/opt/soft/python
        export JAVA_HOME=/opt/soft/java
        export HIVE_HOME=/opt/soft/hive
        export FLINK_HOME=/opt/soft/flink
        export DATAX_HOME=/opt/soft/datax/bin/
     `Note: This step is very important. For example, JAVA_HOME and PATH must be configured. Those that are not used can be ignored or commented out. If you can not find, please run ls -a.`
  • Create JDK soft link to /usr/bin/java (still JAVA_HOME=/opt/soft/java as an example)

    sudo ln -s /opt/soft/java/bin/java /usr/bin/java
  • Modify the parameters in the one-click deployment config file conf/config/install_config.conf, pay special attention to the configuration of the following parameters.

    # choose mysql or postgresql
    # Database connection address and port
    # database name
    # database username
    # database password
    # NOTICE: if there are special characters, please use the \ to escape, for example, `[` escape to `\[`
    # Zookeeper address, localhost:2181, remember port 2181
    # Note: the target installation path for dolphinscheduler, please do not use current path (pwd)
    # deployment user
    # Note: the deployment user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled, the root directory needs to be created by itself
    # alert config,take QQ email for example
    # mail protocol
    # mail server host
    # mail server port
    # note: Different protocols and encryption methods correspond to different ports, when SSL/TLS is enabled, port may be different, make sure the port is correct.
    # mail sender
    # mail user
    # mail sender password
    # note: The mail.passwd is email service authorization code, not the email login password.
    # Whether TLS mail protocol is supported,true is supported and false is not supported
    # Whether TLS mail protocol is supported,true is supported and false is not supported。
    # note: only one of TLS and SSL can be in the true state.
    # note: sslTrust is the same as mailServerHost
    # resource storage type:HDFS,S3,NONE
    # here is an example of saving to a local file system
    # Note: If you want to upload resource file(jar file and so on)to HDFS and the NameNode has HA enabled, you need to put core-site.xml and hdfs-site.xml of hadoop cluster in the installPath/conf directory. In this example, it is placed under /opt/soft/dolphinscheduler/conf, and Configure the namenode cluster name; if the NameNode is not HA, modify it to a specific IP or host name.
    # if not use hadoop resourcemanager, please keep default value; if resourcemanager HA enable, please type the HA ips ; if resourcemanager is single, make this value empty
    # Note: For tasks that depend on YARN to execute, you need to ensure that YARN information is configured correctly in order to ensure successful execution results.
    # if resourcemanager HA enable or not use resourcemanager, please skip this value setting; If resourcemanager is single, you only need to replace yarnIp1 to actual resourcemanager hostname.
    # resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions。/dolphinscheduler is recommended
    # specify the user who have permissions to create directory under HDFS/S3 root path
    # On which machines to deploy the DS service, choose localhost for this machine
    # ssh port, default 22
    # Note: if ssh port is not default, modify here
    # run master machine
    # run worker machine
    # run alert machine
    # run api machine

    Attention: if you need upload resource function, please execute below command:

    sudo mkdir /data/dolphinscheduler
    sudo chown -R dolphinscheduler:dolphinscheduler /data/dolphinscheduler 

7、Automated Deployment

  • Switch to the deployment user and execute the one-click deployment script


    For the first deployment, the following message appears in step 3 of `3, stop server` during operation. This message can be ignored.
    sh: bin/ No such file or directory
  • After script completed, the following 5 services will be started. Use jps command to check whether the services started (jps comes with java JDK)

    MasterServer         ----- master service
    WorkerServer         ----- worker service
    LoggerServer         ----- logger service
    ApiApplicationServer ----- api service
    AlertServer          ----- alert service

If the above services started normally, the automatic deployment is successful.

After the deployment is success, you can view logs. Logs stored in the logs folder.

    ├── dolphinscheduler-alert-server.log
    ├── dolphinscheduler-master-server.log
    |—— dolphinscheduler-worker-server.log
    |—— dolphinscheduler-api-server.log
    |—— dolphinscheduler-logger-server.log


9、Start and stop service

  • Stop all services

    sh ./bin/

  • Start all services

    sh ./bin/

  • Start and stop master service

sh ./bin/ start master-server
sh ./bin/ stop master-server
  • Start and stop worker Service
sh ./bin/ start worker-server
sh ./bin/ stop worker-server
  • Start and stop api Service
sh ./bin/ start api-server
sh ./bin/ stop api-server
  • Start and stop logger Service
sh ./bin/ start logger-server
sh ./bin/ stop logger-server
  • Start and stop alert service
sh ./bin/ start alert-server
sh ./bin/ stop alert-server

Note: Please refer to the "Architecture Design" section for service usage