If you want to use the resource upload function, you can select the local file directory for a single machine(this operation does not need to deploy Hadoop). Or you can also upload to a Hadoop or MinIO cluster, at this time, you need to have Hadoop (2.6+) or MinIO and other related environments
- If the resource upload function is used, the deployment user in installation and deployment must to have operation authority
- If you using Hadoop cluster with HA, you need to enable HDFS resource upload, and you need to copy the
hdfs-site.xmlunder the Hadoop cluster to
/opt/dolphinscheduler/conf, otherwise Skip step
conf/common/common.properties # Users who have permission to create directories under the HDFS root path hdfs.root.user=hdfs # data base dir, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions。"/escheduler" is recommended data.store2hdfs.basepath=/dolphinscheduler # resource upload startup type : HDFS,S3,NONE res.upload.startup.type=HDFS # whether kerberos starts hadoop.security.authentication.startup.state=false # java.security.krb5.conf path java.security.krb5.conf.path=/opt/krb5.conf # loginUserFromKeytab user login.user.keytab.username=hdfs-mycluster@ESZ.COM # loginUserFromKeytab path login.user.keytab.path=/opt/hdfs.headless.keytab conf/common/hadoop.properties # ha or single namenode,If namenode ha needs to copy core-site.xml and hdfs-site.xml # to the conf directory，support s3，for example : s3a://dolphinscheduler fs.defaultFS=hdfs://mycluster:8020 #resourcemanager ha note this need ips , this empty if single yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx # If it is a single resourcemanager, you only need to configure one host name. If it is resourcemanager HA, the default configuration is fine yarn.application.status.address=http://xxxx:8088/ws/v1/cluster/apps/%s
It is the management of various resource files, including creating basic txt/log/sh/conf/py/java and other files, uploading jar packages and other types of files, and can do edit, rename, download, delete and other operations.
The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties
Upload file: Click the "Upload File" button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name
For the file types that can be viewed, click the file name to view the file details
Click the "Download" button in the file list to download the file or click the "Download" button in the upper right corner of the file details to download the file
File list -> Click the "Delete" button to delete the specified file
The resource management and file management functions are similar. The difference is that the resource management is the uploaded UDF function, and the file management uploads the user program, script and configuration file. Operation function: rename, download, delete.
Same as uploading files.
Click "Create UDF Function", enter the udf function parameters, select the udf resource, and click "Submit" to create the udf function.
Currently only supports temporary UDF functions of HIVE