Flink1.18.1 本地单机部署 及 flinkCDC3.0 测试
所属分类 flink
浏览量 390
https://nightlies.apache.org/flink/flink-docs-release-1.8/
https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/docs/get-started/quickstart/mysql-to-doris/
阿里云开发者社区 实时计算 Flink
https://developer.aliyun.com/group/sc/
jdk8 环境准备
下载安装flink1.18.1
https://archive.apache.org/dist/flink/flink-1.18.1/
flink-1.18.1-bin-scala_2.12.tgz
解压
tar xvf flink-1.18.1-bin-scala_2.12.tgz
cd flink-1.18.0/bin
/Users/dugang/work/test/flink-1.18.1/bin
./start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host master.
Starting taskexecutor daemon on host master.
http://127.0.0.1:8081
conf/flink-conf.yaml
#rest.port: 8081
https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/zh/docs/get-started/quickstart/mysql-to-doris/
https://www.apache.org/dyn/closer.lua/flink/flink-1.18.1/flink-1.18.1-bin-scala_2.12.tgz
https://dlcdn.apache.org/flink/flink-1.18.1/flink-1.18.1-bin-scala_2.12.tgz
这个下载快
下载 flinkcdc3.0.0的 mysql和 doris 连接器放到 lib目录下
/Users/dugang/work/test/soft/flink/flink-cdc-3.0.0/lib/
flink-cdc-pipeline-connector-mysql-3.0.0.jar
flink-cdc-pipeline-connector-doris-3.0.0.jar
flinkcdc3.0 测试
cd /Users/dugang/work/test/soft/flink/flink-cdc-3.0.0
./bin/flink-cdc.sh --flink-home /Users/dugang/work/test/flink-1.18.1 conf/flink-cdc-mysql-doris-001.yaml
Pipeline has been submitted to cluster.
Job ID: 895e24f476ec88c695dbb6859c6c845b
Job Description: mysql-doris-001
flink-cdc-mysql-doris-001.yaml
source:
type: mysql
hostname: 192.168.1.10
port: 6454
username: user
password: user123
tables: demo.t1,
demo.t2
# server-id: 5400-5404
server-time-zone: Asia/Shanghai
sink:
type: doris
fenodes: 192.168.10.20:8030
username: root
password: root123
table.create.properties.light_schema_change: true
table.create.properties.replication_num: 3
pipeline:
name: mysql-doris-001
parallelism: 1
route:
- source-table: demo.t1
sink-table: demo.t1
- source-table: demo.t2
sink-table: demo.t2
上一篇
下一篇
杭州市内游玩交通
Kube-Install 一键离线安装多K8S集群
flink checkpoint 检查点
Skywalking 术语
linux安装切换多个版本jdk
doris 简介