1. flink集群启动
2. flink数据同步任务启动
3. 数据库自动变更程序启动
4. 同步任务配置文件




flink集群启动


ssh admin@192.168.1.37
ssh admin@192.168.1.38
ssh admin@192.168.1.39

切换用户
sudo su flink

/app/flink/flink-1.18.0/
/app/flink/flink-1.18.0/bin

 ./start-cluster.sh
Starting HA cluster with 3 masters.
Starting standalonesession daemon on host X37.
Starting standalonesession daemon on host X38.
Starting standalonesession daemon on host X39.
Starting taskexecutor daemon on host X37.
Starting taskexecutor daemon on host X38.
Starting taskexecutor daemon on host X39.

ps aux|grep flink
org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
org.apache.flink.runtime.taskexecutor.TaskManagerRunner




flink数据同步任务启动
192.168.1.37
/home/admin/soft/flinkcdc/flink-cdc-3.0.0
sudo ./bin/flink-cdc.sh   --flink-home /app/flink/flink-1.18.0  /home/admin/soft/flinkcdc/conf/sync_003.yaml
sudo ./bin/flink-cdc.sh   --flink-home /app/flink/flink-1.18.0  /home/admin/soft/flinkcdc/conf/sync_004.yaml

flink控制台
http://192.168.1.37:8081




数据库实时变更模拟 启动
启动脚本
/home/admin/soft/app/startdata.sh
nohup java -jar mysql_data_auto_change-1.0.jar &
http://192.168.1.37:8065/runstatus




sync_003.yaml


source:
  type: mysql
  hostname: 192.168.1.2
  port: 3306
  username: xxxx
  password: 123456
  tables: db05.\.*,
  server-id: 5621-5625
  server-time-zone: Asia/Shanghai

sink:
  type: doris
  fenodes: 192.168.1.30:8030
  username: xxx
  password: 123456
  table.create.properties.light_schema_change: true
  table.create.properties.replication_num: 3
  enable-2pc: false
  enable.batch-mode: true
  # 测试新增
  sink.enable-delete: true

pipeline:
  name: sync_003
  parallelism: 1

route:
    - source-table: db05.t_detail_1_\.*
      sink-table: db05.t_detail_1_all
    - source-table: db05.t_detail_2_\.*
      sink-table: db05.t_detail_2_all




sync_004.yaml


source:
  type: mysql
  hostname: 192.168.1.2
  port: 3306
  username: xxxx
  password: 123456
  tables: db06.\.*,db06.\.*,db08.\.*
  server-id: 5631-5635
  server-time-zone: Asia/Shanghai

sink:
  type: doris
  fenodes: 192.168.1.30:8030
  username: xxx
  password: 123456
  table.create.properties.light_schema_change: true
  table.create.properties.replication_num: 3
  enable-2pc: false
  enable.batch-mode: true
  # 测试新增
  sink.enable-delete: true

pipeline:
  name: sync_004
  parallelism: 1







https://gitee.com/dyyx/work2024/tree/master/demo/mysql_data_auto_change