1. flink集群启动
2. flink数据同步任务启动
3. 数据库自动变更程序启动
4. 同步任务配置文件


flinkcdc3.0.0 + mysql doris connector 3.0.0 
问题比较多 
1 doris load error
2 表结构变更导致任务失败 
3 新建表无法直接同步，需要重启任务
4 偶现即使任务重启，也会出现表结构变更无法同步，需要删表重新同步
5 并发度超过1 会报下标溢出问题
6 业务库数据量比较大，一旦重启需要4个小时以上才会重新同步一遍数据，这期间新数据无法同步


flinkcdc3.2.0 + mysql doris connector 3.1.0  运行稳定
并行度3  业务库 全量同步一次  1.6亿记录数   2个小时 左右   64GB bytes
新建表无法自动同步，后续版本会支持





flink集群启动


ssh admin@192.168.1.37
ssh admin@192.168.1.38
ssh admin@192.168.1.39

切换用户
sudo su flink

/app/flink/flink-1.18.0/
/app/flink/flink-1.18.0/bin

 ./start-cluster.sh
Starting HA cluster with 3 masters.
Starting standalonesession daemon on host X37.
Starting standalonesession daemon on host X38.
Starting standalonesession daemon on host X39.
Starting taskexecutor daemon on host X37.
Starting taskexecutor daemon on host X38.
Starting taskexecutor daemon on host X39.

ps aux|grep flink
org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
org.apache.flink.runtime.taskexecutor.TaskManagerRunner




flink数据同步任务启动
192.168.1.37
/home/admin/soft/flinkcdc/flink-cdc-3.0.0
sudo ./bin/flink-cdc.sh   --flink-home /app/flink/flink-1.18.0  /home/admin/soft/flinkcdc/conf/sync_003.yaml
sudo ./bin/flink-cdc.sh   --flink-home /app/flink/flink-1.18.0  /home/admin/soft/flinkcdc/conf/sync_004.yaml

flink控制台
http://192.168.1.37:8081




数据库实时变更模拟 启动
启动脚本
/home/admin/soft/app/startdata.sh
nohup java -jar mysql_data_auto_change-1.0.jar &
http://192.168.1.37:8065/runstatus




sync_003.yaml


source:
  type: mysql
  hostname: 192.168.1.2
  port: 3306
  username: xxxx
  password: 123456
  tables: db05.\.*,
  server-id: 5621-5625
  server-time-zone: Asia/Shanghai

sink:
  type: doris
  fenodes: 192.168.1.30:8030
  username: xxx
  password: 123456
  table.create.properties.light_schema_change: true
  table.create.properties.replication_num: 3
  enable-2pc: false
  enable.batch-mode: true
  # 测试新增
  sink.enable-delete: true

pipeline:
  name: sync_003
  parallelism: 1

route:
    - source-table: db05.t_detail_1_\.*
      sink-table: db05.t_detail_1_all
    - source-table: db05.t_detail_2_\.*
      sink-table: db05.t_detail_2_all




sync_004.yaml


source:
  type: mysql
  hostname: 192.168.1.2
  port: 3306
  username: xxxx
  password: 123456
  tables: db06.\.*,db06.\.*,db08.\.*
  server-id: 5631-5635
  server-time-zone: Asia/Shanghai

sink:
  type: doris
  fenodes: 192.168.1.30:8030
  username: xxx
  password: 123456
  table.create.properties.light_schema_change: true
  table.create.properties.replication_num: 3
  enable-2pc: false
  enable.batch-mode: true
  # 测试新增
  sink.enable-delete: true

pipeline:
  name: sync_004
  parallelism: 1







https://gitee.com/dyyx/work2024/tree/master/demo/mysql_data_auto_change