TungNT (Blue)

tungnt.blue@gmail.com

User Tools

Site Tools


development:database:mysql:debezium

This is an old revision of the document!


Debezium MySQL

Binlog (binary log)

Kiểm tra xem MySQL đã bật chế độ ghi log nhị phân hay chưa bằng lệnh:

SHOW VARIABLES LIKE 'log_bin';

Nếu kết quả là 'ON', binlog đang được bật.

Liệt kê các tệp binlog:

SHOW BINARY LOGS;

Xem nội dung binlog:

tungnt@MacBook-Pro-cua-Nguyen-2 mysql % mysqlbinlog /opt/homebrew/var/mysql/binlog.000217
# The proper term IS pseudo_replica_mode, but we USE this compatibility alias
# TO make the statement usable ON server versions 8.0.24 AND older.
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#241021 13:19:55 server id 1  end_log_pos 127 CRC32 0x609227a3 	START: binlog v 4, server v 8.4.2 created 241021 13:19:55 at startup
# Warning: this binlog IS either IN USE OR was NOT closed properly.
ROLLBACK/*!*/;
BINLOG '
i/IVZw8BAAAAewAAAH8AAAABAAQAOC40LjIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAACL8hVnEwANAAgAAAAABAAEAAAAYwAEGggAAAAAAAACAAAACgoKKioAEjQA
CigAAAGjJ5Jg
'/*!*/;
# at 127
#241021 13:19:55 server id 1  end_log_pos 158 CRC32 0xcf101e2a 	Previous-GTIDs
# [empty]
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# END OF log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

Cài đặt Kafka và Zookeeper

https://kafka.apache.org/downloads

Khởi động Zookeeper:

% zookeeper-server-start config/zookeeper.properties

Khởi động Kafka:

% kafka-server-start config/server.properties

Cài đặt Debezium

https://debezium.io/releases/

wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/1.9.5.Final/debezium-connector-mysql-1.9.5.Final-plugin.tar.gz
 
tar -xvf debezium-connector-mysql-1.9.5.Final-plugin.tar.gz
 
mkdir kafka_2.12-3.8.0/plugins/debezium-connector-mysql
 
cp -r debezium-connector-mysql kafka_2.12-3.8.0/plugins/debezium-connector-mysql

Cấu hình Kafka Connect

vim kafka_2.12-3.8.0/config/connect-standalone.properties
 
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/Users/tungnt/Downloads/Libs/kafka_2.12-3.8.0/plugins
vim kafka_2.12-3.8.0/config/mysql-connector.properties
 
name=mysql-connector
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=localhost
database.port=3306
database.user=xxx
database.password=xxx
database.server.id=184054
database.server.name=dbserver1
database.include.list=testdb
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=schema-changes.testdb

Khởi động Kafka Connect và Debezium

./bin/connect-standalone.sh config/connect-standalone.properties config/mysql-connector.properties

Một số lỗi có thể gặp

Lỗi Unable to connect: Plugin 'mysql_native_password' is not loaded

Lỗi Unable to connect: Plugin 'mysql_native_password' is not loaded xảy ra khi MySQL đang sử dụng một cơ chế xác thực khác (ví dụ: caching_sha2_password) thay vì mysql_native_password. Điều này thường gặp phải khi bạn sử dụng MySQL 8.0 trở lên, trong đó mặc định là caching_sha2_password, nhưng Debezium yêu cầu mysql_native_password để kết nối.

mysql -u root -p
development/database/mysql/debezium.1729762937.txt.gz · Last modified: 2024/10/24 09:42 by tungnt

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki