多应用+插件架构,代码干净,二开方便,首家独创一键云编译技术,文档视频完善,免费商用码云13.8K 广告
<blockquote class="success">注意事项</blockquote> ``` 第一次执行全量同步,后续的执行增量同步。注意全量同步时,关闭logstash定时任务设置,否则可能导致虚拟内存不足程序崩溃和数据重复问题。 ``` ### 一、下载mysql驱动 mysql-connector-java下载地址: [https://mvnrepository.com/artifact/mysql/mysql-connector-java](https://mvnrepository.com/artifact/mysql/mysql-connector-java) 选择自己的版本: ![](http://cloud.ryloo.icu/da7be029d9b0b264252ef010e5aeec55) 选择对应的jar包: ![](http://cloud.ryloo.icu/a6747f1c2f6821a439df04da70baaf53) ### 二、上传mysql驱动 上传到 /usr/local/logstash/logstash-7.6.2/logstash-core/lib/jars 目录下 ![](http://cloud.ryloo.icu/9be699be8e59927436cb67bed5038261) ### 三、配置jdbc.conf ``` input { jdbc { # 驱动 jdbc_driver_library => "/usr/local/logstash/logstash-7.6.2/logstash-core/lib/jars/mysql-connector-java-8.0.26.jar" #JDBC连接Mysql6需用com.mysql.cj.jdbc.Driver,同时需要指定时区serverTimezone jdbc_driver_class => "com.mysql.cj.jdbc.Driver" jdbc_default_timezone => "Asia/Shanghai" #jdbc_paging_enabled => "true" #jdbc_page_size => "50000" # mysql数据库链接,test为数据库名 jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test" jdbc_user => "root" jdbc_password => "root" # 数据库重连尝试次数 #connection_retry_attempts => "3" # 执行的sql语句 #statement => "SELECT * FROM law_advices WHERE id > :sql_last_value" statement_filepath => "/usr/local/logstash/logstash-7.6.2/config/law_advices.sql" # 定时任务 schedule => "* * * * *" # 需要记录查询结果某字段的值时,此字段为true use_column_value => true # 需要记录的字段,用于增量同步 tracking_column => id # 记录上一次运行记录 record_last_run => true # 上一次运行记录值的存放文件路径 last_run_metadata_path => "/usr/local/logstash/logstash-7.6.2/config/sync_log.txt" # 是否清除last_run_metadata_path的记录,需要增量同步时此字段必须为false; #clean_run => false } } output { elasticsearch { hosts => ["localhost:9200"] index => "law_advices" document_id => "%{id}" } # 这里输出调试,正式运行时可以注释掉 stdout { codec => json_lines } } ``` mysql-connector-java与mysql版本的对应 ![](http://cloud.ryloo.icu/474027909279b8c9e3205dcdbbbbc59d) ### 四、上传SQL文件law_advices.sql和日志记录文件sync_log.txt ![](http://cloud.ryloo.icu/d0d7122b6888bf800687354e54f98960) #### 执行同步数据 ``` cd /usr/local/logstash/logstash-7.6.2/bin/ ./logstash -f /usr/local/logstash/logstash-7.6.2/config/jdbc.conf nohup ./logstash -f /usr/local/logstash/logstash-7.6.2/config/law_advices_v1.conf >/dev/null & ``` ![](http://cloud.ryloo.icu/452f160fa30177202d6a8e72f9213883) ### 五、推荐阅读 1、# [logstash全量和增量同步数据到mysql](https://blog.csdn.net/lumengmeng_csdn/article/details/89482086)