2. Logstash实战 · 运维工作笔记

#### 请先看看结尾的坑 ### logstash注意事项 - ##### logstash可以启动多个端口接收数据 - ##### 重启logstash可能会卡住，一般由于输出到其他服务导致（Reids服务没启动，验证没通过），一般会选择kill - ##### 执行前先检查语法，并前台测试启动 #### 检查logstash语法 ```shell /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t # logstash前台启动 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf ``` 测试阶段可以将收集的日志输出到本地文件中，调试后发往目标 ### 收集日志，存储在本地文件 ```shell #cat /etc/logstash/conf.d/system.conf input{ file { type => "systemlog" path => "/var/log/messages" start_position => "beginning" stat_interval => "5" } } output { file { path => "/tmp/systemlog.log" } } ``` ### 使用logstash收集系统messages日志文件无权限报错 ```shell [2017-08-29T16:50:00,834][INFO ][logstash.pipeline ] Pipeline main started [2017-08-29T16:50:00,852][WARN ][logstash.inputs.file ] failed to open /var/log/messages: Permission denied - /var/log/messages [2017-08-29T16:50:00,886][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} ``` ```shell #系统日志，默认600,会报错 chmod 644 /var/log/messages ``` #### logstash中增加配置（/etc/logstash/conf.d/systemlog.conf） ```shell #cat /etc/logstash/conf.d/system.conf input{ file { type => "systemlog" path => "/var/log/messages" start_position => "beginning" stat_interval => "5" } } output{ elasticsearch{ hosts => ["192.168.0.231:9200"] index => "logstash-systemlog-%{+YYYY.MM.dd}" } } ``` ### logstash收集Nginx日志 #### Nginx配置成Json格式日志配置摘自[jack.zhang的博客](http://www.cnblogs.com/zhang-shijie/p/5384624.html "jack.zhang的博客")，感谢分享 ```json log_format logstash_json '{"@timestamp":"$time_local",' '"remote_addr":"$remote_addr",' '"remote_user":"$remote_user",' '"body_bytes_sent":"$body_bytes_sent",' '"request_time":"$request_time",' '"status":"$status",' '"request":"$request",' '"request_method":"$request_method",' '"http_referrer":"$http_referer",' '"body_bytes_sent":"$body_bytes_sent",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"http_user_agent":"$http_user_agent"}'; access_log /var/log/nginx/access.log logstash_json; ``` json日志 ```json { "@timestamp": "30/Aug/2017:10:18:59 +0800", "remote_addr": "192.168.2.64", "remote_user": "-", "body_bytes_sent": "0", "request_time": "0.000", "status": "304", "request": "GET /nginxweb/ HTTP/1.1", "request_method": "GET", "http_referrer": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" } ``` logstash中增加配置（/etc/logstash/conf.d/nginx.conf） ```json input { file { path => "/var/log/nginx/access.log" type => "nginx-accesslog" start_position => "beginning" } } output { if [type] == "nginx-accesslog" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "nginx-accesslog-%{+YYYY.MM.dd}" } } } ``` ### logstash收集Tomcat访问日志 #### 配置Tomcat日志为Json格式配置摘自[jack.zhang的博客](http://www.cnblogs.com/zhang-shijie/p/5384624.html "jack.zhang的博客")，感谢分享 #### 修改tomcat/conf/server.xml结尾部分的日志配置修改日志名称，结尾格式，和pattern ```xml <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="tomcat_access" suffix=".log" pattern="{"client":"%h","client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s","send bytes":"%b","Query?string":"%q","partner":"%{Referer}i","Agent version":"%{User-Agent}i"}"/> ``` 重启Tomcat，查看日志格式 ```json { "client": "192.168.2.64", "client user": "-", "authenticated": "-", "access time": "[31/Aug/2017:09:48:29 +0800]", "method": "GET /web/index.html HTTP/1.1", "status": "304", "send bytes": "-", "Query?string": "", "partner": "-", "Agent version": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" } ``` logstash中增加配置（/etc/logstash/conf.d/tomcat.conf） ```json input{ file { path => "/app/tomcat1/logs/tomcat_access.*.log" type => "tomcatlog" start_position => "beginning" stat_interval => "5" } } output{ if[type] == "tomcatlog" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "tomcatlog-%{+YYYY.MM.dd}" } } } ``` ### rsyslog收集HAproxy日志，发送到logstash #### haproxy日志配置设置HAproxy日志输出到local6 ```shell global ...(略) log 127.0.0.1 local6 info ...(略) ``` #### rsyslog配置将local6的所有日志发送到logstash的5555端口 ```shell # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 local6.* @@192.168.0.230:5555 ``` #### logstash配置 ```shell input { syslog { type => "rsyslog-haproxy-8888" port => "5555" } } output{ if [type] == "rsyslog-haproxy-8888" { elasticsearch { hosts => ["192.168.0.232:9200"] index => "haproxy-%{+YYYY.MM.dd}" } } } ``` ### 一个logstash配置文件中，收集多个日志，建立不同索引 #### logstash中增加配置（/etc/logstash/conf.d/systemlog.conf） ```shell input{ file { #定义type类型，用于输出判断 type => "systemlog" path => "/var/log/messages" start_position => "beginning" stat_interval => "5" } file { path => "/var/log/lastlog" #定义type类型，用于输出判断 type => "system-last" start_position => "beginning" stat_interval => "5" } } output{ #判断输入类型，输出不同索引名称 if [type] == "systemlog" { elasticsearch{ hosts => ["192.168.0.231:9200"] index => "logstash-systemlog-%{+YYYY.MM.dd}" } } #判断输入类型，输出不同索引名称 if [type] == "system-last" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "logstash-lastlog-%{+YYYY.MM.dd}" } } } ``` ### logstash收集beats组件信息，并转发到不同的目标 ```shell input { beats { port => 5044 } } output { if [type] == "web02-tomcat-info" { redis { host => ["192.168.0.106"] data_type => "list" db => "3" key => "web02-tomcat-info" port => "6400" password => "123456" batch => "true" } } if [type] == "web02-tomcat-error" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "web02-tomcat-error" } } } ``` ### logstash消费redis中的日志 ```shell input { redis { data_type => "list" db => "3" host => "192.168.0.106" port => "6400" key => "web02-tomcat-info" password => "123456" } } output { if [type] == "web02-tomcat-info" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "tomcat-info-%{+YYYY.MM.dd}" } } } ``` ### 跳坑 #### 坑1. A数据，写入到了B索引中，造成数据混乱 ##### ELK版本: 5.5.2 现状描述：logstash的配置中，如果有A日志配置中使用了if [type]判断，B日志收集配置未指定if[type]，那么，B的索引中，会写入A中的数据（如果有C配置，设置了if[type]，C的数据也会写入到B中）目前只判断了type，其他判断未测试结论：如果logstash中使用了if[type]判断，所有配置中都要使用判断 #### 坑2. 644也无法读取日志也许有时候需要777，rpm装的nginx，日志目录和文件644也没权限读 nginx（rpm）日志权限 ```shell Nginx yum安装，日志位置：/var/log/nginx/access.log /var/log/nginx/权限为700 /var/log/nginx/access.log权限为644 以上情况获取不到日志把/var/log/nginx权限调整为644，依然获取不到日志，直到调整为777，获取日志正常。 ```