# 1.FileBeat ## 1. 1 读取Java日志 **1. 配置input读取Java日志** 日志格式不同，通过`multiline`分隔合并确定是一条日志，如下： ``` 2021-12-07 09:14:23.645 INFO 6 --- [io-10006-exec-4] com.yqcx.cp.core.filter.UserCPFilter : traceId=ef82e6f6b33a4e5e8d43ad77eecd91b7,URL:【http://localhost:8080/sysEquType/validateName】,耗时:【199】ms 2021-12-07 09:14:24.875 INFO 6 --- [io-10006-exec-5] com.yqcx.cp.core.filter.UserCPFilter : not allowedPath url [/sysEquType/validateName] 2021-12-07 09:14:24.875 INFO 6 --- [io-10006-exec-5] com.yqcx.cp.core.filter.UserCPFilter : UserCPFilter dealWithUser userId [1400279509741273090] Creating a new SqlSession SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@15f227b] was not registered for synchronization because synchronization is not active JDBC Connection [com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl@6f914542] will not be managed by Spring ==> Preparing: SELECT id,`name`,code,pid,company_id AS companyId,disable_flag AS disableFlag,disable_time AS disableTime,create_time AS createTime,create_user AS `createUser`,modified_time AS modifiedTime,modified_user AS modifiedUser FROM sys_equ_type WHERE (name = ? AND company_id = ?) ==> Parameters: eeeeeeeeeee(String), 1368801562103226369(Long) <== Total: 0 Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@15f227b] 2021-12-07 09:14:24.884 ERROR 6 --- [io-10006-exec-5] c.f.a.c.e.OpsGlobalExceptionHandler : 运行时异常: java.lang.NullPointerException: null at java.util.Objects.requireNonNull(Objects.java:203) at com.faw_qm.ad_ops.close_test.controller.SysEquTypeController.validateName(SysEquTypeController.java:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ``` 如下通过日期开始区分为一条日志，跨行合并，并简单输出到控制台 ``` filebeat.inputs: - type: log enabled: true paths: - /root/*.log multiline.type: pattern multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after output: console: codec.json: pretty: true escape_html: true ``` 启动，先不后台启动 ``` /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml ``` 成功读取到日志，并且按照日期分隔日志  ## 1.2 输出到logstash **1.修改output配置** ``` vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /root/*.log multiline.type: pattern multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after output: #console: # codec.json: # pretty: true # escape_html: true logstash: enabled: true hosts: ["192.168.56.10:5044"] ``` **2. 后台启动** ``` systemctl start filebeat ``` # 2. logstash ## 2.1 接收filebeat传入数据 logstash配置filebeat传入，并控制台打出测试 ``` input { beats { port => 5044 } } output { stdout { codec => rubydebug } } ``` 同样不要后台启动，使得可以观察到控制台打印 ``` /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat.conf ``` 可以看到控制台打印  ## 2.2 配置输出到es ``` vim /etc/logstash/conf.d/filebeat.conf input { beats { port => 5044 } } output { # stdout { # codec => rubydebug # } elasticsearch { hosts => ["http://192.168.56.10:9200"] #user => "elastic" #password => "qEnNfKNujqNrOPD9q5kb" index => "sys-log-%{+YYYY.MM.dd}" } } ```  在es中存储如下  异常日志（跨行也正常收集了） # 3. logstash过滤数据 ## 3.1 时间和无用字段