快速安装部署ELK · linux运维

[TOC] ## 一安装JDK8环境和reids ### A JDK安装 jkd版本:`jdk-8u151-linux-x64.rpm`,大版本一致即可 jdk下载链接可能因为小版本更新导致不可用,可以进下面链接找到最新的jdk8的下载连接系统版本选择:https://pkgs.org/download/java 当前最小版本:https://forensics.cert.org/centos/cert/7/x86_64//jdk-8u221-linux-x64.rpm ```sh mkdir -p /server/tools/ cd /server/tools/ wget https://forensics.cert.org/centos/cert/7/x86_64//jdk-8u221-linux-x64.rpm rpm -ivh jdk-8u221-linux-x64.rpm ``` ### B redis快速安装 redis不是这里要学习的重点,所以只需快速部署启动即可 ```sh yum install -y redis sed -i 's#127.0.0.1#10.0.0.11#g' /etc/redis.conf sed -i '/# requirepass/i requirepass abcd1234e' /etc/redis.conf cp /etc/redis.conf{,.bak} egrep -v "^#|^$" /etc/redis.conf.bak >/etc/redis.conf systemctl start redis.service [root@file_redis ~]# ss -lntuo|grep 6379 tcp LISTEN 0 128 10.0.0.11:6379 *:* ``` ## 二 Filebeat快速安装配置 **Filebeat的工作原理：** 启动Filebeat时，它会启动一个或多个inputs，这些inputs将查找指定的log的路径。对于查找到的每个日志，Filebeat将启动一个harvester。每个harvester读取单个日志的新内容，并将新日志数据发送到libbeat，libbeat聚合事件并将聚合数据发送到配置的output。 ### A filebeat安装 ```sh #ELK的安装源在上一章已经部署好 yum install -y filebeat cp /etc/filebeat/filebeat.yml{,.bak} egrep -v "#|^$" /etc/filebeat/filebeat.yml.bak >/etc/filebeat/filebeat.yml ``` ### B 当前filebeat配置 ```yml [root@file_redis ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: false paths: - /var/log/*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: output.elasticsearch: hosts: ["localhost:9200"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~ ``` ### C 精简并修改配置收集ssh登录日志,打上tag:`ssh`,存入redis第二个库中,并制定key名为`filebeat-1101` ```sh [root@zhimai-test ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/secure tags: ["web","ssh"] output.redis: hosts: ["10.0.0.11"] port: 6379 password : 'abcd1234e' key: "filebeat-1011" db: 2 timeout: 5 ``` 清空安全ssh日志,启动filebeat ```sh >/var/log/secure systemctl start filebeat.service redis-cli -h 10.0.0.11 -a abcd1234e # 查看redis是否有数据 10.0.0.11:6379> select 2 OK 10.0.0.11:6379[2]> keys * (empty list or set) #现在看到的redis库中还没有数据 ``` 新开xshell窗口登录服务器后,再查看redis信息 ```sh 10.0.0.11:6379[2]> keys * 1) "filebeat-1101" 10.0.0.11:6379[2]> lrange filebeat-1011 0 -1 ...很多数据,省略显示... ``` ## 三安装部署elasticserach ### A 安装并配置es ```sh yum install -y elasticsearch sed -i '/^#network.host:/a network.host: 10.0.0.12' /etc/elasticsearch/elasticsearch.yml sed -i '/^#http.port:/a http.port: 9200' /etc/elasticsearch/elasticsearch.yml cp /etc/elasticsearch/elasticsearch.yml{,.bak} egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml.bak >/etc/elasticsearch/elasticsearch.yml systemctl start elasticsearch.service ``` ### B 验证es启动结果 ```sh [root@log_es ~]# ss -lntup|grep 9200 tcp LISTEN 0 128 ::ffff:10.0.0.12:9200 :::* users:(("java",pid=31378,fd=197)) [root@log_es ~]# curl 10.0.0.12:9200 { "name" : "GCQ8mIk", "cluster_name" : "elasticsearch", "cluster_uuid" : "QIjZQfUrQwO0Keh-XPn8QQ", "version" : { "number" : "6.8.4", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "bca0c8d", "build_date" : "2019-10-16T06:19:49.319352Z", "build_snapshot" : false, "lucene_version" : "7.7.2", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } ``` ## 四安装部署logstash ### A 安装并验证logstash **安装logstash:** ```sh yum install -y logstash ``` **命令行启动验证** 通过命令行启动,验证屏幕输入是否输出是否正常,这是调试logstash的常用方式 ```sh [root@log_es ~]# /usr/share/logstash/bin/logstash -e "input { stdin { type => stdin } } output { stdout { codec => rubydebug } }" ....启动过程....省略..... abcd test { "@timestamp" => 2019-11-15T03:43:55.953Z, "message" => "abcd test", "type" => "stdin", "host" => "log_es", "@version" => "1" } # "abcd test" 是键盘输入的内容,后面的是输出到屏幕的内容 ``` **配置文件启动验证** 将命令行内容写入配置文件,然后启动验证 ``` #1. 配置文件 [root@log_es ~]# cat /etc/logstash/conf.d/test.conf input { stdin { type => "stdin" } } output { stdout { codec => rubydebug } } #2. 验证配置文件 [root@log_es ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t ...... Configuration OK ...... 3. 用配置文件启动 [root@log_es ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf ........ [INFO ] 2019-11-15 17:30:35.777 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} conf test { "@version" => "1", "host" => "log_es", "@timestamp" => 2019-11-15T09:32:24.835Z, "message" => "conf test", "type" => "stdin" } ``` ### B 编写配置文件 **配置文件** ```sh [root@log_es ~]# cat /etc/logstash/conf.d/test.conf input { redis { data_type => "list" key => "filebeat-1011" host => "10.0.0.11" port => 6379 password => 'abcd1234e' db => "2" threads => 5 codec => "json" } } output { elasticsearch { hosts => ["10.0.0.12:9200"] index => "1011-logs-%{+YYYY.MM}" } } ``` ### C 启动logstash并在es中查看结果 ```sh [root@log_es ~]# systemctl restart logstash.service [root@log_es ~]# ss -lntup|grep 9600 tcp LISTEN 0 50 ::ffff:127.0.0.1:9600 :::* users:(("java",pid=33956,fd=104)) [root@log_es ~]# curl 10.0.0.12:9200/_cat/indices yellow open 1011-logs-2019.11 zJ-O37DWSzKfGWv6mXJnYw 5 1 5 0 45.3kb 45.3kb [root@log_es ~]# curl -XGET 'http://10.0.0.12:9200/1011-logs-2019.11' .....一大串json格式内容....... ``` ## 五安装部署kibana ### A **安装启动kibana:** ```sh yum install -y kibana cp /etc/kibana/kibana.yml{,.bak} # 写入配置文件 cat >/etc/kibana/kibana.yml <<EOF server.port: 5601 server.host: "10.0.0.12" elasticsearch.url: "http://10.0.0.12:9200" EOF # 启动kibana systemctl start kibana ``` ### B 浏览器访问kibana 浏览器访问`http://10.0.0.12:5601`,可以进入kibana的web界面即可 ## 结束语至此,filebeat+redis+logstash+elasticsearch+kibana的ELK架构已经搭建完成,关于各个组件的详细内容,在后续章节再讨论