elasticsearch6所需配置文件 · 运维知识记录

[TOC] ### 目录结构假设安装在 /opt/elk6目录下整体目录为 ``` root@test:~# tree -d -L 2 /opt/ /opt/ └── elk6 ├── elasticsearch -> elasticsearch-6.1.2/ ├── elasticsearch-6.1.2 ├── elasticsearch-head ├── kibana -> kibana-6.1.2-linux-x86_64/ ├── kibana-6.1.2-linux-x86_64 ├── logstash -> logstash-6.1.2/ └── logstash-6.1.2 ``` elasticsearch目录为 ``` root@Haproxy-A:/opt/elk6# tree -Ld 1 elasticsearch elasticsearch ├── bin ├── config ├── default ├── lib ├── logs ├── modules ├── pid └── plugins ``` ### 目录用户要非root ``` root@test:/opt# ls -lh elk6/ total 16K lrwxrwxrwx 1 elasticsearch elasticsearch 20 Jan 19 17:31 elasticsearch -> elasticsearch-6.1.2/ drwxr-xr-x 10 elasticsearch elasticsearch 4.0K Feb 28 14:35 elasticsearch-6.1.2 drwxr-xr-x 8 root root 4.0K Jan 19 19:08 elasticsearch-head lrwxrwxrwx 1 elasticsearch elasticsearch 26 Jan 19 17:32 kibana -> kibana-6.1.2-linux-x86_64/ drwxrwxr-x 16 elasticsearch elasticsearch 4.0K Jan 19 18:12 kibana-6.1.2-linux-x86_64 lrwxrwxrwx 1 root root 15 Jan 19 17:32 logstash -> logstash-6.1.2/ drwxr-xr-x 11 root root 4.0K Jan 19 17:31 logstash-6.1.2 ``` ### 主配置文件 ``` root@test:~# cat /opt/elk6/elasticsearch/config/elasticsearch.yml # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: elk6 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: elk6-node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /data/elk6_data # # Path to log files: # path.logs: /opt/elk6/elasticsearch/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.100.100 # # Set a custom port for HTTP: # http.port: 9500 transport.tcp.port: 9600 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["192.168.5.30:9600"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: 2 # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true http.cors.enabled: true http.cors.allow-origin: "*" ``` ### default环境变量文件 ``` root@test:~# cat /opt/elk6/elasticsearch/default/elasticsearch ################################ # Elasticsearch ################################ # Elasticsearch home directory ES_HOME=/opt/elk6/elasticsearch # Elasticsearch Java path JAVA_HOME=/application/jdk # Elasticsearch configuration directory ES_PATH_CONF=/opt/elk6/elasticsearch/config # Elasticsearch PID directory PID_DIR=/opt/elk6/elasticsearch/pid # Additional Java OPTS #ES_JAVA_OPTS= # Configure restart on package upgrade (true, every other setting will lead to not restarting) #RESTART_ON_UPGRADE=true ################################ # Elasticsearch service ################################ # SysV init.d # # The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process ES_STARTUP_SLEEP_TIME=5 ################################ # System properties ################################ # Specifies the maximum file descriptor number that can be opened by this process # When using Systemd, this setting is ignored and the LimitNOFILE defined in # /usr/lib/systemd/system/elasticsearch.service takes precedence MAX_OPEN_FILES=65536 # The maximum number of bytes of memory that may be locked into RAM # Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option # in elasticsearch.yml. # When using systemd, LimitMEMLOCK must be set in a unit file such as # /etc/systemd/system/elasticsearch.service.d/override.conf. MAX_LOCKED_MEMORY=unlimited # Maximum number of VMA (Virtual Memory Areas) a process can own # When using Systemd, this setting is ignored and the 'vm.max_map_count' # property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf MAX_MAP_COUNT=262144 ``` ### jvm配置文件 ``` root@test:/opt/elk6/elasticsearch/config# cat jvm.options ## JVM configuration ################################################################ ## IMPORTANT: JVM heap size ################################################################ ## ## You should always set the min and max JVM heap ## size to the same value. For example, to set ## the heap to 4 GB, set: ## ## -Xms4g ## -Xmx4g ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html ## for more information ## ################################################################ # Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms4g -Xmx4g ################################################################ ## Expert settings ################################################################ ## ## All settings below this section are considered ## expert settings. Don't tamper with them unless ## you understand what you are doing ## ################################################################ ## GC configuration -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly ## optimizations # pre-touch memory pages used by the JVM during initialization -XX:+AlwaysPreTouch ## basic # force the server VM -server # explicitly set the stack size -Xss1m # set to headless, just in case -Djava.awt.headless=true # ensure UTF-8 encoding by default (e.g. filenames) -Dfile.encoding=UTF-8 # use our provided JNA always versus the system one -Djna.nosys=true # turn off a JDK optimization that throws away stack traces for common # exceptions because stack traces are important for debugging -XX:-OmitStackTraceInFastThrow # flags to configure Netty -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 # log4j 2 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true ## heap dumps # generate a heap dump when an allocation from the Java heap fails # heap dumps are created in the working directory of the JVM -XX:+HeapDumpOnOutOfMemoryError # specify an alternative path for heap dumps # ensure the directory exists and has sufficient space #-XX:HeapDumpPath=/heap/dump/path ## GC logging #-XX:+PrintGCDetails #-XX:+PrintGCTimeStamps #-XX:+PrintGCDateStamps #-XX:+PrintClassHistogram #-XX:+PrintTenuringDistribution #-XX:+PrintGCApplicationStoppedTime # log GC status to a file with time stamps # ensure the directory exists #-Xloggc:${loggc} # By default, the GC log file will not rotate. # By uncommenting the lines below, the GC log file # will be rotated every 128MB at most 32 times. #-XX:+UseGCLogFileRotation #-XX:NumberOfGCLogFiles=32 #-XX:GCLogFileSize=128M ``` ### systemd启动管理文件 ``` root@test:~# cat /usr/lib/systemd/system/elasticsearch.service [Unit] Description=Elasticsearch Documentation=http://www.elastic.co Wants=network-online.target After=network-online.target [Service] RuntimeDirectory=elasticsearch Environment=ES_HOME=/opt/elk6/elasticsearch Environment=ES_PATH_CONF=/opt/elk6/elasticsearch/config Environment=PID_DIR=/opt/elk6/elasticsearch/pid EnvironmentFile=-/opt/elk6/elasticsearch/default/elasticsearch WorkingDirectory=/opt/elk6/elasticsearch User=elasticsearch Group=elasticsearch ExecStart=/opt/elk6/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet # StandardOutput is configured to redirect to journalctl since # some error messages may be logged in standard output before # elasticsearch logging system is initialized. Elasticsearch # stores its logs in /var/log/elasticsearch and does not use # journalctl by default. If you also want to enable journalctl # logging, you can simply remove the "quiet" option from ExecStart. StandardOutput=journal StandardError=inherit # Specifies the maximum file descriptor number that can be opened by this process LimitNOFILE=65536 # Specifies the maximum number of processes LimitNPROC=4096 # Specifies the MEMLOCK LimitMEMLOCK=infinity # Specifies the maximum size of virtual memory LimitAS=infinity # Specifies the maximum file size LimitFSIZE=infinity # Disable timeout logic and wait until process is stopped TimeoutStopSec=0 # SIGTERM signal is used to stop the Java process KillSignal=SIGTERM # Send the signal only to the JVM rather than its control group KillMode=process # Java process is never killed SendSIGKILL=no # When a JVM receives a SIGTERM signal it exits with code 143 SuccessExitStatus=143 [Install] WantedBy=multi-user.target # Built for distribution-6.1.2 (distribution) ``` ### init.d启动脚本 ``` root@test:~# cat /etc/init.d/elasticsearch #!/bin/bash # # /etc/init.d/elasticsearch -- startup script for Elasticsearch # ### BEGIN INIT INFO # Provides: elasticsearch # Required-Start: $network $remote_fs $named # Required-Stop: $network $remote_fs $named # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Starts elasticsearch # Description: Starts elasticsearch using start-stop-daemon ### END INIT INFO JAVA_HOME=/application/jdk PATH=/bin:/usr/bin:/sbin:/usr/sbin NAME=elasticsearch DESC="Elasticsearch Server" DEFAULT=/opt/elk6/elasticsearch/default/${NAME} if [ `id -u` -ne 0 ]; then echo "You need root privileges to run this script" exit 1 fi . /lib/lsb/init-functions if [ -r /etc/default/rcS ]; then . /etc/default/rcS fi # The following variables can be overwritten in $DEFAULT # Directory where the Elasticsearch binary distribution resides ES_HOME=/opt/elk6/$NAME # Additional Java OPTS #ES_JAVA_OPTS= # Maximum number of open files MAX_OPEN_FILES=65536 # Maximum amount of locked memory #MAX_LOCKED_MEMORY= # Elasticsearch configuration directory ES_PATH_CONF=$ES_HOME/config # Maximum number of VMA (Virtual Memory Areas) a process can own MAX_MAP_COUNT=262144 # Elasticsearch PID file directory PID_DIR="$ES_HOME/pid" # End of variables that can be overwritten in $DEFAULT # overwrite settings from default file if [ -f "$DEFAULT" ]; then . "$DEFAULT" fi # ES_USER and ES_GROUP settings were removed if [ ! -z "$ES_USER" ] || [ ! -z "$ES_GROUP" ]; then echo "ES_USER and ES_GROUP settings are no longer supported. To run as a custom user/group use the archive distribution of Elasticsearch." exit 1 fi # Define other required variables PID_FILE="$PID_DIR/$NAME.pid" DAEMON=$ES_HOME/bin/elasticsearch DAEMON_OPTS="-d -p $PID_FILE" export ES_JAVA_OPTS export JAVA_HOME export ES_PATH_CONF if [ ! -x "$DAEMON" ]; then echo "The elasticsearch startup script does not exists or it is not executable, tried: $DAEMON" exit 1 fi checkJava() { if [ -x "$JAVA_HOME/bin/java" ]; then JAVA="$JAVA_HOME/bin/java" else JAVA=`which java` fi if [ ! -x "$JAVA" ]; then echo "Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME" exit 1 fi } case "$1" in start) checkJava log_daemon_msg "Starting $DESC" pid=`pidofproc -p $PID_FILE elasticsearch` if [ -n "$pid" ] ; then log_begin_msg "Already running." log_end_msg 0 exit 0 fi # Ensure that the PID_DIR exists (it is cleaned at OS startup time) if [ -n "$PID_DIR" ] && [ ! -e "$PID_DIR" ]; then mkdir -p "$PID_DIR" && chown elasticsearch:elasticsearch "$PID_DIR" fi if [ -n "$PID_FILE" ] && [ ! -e "$PID_FILE" ]; then touch "$PID_FILE" && chown elasticsearch:elasticsearch "$PID_FILE" fi if [ -n "$MAX_OPEN_FILES" ]; then ulimit -n $MAX_OPEN_FILES fi if [ -n "$MAX_LOCKED_MEMORY" ]; then ulimit -l $MAX_LOCKED_MEMORY fi if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT fi # Start Daemon start-stop-daemon -d $ES_HOME --start --user elasticsearch -c elasticsearch --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS return=$? if [ $return -eq 0 ]; then i=0 timeout=10 # Wait for the process to be properly started before exiting until { kill -0 `cat "$PID_FILE"`; } >/dev/null 2>&1 do sleep 1 i=$(($i + 1)) if [ $i -gt $timeout ]; then log_end_msg 1 exit 1 fi done fi log_end_msg $return exit $return ;; stop) log_daemon_msg "Stopping $DESC" if [ -f "$PID_FILE" ]; then start-stop-daemon --stop --pidfile "$PID_FILE" \ --user elasticsearch \ --quiet \ --retry forever/TERM/20 > /dev/null if [ $? -eq 1 ]; then log_progress_msg "$DESC is not running but pid file exists, cleaning up" elif [ $? -eq 3 ]; then PID="`cat $PID_FILE`" log_failure_msg "Failed to stop $DESC (pid $PID)" exit 1 fi rm -f "$PID_FILE" else log_progress_msg "(not running)" fi log_end_msg 0 ;; status) status_of_proc -p $PID_FILE elasticsearch elasticsearch && exit 0 || exit $? ;; restart|force-reload) if [ -f "$PID_FILE" ]; then $0 stop fi $0 start ;; *) log_success_msg "Usage: $0 {start|stop|restart|force-reload|status}" exit 1 ;; esac exit 0 ```