ClickHouse（八）监控与备份 · Clickhouse学习笔记

<div id="article_content" class="article_content clearfix"> <link rel="stylesheet" href="https://csdnimg.cn/release/blogv2/dist/mdeditor/css/editerView/kdoc_html_views-1a98987dfd.css"> <link rel="stylesheet" href="https://csdnimg.cn/release/blogv2/dist/mdeditor/css/editerView/ck_htmledit_views-25cebea3f9.css"> <div id="content_views" class="htmledit_views"> <h1><a name="t0"></a>1、<a href="https://so.csdn.net/so/search?q=ClickHouse&spm=1001.2101.3001.7020" target="_blank" class="hl hl-1" data-report-click="{"spm":"1001.2101.3001.7020","dest":"https://so.csdn.net/so/search?q=ClickHouse&spm=1001.2101.3001.7020","extra":"{\"searchword\":\"ClickHouse\"}"}" data-tit="ClickHouse" data-pretit="clickhouse">ClickHouse</a>监控概述</h1> <p>        ClickHouse运行时会将一些个自身的运行状态记录到众多系统表中（system.*）。所以我们对CH自身的一些运行指标的监控数据，也主要来自这些系统表。</p> <p>        但是直接查询这些系统表会有一些不足之处：</p> <ul><li>这种方式太过底层，不够直观，我们还需要在此之上实现可视化展示；</li><li>系统表只记录了CH自己的运行指标，有些时候我们需要外部系统的指标进行关联分析，例如Zookeeper、服务器CPU、IO等等。</li></ul> <p>        现在<a href="https://so.csdn.net/so/search?q=Prometheus&spm=1001.2101.3001.7020" target="_blank" class="hl hl-1" data-report-click="{"spm":"1001.2101.3001.7020","dest":"https://so.csdn.net/so/search?q=Prometheus&spm=1001.2101.3001.7020","extra":"{\"searchword\":\"Prometheus\"}"}" data-tit="Prometheus" data-pretit="prometheus">Prometheus</a> + Grafana的组合比较流行，安装简单易上手，可以集成很多框架，包括服务器的负载，<span style="color:#fe2c24;"><strong>其中Prometheus负责收集各类系统的运行指标；Grafana负责可视化的部分</strong></span>。</p> <p>        ClickHouse从v20.1.2.4开始，内置了对接Prometheus的功能，配置的方式也很简单，可以将其作为Prometheus的Endpoint服务，从而自动的将metrics、events和asynchronous_metrics三张系统的表的数据发送给Prometheus。</p> <h1><a name="t1"></a>2、Prometheus&<a href="https://so.csdn.net/so/search?q=Grafana&spm=1001.2101.3001.7020" target="_blank" class="hl hl-1" data-report-click="{"spm":"1001.2101.3001.7020","dest":"https://so.csdn.net/so/search?q=Grafana&spm=1001.2101.3001.7020","extra":"{\"searchword\":\"Grafana\"}"}" data-tit="Grafana" data-pretit="grafana">Grafana</a>的安装</h1> <p>        Prometheus下载地址：https://prometheus.io/download/</p> <p>        Grafana下载地址：https://grafana.com/grafana/download/</p> <h2><a name="t2"></a>2.1、安装Prometheus</h2> <p>        1、解压，修改配置文件prometheus.yml，添加如下</p> <pre data-index="0"><code class="hljs language-cobol"><ol class="hljs-ln" style="width:100%"><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="1"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-comment"> </span>- job_name: <span class="hljs-string">'clickhouse-1'</span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="2"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-comment"> st</span>atic_configs:</div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="3"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-comment"> </span>- targets: [<span class="hljs-string">'hadoop3:9363'</span>]</div></div></li></ol></code><div class="hljs-button {2}" data-title="复制" onclick="hljs.copyCode(event)"></div></pre> <p style="text-align:center;"><img alt="" height="502" src="https://img-blog.csdnimg.cn/63f7935f8ca0465088a54e9524d27e5d.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="711"></p> <p>        配置说明：</p> <ol><li><strong>global配置块</strong>：控制Prometheus服务器的全局配置 <ol><li>scrape_interval：配置拉取数据的时间间隔，默认为1分钟。</li><li>evaluation_interval：规则验证（生成alert）的时间间隔，默认为1分钟。</li></ol></li><li><strong>rule_file配置块</strong>：规则配置文件</li><li><strong>scrape_configs配置块：</strong>配置采集目标相关，prometheus监视的目标。Prometheus自身的运行信息可以通过HTTP访问，所以Prometheus可以监控自己的运行数据。 <ol><li>job_name：监控作业的名称。</li><li>static_configs:表示静态目标配置，就是固定从某个target拉取数据</li><li>targets：指定监控的目标，其实就是从哪里获取数据。Prometheu会从http://hadoop3:9363/metrics上拉取数据。</li></ol></li></ol> <p>        Prometheus是可以在运行时自动加载配置的。启动时需要添加：-web.enable-lifecycle</p> <p>        2、启动Prometheus Server</p> <pre data-index="1"><code class="language-bash hljs"><span class="hljs-built_in">nohup</span> ./prometheus --config.file=prometheus.yml > ./prometheus.log 2>&1 &</code><div class="hljs-button {2}" data-title="复制" onclick="hljs.copyCode(event)"></div></pre> <p>        3、浏览器输入：http://hadoop3:9090/。如下图所示，</p> <p style="text-align:center;"><img alt="" height="514" src="https://img-blog.csdnimg.cn/f20917425a97488f9906c887f4f5171b.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="1200"></p> <p>         prometheus是up状态，表示安装启动成功。</p> <h2><a name="t3"></a>2.2、Grafana安装</h2> <p>        启动Grafana：</p> <pre data-index="2"><code class="language-bash hljs"><span class="hljs-built_in">nohup</span> ./bin/grafana-server web > ./grafana.log 2>&1 &</code><div class="hljs-button {2}" data-title="复制" onclick="hljs.copyCode(event)"></div></pre> <p>        打开web：http://hadoop3:3000，默认用户名和密码是：admin</p> <h1><a name="t4"></a>3、ClickHouse配置</h1> <p>        修改每个节点下的配置文件，并重启服务：</p> <p>        编辑/etc/clickhouse-server/config.xml，打开如下配置：</p> <pre data-index="3"><code class="language-XML hljs"><ol class="hljs-ln" style="width:100%"><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="1"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-tag"><<span class="hljs-name">prometheus</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="2"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">endpoint</span>></span>/metrics<span class="hljs-tag"></<span class="hljs-name">endpoint</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="3"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">port</span>></span>9363<span class="hljs-tag"></<span class="hljs-name">port</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="4"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="5"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">metrics</span>></span>true<span class="hljs-tag"></<span class="hljs-name">metrics</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="6"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">events</span>></span>true<span class="hljs-tag"></<span class="hljs-name">events</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="7"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">asynchronous_metrics</span>></span>true<span class="hljs-tag"></<span class="hljs-name">asynchronous_metrics</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="8"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">status_info</span>></span>true<span class="hljs-tag"></<span class="hljs-name">status_info</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="9"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-tag"></<span class="hljs-name">prometheus</span>></span></div></div></li></ol></code><div class="hljs-button {2}" data-title="复制" onclick="hljs.copyCode(event)"></div></pre> <p>        网页打开http://hadoop3:9363/metrics，如下所示：</p> <p style="text-align:center;"><img alt="" height="602" src="https://img-blog.csdnimg.cn/c7bc0a5789364749b861ec1b8e7922a5.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="992"></p> <h1><a name="t5"></a>4、Grafana集成Prometheus</h1> <h2><a name="t6"></a>4.1、添加数据源Prometheus</h2> <p>        1、点击配置，点击Data Sources：</p> <p style="text-align:center;"><img alt="" height="821" src="https://img-blog.csdnimg.cn/590a58ffdea14c78bd24873b201d6757.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="1200"></p> <h2><a name="t7"></a>4.2、添加监控</h2> <p>        手动一个个添加Dashboard比较繁琐，Grafana社区鼓励用户分享Dashboard，通过https://grafana.com/dashboards网站，可以找到大量可直接使用的Dashboard模版。</p> <p>        Grafana中所有的Dashboard通过JSON进行共享，下载并且导入这些JSON文件，就可以直接使用这些已经定义好的Dashboard。</p> <p>        1、例如，点击Download JSON即可下载</p> <p style="text-align:center;"><img alt="" height="845" src="https://img-blog.csdnimg.cn/5c9e54c66bb84e4abcbf84f4e53cbe13.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="1200"></p> <p>         2、导入</p> <p style="text-align:center;"><img alt="" height="640" src="https://img-blog.csdnimg.cn/547202d08d814c1a994ae01db04e723e.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="1200"></p> <p>         3、最后展示如下：</p> <p style="text-align:center;"><img alt="" height="1016" src="https://img-blog.csdnimg.cn/5b6e50320e2a40d4afd93d91aeb6017f.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAWXVhbl9DU0RG,size_20,color_FFFFFF,t_70,g_se,x_16" width="1200"></p> </div><div data-report-view="{"mod":"1585297308_001","spm":"1001.2101.3001.6548","dest":"https://blog.csdn.net/Yuan_CSDF/article/details/122569175","extend1":"pc","ab":"new"}"><div></div></div> </div>