Thanos Receive 模塊介紹

起初（大概2020上半年之前）使用 thanos 作 prometheus分 布式監(jiān)控方案的時(shí)候，thanos receive 模塊還在試驗(yàn)階段，當(dāng)時(shí)是使用的 thanos sidecar 方式。現(xiàn)在此功能模塊已經(jīng)被社區(qū)正式接受，功能會相對穩(wěn)定了，因此，考慮部分場景使用 receive 模塊替換 sidecar。

目標(biāo)

通過使用receive模塊，期望做到：

• 收攏分散的prometheus采集數(shù)據(jù)，減少sidecar數(shù)量（如果thanos query后面掛過多sidecar會影響性能）
• 盡可能減少采集prometheus實(shí)例本地存儲數(shù)據(jù)量，使重啟、故障恢復(fù)時(shí)間更短

架構(gòu)介紹

架構(gòu)圖

                 +
Tenant's Premise | Provider Premise
                 |
                 |            +------------------------+
                 |            |                        |
                 |  +-------->+     Object Storage     |
                 |  |         |                        |
                 |  |         +-----------+------------+
                 |  |                     ^
                 |  | S3 API              | S3 API
                 |  |                     |
                 |  |         +-----------+------------+
                 |  |         |                        |       Store API
                 |  |         |  Thanos Store Gateway  +<-----------------------+
                 |  |         |                        |                        |
                 |  |         +------------------------+                        |
                 |  |                                                           |
                 |  +---------------------+                                     |
                 |                        |                                     |
+--------------+ |            +-----------+------------+              +---------+--------+
|              | | Remote     |                        |  Store API   |                  |
|  Prometheus  +------------->+     Thanos Receiver    +<-------------+  Thanos Querier  |
|              | | Write      |                        |              |                  |
+--------------+ |            +------------------------+              +---------+--------+
                 |                                                              ^
                 |                                                              |
+--------------+ |                                                              |
|              | |                PromQL                                        |
|    User      +----------------------------------------------------------------+
|              | |
+--------------+ |
                 +

功能說明

負(fù)載分發(fā)

為了支持多臺機(jī)器的擴(kuò)展，時(shí)間序列將被分發(fā)到不同的receiver上。receiver是通過hash的方式來實(shí)現(xiàn)時(shí)間序列的持續(xù)分發(fā)的。每個(gè)receiver在hashring中的位置決定了哪些時(shí)間序列被哪個(gè)receiver接收和存儲。

remote write api過來的請求，經(jīng)過receiver前面的負(fù)載均衡，然后請求隨機(jī)落到receiver節(jié)點(diǎn)上。這時(shí)，會計(jì)算時(shí)間序列標(biāo)簽的哈希值。另外，考慮到對于量很大的時(shí)間序列，不希望全都分發(fā)到某一個(gè)receiver節(jié)點(diǎn)上，通過參數(shù)--receive.tenant-header指定的tenant ID（如果沒有指定，默認(rèn)為空字符串）也會用在hash值的計(jì)算中。大致的hash計(jì)算方式如下：

hash(string(tenant_id) + sort(timeseries.labelset).join())

如果receiver節(jié)點(diǎn)接收到的請求計(jì)算出來的hash值不是分發(fā)到當(dāng)前節(jié)點(diǎn)，當(dāng)前receiver會把他分發(fā)到該去的那個(gè)receiver節(jié)點(diǎn)。這個(gè)路由的功能本可以做成一個(gè)獨(dú)立的模塊，但是考慮到為了便于部署，這個(gè)功能是直接集成在receiver中了。

                                  Soft tenant hashring
                                 +-----------------------+
                                 |                       |
+-----------------+              |  +-----------------+  |
|                 |              |  |                 |  |
|  Load Balancer  +-------+      |  | Thanos receiver |  |
|                 |       |      |  |                 |  |
+-----------------+       |      |  +-----------------+  |
                          |      |                       |
                          |      |                       |
                          |      |  +-----------------+  |
                          |      |  |                 |  |
                          +-------->+ Thanos receiver +-----------+
                                 |  |                 |  |        |
                                 |  +-----------------+  |        |
                                 |                       |        |
                                 +-----------------------+        |
                                                                  |
                                   Hard Tenant A hashring         |
                                 +-----------------------+        |
                                 |                       |        |
                                 |  +-----------------+  |        |
                                 |  |                 |  |        |
                                 |  | Thanos receiver +<----------+
                                 |  |                 |  |        |
                                 |  +-----------------+  |        |
                                 |                       |        |
                                 |                       |        |
                                 |  +-----------------+  |        |
                                 |  |                 |  |        |
                                 |  | Thanos receiver +<----------+
                                 |  |                 |  |
                                 |  +-----------------+  |
                                 |                       |
                                 +-----------------------+

多副本

thanos receiver支持復(fù)制接受到的時(shí)間序列到hashring中的其他receiver。副本數(shù)由參數(shù)--receive.replication-factor來指定。如果被接收的時(shí)間序列，沒有寫入到(副本數(shù)+1)/2節(jié)點(diǎn)數(shù)（即要達(dá)到副本數(shù)半數(shù)以上），receiver將返回錯誤。

例如，要存儲3份時(shí)間序列的副本，則hashring中至少要有2個(gè)目標(biāo)thanos receiver才行；并且，需要receiver配置此參數(shù)：--receive.replication-factor=3。

receiver節(jié)點(diǎn)的縮容/擴(kuò)容/失敗場景

當(dāng)發(fā)生擴(kuò)縮容的時(shí)候，由于hashring發(fā)生變化，所有的節(jié)點(diǎn)需要將write-ahead-log的數(shù)據(jù)flush到TSDB塊并上傳到OSS中（如果配置了的話），因?yàn)檫@些節(jié)點(diǎn)之后將有一個(gè)新的分配。之前已存在節(jié)點(diǎn)上的時(shí)間序列不需要作調(diào)整，只是后面過來的請求按新的分發(fā)來尋找該去的receiver節(jié)點(diǎn)。注意，這種情況發(fā)生的flush可能會產(chǎn)生較小的TSDB塊，但thanos compactor模塊可以將它們優(yōu)化合并，因此不會有什么問題。

當(dāng)有receiver節(jié)點(diǎn)發(fā)生故障時(shí)，prometheus的遠(yuǎn)程寫會在后端目標(biāo)無響應(yīng)或503時(shí)進(jìn)行重試，因此，receiver一定時(shí)間的服務(wù)掛掉是可以容忍的。如果這種掛機(jī)時(shí)間是不可接受的話，可以將副本數(shù)配置為3或以上，這樣即使有一個(gè)receiver節(jié)點(diǎn)掛掉，還有其他receiver節(jié)點(diǎn)來接收寫請求。

原文鏈接：https://blog.csdn.net/felix_yujing/article/details/108302167