Loki 日志系統(tǒng)詳解

最近，在對公司容器云的日志方案進(jìn)行設(shè)計的時候，發(fā)現(xiàn)主流的ELK或者EFK比較重，再加上現(xiàn)階段對于ES復(fù)雜的搜索功能很多都用不上最終選擇了Grafana開源的Loki日志系統(tǒng)，下面介紹下Loki的背景。

-     緣起    -

當(dāng)我們的容器云運行的應(yīng)用或者某個節(jié)點出現(xiàn)問題了，解決思路應(yīng)該如下：

我們的監(jiān)控使用的是基于Prometheus體系進(jìn)行改造的，Prometheus中比較重要的是Metric和Alert，Metric是來說明當(dāng)前或者歷史達(dá)到了某個值，Alert設(shè)置Metric達(dá)到某個特定的基數(shù)觸發(fā)了告警，但是這些信息明顯是不夠的。

我們都知道，Kubernetes的基本單位是Pod，Pod把日志輸出到stdout和stderr，平時有什么問題我們通常在界面或者通過命令查看相關(guān)的日志，舉個例子：當(dāng)我們的某個Pod的內(nèi)存變得很大，觸發(fā)了我們的Alert，這個時候管理員，去頁面查詢確認(rèn)是哪個Pod有問題，然后要確認(rèn)Pod內(nèi)存變大的原因，我們還需要去查詢Pod的日志，如果沒有日志系統(tǒng)，那么我們就需要到頁面或者使用命令進(jìn)行查詢了：

如果，這個時候應(yīng)用突然掛了，這個時候我們就無法查到相關(guān)的日志了，所以需要引入日志系統(tǒng)，統(tǒng)一收集日志，而使用ELK的話，就需要在Kibana和Grafana之間切換，影響用戶體驗。所以 ，loki的第一目的就是最小化度量和日志的切換成本，有助于減少異常事件的響應(yīng)時間和提高用戶的體驗。

-     ELK 存在的問題    -

現(xiàn)有的很多日志采集的方案都是采用全文檢索對日志進(jìn)行索引（如ELK方案），優(yōu)點是功能豐富，允許復(fù)雜的操作。但是，這些方案往往規(guī)模復(fù)雜，資源占用高，操作苦難。很多功能往往用不上，大多數(shù)查詢只關(guān)注一定時間范圍和一些簡單的參數(shù)（如host、service等），使用這些解決方案就有點殺雞用牛刀的感覺了。

因此，Loki的第二個目的是，在查詢語言的易操作性和復(fù)雜性之間可以達(dá)到一個權(quán)衡。

-     成本    -

全文檢索的方案也帶來成本問題，簡單的說就是全文搜索（如ES）的倒排索引的切分和共享的成本較高。后來出現(xiàn)了其他不同的設(shè)計方案如：OKlog，采用最終一致的、基于網(wǎng)格的分布策略。這兩個設(shè)計決策提供了大量的成本降低和非常簡單的操作，但是查詢不夠方便。因此，Loki的第三個目的是，提高一個更具成本效益的解決方案。

-     架構(gòu)    -

整體架構(gòu)

Loki的架構(gòu)如下：

不難看出，Loki的架構(gòu)非常簡單，使用了和Prometheus一樣的標(biāo)簽來作為索引，也就是說，你通過這些標(biāo)簽既可以查詢?nèi)罩镜膬?nèi)容也可以查詢到監(jiān)控的數(shù)據(jù)，不但減少了兩種查詢之間的切換成本，也極大地降低了日志索引的存儲。

Loki將使用與Prometheus相同的服務(wù)發(fā)現(xiàn)和標(biāo)簽重新標(biāo)記庫，編寫了pormtail，在Kubernetes中promtail以DaemonSet方式運行在每個節(jié)點中，通過Kubernetes API等到日志的正確元數(shù)據(jù)，并將它們發(fā)送到Loki。下面是日志的存儲架構(gòu)：

-     讀寫    -

日志數(shù)據(jù)的寫主要依托的是Distributor和Ingester兩個組件，整體的流程如下：

-     Distributor    -

一旦promtail收集日志并將其發(fā)送給loki，Distributor就是第一個接收日志的組件。由于日志的寫入量可能很大，所以不能在它們傳入時將它們寫入數(shù)據(jù)庫。這會毀掉數(shù)據(jù)庫。我們需要批處理和壓縮數(shù)據(jù)。

Loki通過構(gòu)建壓縮數(shù)據(jù)塊來實現(xiàn)這一點，方法是在日志進(jìn)入時對其進(jìn)行g(shù)zip操作，組件ingester是一個有狀態(tài)的組件，負(fù)責(zé)構(gòu)建和刷新chunck，當(dāng)chunk達(dá)到一定的數(shù)量或者時間后，刷新到存儲中去。每個流的日志對應(yīng)一個ingester，當(dāng)日志到達(dá)Distributor后，根據(jù)元數(shù)據(jù)和hash算法計算出應(yīng)該到哪個ingester上面。

此外，為了冗余和彈性，我們將其復(fù)制n（默認(rèn)情況下為3）次。

-     Ingester    -

Ingester接收到日志并開始構(gòu)建chunk：

基本上就是將日志進(jìn)行壓縮并附加到chunk上面。一旦chunk“填滿”（數(shù)據(jù)達(dá)到一定數(shù)量或者過了一定期限），ingester將其刷新到數(shù)據(jù)庫。我們對塊和索引使用單獨的數(shù)據(jù)庫，因為它們存儲的數(shù)據(jù)類型不同。

刷新一個chunk之后，ingester然后創(chuàng)建一個新的空chunk并將新條目添加到該chunk中。

-     Querier    -

讀取就非常簡單了，由Querier負(fù)責(zé)給定一個時間范圍和標(biāo)簽選擇器，Querier查看索引以確定哪些塊匹配，并通過greps將結(jié)果顯示出來。它還從Ingester獲取尚未刷新的最新數(shù)據(jù)。

對于每個查詢，一個查詢器將為您顯示所有相關(guān)日志。實現(xiàn)了查詢并行化，提供分布式grep，使即使是大型查詢也是足夠的。

-     可擴(kuò)展性    -

Loki的索引存儲可以是cassandra/bigtable/dynamodb，而chuncks可以是各種對象存儲，Querier和Distributor都是無狀態(tài)的組件。對于ingester他雖然是有狀態(tài)的但是，當(dāng)新的節(jié)點加入或者減少，整節(jié)點間的chunk會重新分配，已適應(yīng)新的散列環(huán)。而Loki底層存儲的實現(xiàn)Cortex已經(jīng) 在實際的生產(chǎn)中投入使用多年了。有了這句話，我可以放心的在環(huán)境中實驗一把了。

-     部署    -

oc new-project loki

oc adm policy add-scc-to-user anyuid -z default -n loki
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:loki:default

oc create -f statefulset.json -n loki

{
    "apiVersion": "apps/v1",
    "kind": "StatefulSet",
    "metadata": {
        "name": "loki"
    },
    "spec": {
        "podManagementPolicy": "OrderedReady",
        "replicas": 1,
        "revisionHistoryLimit": 10,
        "selector": {
            "matchLabels": {
                "app": "loki"
            }
        },
        "serviceName": "womping-stoat-loki-headless",
        "template": {
            "metadata": {
                "annotations": {
                    "checksum/config": "da297d66ee53e0ce68b58e12be7ec5df4a91538c0b476cfe0ed79666343df72b",
                    "prometheus.io/port": "http-metrics",
                    "prometheus.io/scrape": "true"
                },
                "creationTimestamp": null,
                "labels": {
                    "app": "loki",
                    "name": "loki"
                }
            },
            "spec": {
                "affinity": {},
                "containers": [
                    {
                        "args": [
                            "-config.file=/etc/loki/local-config.yaml"
                        ],
                        "image": "grafana/loki:latest",
                        "imagePullPolicy": "IfNotPresent",
                        "livenessProbe": {
                            "failureThreshold": 3,
                            "httpGet": {
                                "path": "/ready",
                                "port": "http-metrics",
                                "scheme": "HTTP"
                            },
                            "initialDelaySeconds": 45,
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "name": "loki",
                        "ports": [
                            {
                                "containerPort": 3100,
                                "name": "http-metrics",
                                "protocol": "TCP"
                            }
                        ],
                        "readinessProbe": {
                            "failureThreshold": 3,
                            "httpGet": {
                                "path": "/ready",
                                "port": "http-metrics",
                                "scheme": "HTTP"
                            },
                            "initialDelaySeconds": 45,
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "resources": {},
                        "terminationMessagePath": "/dev/termination-log",
                        "terminationMessagePolicy": "File",
                        "volumeMounts": [
                            {
                                "mountPath": "/tmp/loki",
                                "name": "storage"
                            }
                        ]
                    }
                ],
                "dnsPolicy": "ClusterFirst",
                "restartPolicy": "Always",
                "schedulerName": "default-scheduler",
                "terminationGracePeriodSeconds": 30,
                "volumes": [
                    {
                        "emptyDir": {},
                        "name": "storage"
                    }
                ]
            }
        },
        "updateStrategy": {
            "type": "RollingUpdate"
        }
    }
}

oc create -f configmap.json -n loki

{
    "apiVersion": "v1",
    "data": {
        "promtail.yaml": "client:\n  backoff_config:\n    maxbackoff: 5s\n    maxretries: 5\n    minbackoff: 100ms\n  batchsize: 102400\n  batchwait: 1s\n  external_labels: {}\n  timeout: 10s\npositions:\n  filename: /run/promtail/positions.yaml\nserver:\n  http_listen_port: 3101\ntarget_config:\n  sync_period: 10s\n\nscrape_configs:\n- job_name: kubernetes-pods-name\n  pipeline_stages:\n    - docker: {}\n    \n  kubernetes_sd_configs:\n  - role: pod\n  relabel_configs:\n  - source_labels:\n    - __meta_kubernetes_pod_label_name\n    target_label: __service__\n  - source_labels:\n    - __meta_kubernetes_pod_node_name\n    target_label: __host__\n  - action: drop\n    regex: ^$\n    source_labels:\n    - __service__\n  - action: labelmap\n    regex: __meta_kubernetes_pod_label_(.+)\n  - action: replace\n    replacement: $1\n    separator: /\n    source_labels:\n    - __meta_kubernetes_namespace\n    - __service__\n    target_label: job\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_namespace\n    target_label: namespace\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_name\n    target_label: instance\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_container_name\n    target_label: container_name\n  - replacement: /var/log/pods/*$1/*.log\n    separator: /\n    source_labels:\n    - __meta_kubernetes_pod_uid\n    - __meta_kubernetes_pod_container_name\n    target_label: __path__\n- job_name: kubernetes-pods-app\n  pipeline_stages:\n    - docker: {}\n    \n  kubernetes_sd_configs:\n  - role: pod\n  relabel_configs:\n  - action: drop\n    regex: .+\n    source_labels:\n    - __meta_kubernetes_pod_label_name\n  - source_labels:\n    - __meta_kubernetes_pod_label_app\n    target_label: __service__\n  - source_labels:\n    - __meta_kubernetes_pod_node_name\n    target_label: __host__\n  - action: drop\n    regex: ^$\n    source_labels:\n    - __service__\n  - action: labelmap\n    regex: __meta_kubernetes_pod_label_(.+)\n  - action: replace\n    replacement: $1\n    separator: /\n    source_labels:\n    - __meta_kubernetes_namespace\n    - __service__\n    target_label: job\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_namespace\n    target_label: namespace\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_name\n    target_label: instance\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_container_name\n    target_label: container_name\n  - replacement: /var/log/pods/*$1/*.log\n    separator: /\n    source_labels:\n    - __meta_kubernetes_pod_uid\n    - __meta_kubernetes_pod_container_name\n    target_label: __path__\n- job_name: kubernetes-pods-direct-controllers\n  pipeline_stages:\n    - docker: {}\n    \n  kubernetes_sd_configs:\n  - role: pod\n  relabel_configs:\n  - action: drop\n    regex: .+\n    separator: ''\n    source_labels:\n    - __meta_kubernetes_pod_label_name\n    - __meta_kubernetes_pod_label_app\n  - action: drop\n    regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$\n    source_labels:\n    - __meta_kubernetes_pod_controller_name\n  - source_labels:\n    - __meta_kubernetes_pod_controller_name\n    target_label: __service__\n  - source_labels:\n    - __meta_kubernetes_pod_node_name\n    target_label: __host__\n  - action: drop\n    regex: ^$\n    source_labels:\n    - __service__\n  - action: labelmap\n    regex: __meta_kubernetes_pod_label_(.+)\n  - action: replace\n    replacement: $1\n    separator: /\n    source_labels:\n    - __meta_kubernetes_namespace\n    - __service__\n    target_label: job\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_namespace\n    target_label: namespace\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_name\n    target_label: instance\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_container_name\n    target_label: container_name\n  - replacement: /var/log/pods/*$1/*.log\n    separator: /\n    source_labels:\n    - __meta_kubernetes_pod_uid\n    - __meta_kubernetes_pod_container_name\n    target_label: __path__\n- job_name: kubernetes-pods-indirect-controller\n  pipeline_stages:\n    - docker: {}\n    \n  kubernetes_sd_configs:\n  - role: pod\n  relabel_configs:\n  - action: drop\n    regex: .+\n    separator: ''\n    source_labels:\n    - __meta_kubernetes_pod_label_name\n    - __meta_kubernetes_pod_label_app\n  - action: keep\n    regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$\n    source_labels:\n    - __meta_kubernetes_pod_controller_name\n  - action: replace\n    regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$\n    source_labels:\n    - __meta_kubernetes_pod_controller_name\n    target_label: __service__\n  - source_labels:\n    - __meta_kubernetes_pod_node_name\n    target_label: __host__\n  - action: drop\n    regex: ^$\n    source_labels:\n    - __service__\n  - action: labelmap\n    regex: __meta_kubernetes_pod_label_(.+)\n  - action: replace\n    replacement: $1\n    separator: /\n    source_labels:\n    - __meta_kubernetes_namespace\n    - __service__\n    target_label: job\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_namespace\n    target_label: namespace\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_name\n    target_label: instance\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_container_name\n    target_label: container_name\n  - replacement: /var/log/pods/*$1/*.log\n    separator: /\n    source_labels:\n    - __meta_kubernetes_pod_uid\n    - __meta_kubernetes_pod_container_name\n    target_label: __path__\n- job_name: kubernetes-pods-static\n  pipeline_stages:\n    - docker: {}\n    \n  kubernetes_sd_configs:\n  - role: pod\n  relabel_configs:\n  - action: drop\n    regex: ^$\n    source_labels:\n    - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_label_component\n    target_label: __service__\n  - source_labels:\n    - __meta_kubernetes_pod_node_name\n    target_label: __host__\n  - action: drop\n    regex: ^$\n    source_labels:\n    - __service__\n  - action: labelmap\n    regex: __meta_kubernetes_pod_label_(.+)\n  - action: replace\n    replacement: $1\n    separator: /\n    source_labels:\n    - __meta_kubernetes_namespace\n    - __service__\n    target_label: job\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_namespace\n    target_label: namespace\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_name\n    target_label: instance\n  - action: replace\n    source_labels:\n    - __meta_kubernetes_pod_container_name\n    target_label: container_name\n  - replacement: /var/log/pods/*$1/*.log\n    separator: /\n    source_labels:\n    - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror\n    - __meta_kubernetes_pod_container_name\n    target_label: __path__\n"
    },
    "kind": "ConfigMap",
    "metadata": {
        "creationTimestamp": "2019-09-05T01:05:03Z",
        "labels": {
            "app": "promtail",
            "chart": "promtail-0.12.0",
            "heritage": "Tiller",
            "release": "lame-zorse"
        },
        "name": "lame-zorse-promtail",
        "namespace": "loki",
        "resourceVersion": "17921611",
        "selfLink": "/api/v1/namespaces/loki/configmaps/lame-zorse-promtail",
        "uid": "30fcb896-cf79-11e9-b58e-e4a8b6cc47d2"
    }
}




oc create -f daemonset.json -n loki

     {
            "apiVersion": "apps/v1",
            "kind": "DaemonSet",
            "metadata": {
                "annotations": {
                    "deployment.kubernetes.io/revision": "2"
                },
                "creationTimestamp": "2019-09-05T01:16:37Z",
                "generation": 2,
                "labels": {
                    "app": "promtail",
                    "chart": "promtail-0.12.0",
                    "heritage": "Tiller",
                    "release": "lame-zorse"
                },
                "name": "lame-zorse-promtail",
                "namespace": "loki"
            },
            "spec": {
                "progressDeadlineSeconds": 600,
                "replicas": 1,
                "revisionHistoryLimit": 10,
                "selector": {
                    "matchLabels": {
                        "app": "promtail",
                        "release": "lame-zorse"
                    }
                },
                "strategy": {
                    "rollingUpdate": {
                        "maxSurge": 1,
                        "maxUnavailable": 1
                    },
                    "type": "RollingUpdate"
                },
                "template": {
                    "metadata": {
                        "annotations": {
                            "checksum/config": "75a25ee4f2869f54d394bf879549a9c89c343981a648f8d878f69bad65dba809",
                            "prometheus.io/port": "http-metrics",
                            "prometheus.io/scrape": "true"
                        },
                        "creationTimestamp": null,
                        "labels": {
                            "app": "promtail",
                            "release": "lame-zorse"
                        }
                    },
                    "spec": {
                        "affinity": {},
                        "containers": [
                            {
                                "args": [
                                    "-config.file=/etc/promtail/promtail.yaml",
                                    "-client.url=http://loki.loki.svc:3100/api/prom/push"
                                ],
                                "env": [
                                    {
                                        "name": "HOSTNAME",
                                        "valueFrom": {
                                            "fieldRef": {
                                                "apiVersion": "v1",
                                                "fieldPath": "spec.nodeName"
                                            }
                                        }
                                    }
                                ],
                                "image": "grafana/promtail:v0.3.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "promtail",
                                "ports": [
                                    {
                                        "containerPort": 3101,
                                        "name": "http-metrics",
                                        "protocol": "TCP"
                                    }
                                ],
                                "readinessProbe": {
                                    "failureThreshold": 5,
                                    "httpGet": {
                                        "path": "/ready",
                                        "port": "http-metrics",
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 10,
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 1
                                },
                                "resources": {},
                                "securityContext": {
                                    "readOnlyRootFilesystem": true,
                                    "runAsUser": 0
                                },
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File",
                                "volumeMounts": [
                                    {
                                        "mountPath": "/etc/promtail",
                                        "name": "config"
                                    },
                                    {
                                        "mountPath": "/run/promtail",
                                        "name": "run"
                                    },
                                    {
                                        "mountPath": "/var/lib/docker/containers",
                                        "name": "docker",
                                        "readOnly": true
                                    },
                                    {
                                        "mountPath": "/var/log/pods",
                                        "name": "pods",
                                        "readOnly": true
                                    }
                                ]
                            }
                        ],
                        "dnsPolicy": "ClusterFirst",
                        "restartPolicy": "Always",
                        "schedulerName": "default-scheduler",
                        "securityContext": {},
                        "terminationGracePeriodSeconds": 30,
                        "volumes": [
                            {
                                "configMap": {
                                    "defaultMode": 420,
                                    "name": "lame-zorse-promtail"
                                },
                                "name": "config"
                            },
                            {
                                "hostPath": {
                                    "path": "/run/promtail",
                                    "type": ""
                                },
                                "name": "run"
                            },
                            {
                                "hostPath": {
                                    "path": "/var/lib/docker/containers",
                                    "type": ""
                                },
                                "name": "docker"
                            },
                            {
                                "hostPath": {
                                    "path": "/var/log/pods",
                                    "type": ""
                                },
                                "name": "pods"
                            }
                        ]
                    }
                }
            }
        }

安裝服務(wù)

oc create -f service.json -n loki

{
    "apiVersion": "v1",
    "kind": "Service",
    "metadata": {
        "creationTimestamp": "2019-09-04T09:37:49Z",
        "name": "loki",
        "namespace": "loki",
        "resourceVersion": "17800188",
        "selfLink": "/api/v1/namespaces/loki/services/loki",
        "uid": "a87fe237-cef7-11e9-b58e-e4a8b6cc47d2"
    },
    "spec": {
        "externalTrafficPolicy": "Cluster",
        "ports": [
            {
                "name": "lokiport",
                "port": 3100,
                "protocol": "TCP",
                "targetPort": 3100
            }
        ],
        "selector": {
            "app": "loki"
        },
        "sessionAffinity": "None",
        "type": "NodePort"
    },
    "status": {
        "loadBalancer": {}
    }

curl http://192.168.25.30:30972/api/prom/label
{
 "values": ["alertmanager", "app", "component", "container_name", "controller_revision_hash", "deployment", "deploymentconfig", "docker_registry", "draft", "filename", "instance", "job", "logging_infra", "metrics_infra", "name", "namespace", "openshift_io_component", "pod_template_generation", "pod_template_hash", "project", "projectname", "prometheus", "provider", "release", "router", "servicename", "statefulset_kubernetes_io_pod_name", "stream", "tekton_dev_pipeline", "tekton_dev_pipelineRun", "tekton_dev_pipelineTask", "tekton_dev_task", "tekton_dev_taskRun", "type", "webconsole"]
}

curl http://192.168.25.30:30972/api/prom/label/namespace/values
{"values":["cicd","default","gitlab","grafanaserver","jenkins","jx-staging","kube-system","loki","mysql-exporter","new2","openshift-console","openshift-infra","openshift-logging","openshift-monitoring","openshift-node","openshift-sdn","openshift-web-console","tekton-pipelines","test111"]}

http://192.168.25.30:30972/api/prom/query?direction=BACKWARD&limit=1000&regexp=&query={namespace="cicd"}&start=1567644457221000000&end=1567730857221000000&refId=A

{app="mysql",name="mysql-backup"}