使用 Loki 收集 Traefik 日志
前面我們介紹了 Loki 的實(shí)現(xiàn)架構(gòu)以及 Promtail 的相關(guān)配置，本文我們將來介紹如何安裝 Loki，并為 Traefik 的日志設(shè)置一個可視化的 Dashboard。
首先添加 Loki 的 Chart 倉庫：
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
獲取 loki-stack 的 Chart 包并解壓：
helm pull grafana/loki-stack --untar --version 2.3.1
loki-stack 這個 Chart 包里面包含所有的 Loki 相關(guān)工具依賴，在安裝的時候可以根據(jù)需要開啟或關(guān)閉，比如我們想要安裝 Grafana，則可以 在安裝的時候簡單設(shè)置 --set grafana.enabled=true 即可。默認(rèn)情況下 loki、promtail 是自動開啟的，也可以根據(jù)我們的需要選擇使用 filebeat 或者 logstash，同樣在 Chart 包根目錄下面創(chuàng)建用于安裝的 Values 文件：
# values-prod.yaml
loki:
enabled: true
replicas: 1
persistence:
enabled: true
storageClassName: nfs-storage
promtail:
enabled: true
grafana:
enabled: true
service:
type: NodePort
persistence:
enabled: true
storageClassName: nfs-storage
accessModes:
- ReadWriteOnce
size: 1Gi
然后直接使用上面的 Values 文件進(jìn)行安裝即可：
helm upgrade --install loki -n logging -f values-prod.yaml .
Release "loki" does not exist. Installing it now.
NAME: loki
LAST DEPLOYED: Sat May 8 11:58:50 2021
NAMESPACE: logging
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
安裝完成后可以查看 Pod 的狀態(tài)：
$ kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
loki-0 1/1 Running 0 153m
loki-grafana-86f4f9cbcc-kls6j 1/1 Running 0 153m
loki-promtail-69w7b 1/1 Running 0 153m
loki-promtail-mzk77 1/1 Running 0 150m
loki-promtail-pnn97 1/1 Running 0 151m
這里我們?yōu)?Grafana 設(shè)置的 NodePort 類型的 Service：
$ kubectl get svc -n logging
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loki ClusterIP 10.105.185.97 <none> 3100/TCP 156m
loki-grafana NodePort 10.102.226.255 <none> 80:30029/TCP 156m
loki-headless ClusterIP None <none> 3100/TCP 156m
可以通過 NodePort 端口 30029 訪問 Grafana，使用下面的命令獲取 Grafana 的登錄密碼：
kubectl get secret --namespace logging loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
使用用戶名 admin 和上面的獲取的密碼即可登錄 Grafana，由于 Helm Chart 已經(jīng)為 Grafana 配置好了 Loki 的數(shù)據(jù)源，所以我們可以直接獲取到日志數(shù)據(jù)了。點(diǎn)擊左側(cè) Explore 菜單，然后就可以篩選 Loki 的日志數(shù)據(jù)了：
我們使用 Helm 安裝的 Promtail 默認(rèn)已經(jīng)幫我們做好了配置，已經(jīng)針對 Kubernetes 做了優(yōu)化，我們可以查看其配置：
$ kubectl get cm loki-promtail -n logging -o yaml
apiVersion: v1
data:
promtail.yaml: |
client:
backoff_config:
max_period: 5m
max_retries: 10
min_period: 500ms
batchsize: 1048576
batchwait: 1s
external_labels: {}
timeout: 10s
positions:
filename: /run/promtail/positions.yaml
server:
http_listen_port: 3101
target_config:
sync_period: 10s
scrape_configs:
- job_name: kubernetes-pods-name
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-app
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
source_labels:
- __meta_kubernetes_pod_label_name
- source_labels:
- __meta_kubernetes_pod_label_app
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-direct-controllers
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_name
- __meta_kubernetes_pod_label_app
- action: drop
regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
- source_labels:
- __meta_kubernetes_pod_controller_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-indirect-controller
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_name
- __meta_kubernetes_pod_label_app
- action: keep
regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
- action: replace
regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-static
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: ''
source_labels:
- __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
- action: replace
source_labels:
- __meta_kubernetes_pod_label_component
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
- __meta_kubernetes_pod_container_name
target_label: __path__
......
這里我們以收集 Traefik 為例，為 Traefik 定制一個可視化的 Dashboard，默認(rèn)情況下訪問日志沒有輸出到 stdout，我們可以通過在命令行參數(shù)中設(shè)置 --accesslog=true 來開啟，此外我們還可以設(shè)置訪問日志格式為 json，這樣更方便在 Loki 中查詢使用：
containers:
- args:
- --accesslog=true
- --accesslog.format=json
......
默認(rèn) traefik 的日志輸出為 stdout，如果你的采集端是通過讀取文件的話，則需要用 filePath 參數(shù)將 traefik 的日志重定向到文件目錄。
修改完成后正常在 Grafana 中就可以看到 Traefik 的訪問日志了：
然后我們還可以導(dǎo)入 Dashboard 來展示 Traefik 的信息：https://grafana.com/grafana/dashboards/13713，在 Grafana 中導(dǎo)入 13713 號 Dashboard：
不過要注意我們需要更改 Dashboard 里面圖表的查詢語句，將 job 的值更改為你實(shí)際的標(biāo)簽，比如我這里采集 Traefik 日志的最終標(biāo)簽為 job="kube-system/traefik"：
此外該 Dashboard 上還出現(xiàn)了 Panel plugin not found: grafana-piechart-panel 這樣的提示，這是因?yàn)樵撁姘逡蕾?grafana-piechart-panel 這個插件，我們進(jìn)入 Grafana 容器內(nèi)安裝重建 Pod 即可：
$ kubectl exec -it loki-grafana-864fc6999c-z9587 -n logging -- /bin/bash
bash-5.0$ grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.6.1
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download
into: /var/lib/grafana/plugins
? Installed grafana-piechart-panel successfully
Restart grafana after installing plugins . <service grafana-server restart>
由于上面我們安裝的時候?yàn)?Grafana 持久化了數(shù)據(jù)，所以刪掉 Pod 重建即可：
kubectl delete pod loki-grafana-864fc6999c-z9587 -n logging
pod "loki-grafana-864fc6999c-z9587" deleted
最后調(diào)整過后的 Traefik Dashboard 大盤效果如下所示：
評論
圖片
表情