Vladimir日志采集工具

Vladimir 采集系統(tǒng)

日志采集工具，支持正則 熱加載日志文件，發(fā)送狀態(tài)等信息都會實時上報 ETCD

TODO

• 將日志文件一行一行讀取出來發(fā)送到自定義的 Sink | 100%
• 支持自動重試（fib）
• 心跳檢測發(fā)送到自定義的 Topic[health] | 100%
• 日志啟動點(創(chuàng)建 CheckPoint) | 100%
• Offset 同步到 Etcd | 100%
• Web 監(jiān)控心跳以及創(chuàng)建新的采集任務(wù) | 100%
• 熱啟動插件 | 10%

Logs - 采集工具

./Logs_Pulse  install   -- 安裝服務(wù)
./Logs_Pulse  start     -- 啟動服務(wù)
./Logs_Pulse  stop      -- 停止服務(wù)
./Logs_Pulse  restart   -- 重啟服務(wù)

Owner

• 郵箱echo emhhbmdqaWFueGlubmV0QGdtYWlsLmNvbQo= | base64 --decode

命令行客戶端使用教程

(base) ?  cli ./cli         
NAME:
   pulse client - a tool for driving pulse cli

USAGE:
   cli [global options] command [command options] [arguments...]

VERSION:
   v1 darwin/amd64 go1.11.4

AUTHOR:
   Zhangjianxin <[email protected]>

COMMANDS:
     list      pulse on etcd config
     services  pulse on services
     task      addtask on services
     clear     delete root path / data
     host      host list
     help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version



(base) ?  cli ./cli host
|---------------|
| Host          |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|


(base) ?  cli ./cli list
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| Topic | Log_path                                                                          | Service             | Send_rate | IP            | Type  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | sentry-test-service | 1000      | 192.168.2.150 | kafka |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | sentry-test-service | 1000      | 192.168.2.150 | http  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | sentry-test-service | 1000      | 192.168.2.150 | http  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|


(base) ?  cli ./cli services
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Offset | FilePatch                                                                         | Time                | Topic | TPS | QPS | IP            |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 39     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | 2019-04-27 03:38:32 | test  | 0   | 0   | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 63     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | 2019-04-27 03:45:17 | test  | 0   | 0   | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 87     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | 2019-04-27 03:49:32 | test  | 0   | 0   | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|




(base) ?  cli 
    ./cli task  -host 192.168.43.223  -t buried-point-mall -f /var/log/system.log -s buried-point-mall-service-test -p http
    ./cli task   -re "" -host 192.168.2.150  -t buried-point-mall -f /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/ -s buried-point-mall-service-test -p http
    
    ./cli task   -re "" -host 172.17.193.23  -t sentry-test -f /var/log/messages -s service-test -p http
    
    ./cli task  -re "" -host  172.19.184.193 -t sentry-test -f /home/soft/Octopoda/release/logs/pulse.log -s system-pulse -p http
    ./cli task  -re "" -host  172.19.184.193 -t buried-point-mall -f /data/sa/ -s sa-pulse -p http
    
    
    ./cli task  -re "" -host 172.17.193.23  -t sentry-test -f /home/Octopoda/release/logs/ -s service-test2 -p http
    ./cli task  -re "" -host 192.168.131.249  -t buried-point-sma-r2p10 -f  /home/admin/app/logs/ -s service-test4 -p http
    ./cli task  -re "" -host 192.168.131.249  -t buried-point-sma-r2p10 -f  /home/admin/app/logs/behaviorSave.log -s service-test4 -p http
    
    
    
    ./cli task  -re "" -host 192.168.131.249  -t sentry-test -f  /home/Octopoda/release/logs/pulse.log -s system-pulse -p http

提交作業(yè)

./cli task 
         -host 192.168.2.150  #采集某個節(jié)點
         -t test  #發(fā)送到topic
         -f /var/log/system.log # 采集日志
         -s sentry-test-service # 服務(wù)名字定義
         -p http #發(fā)送方式


# create topic
kafka-topics --create --zookeeper cdh-m1.sxkj.online:2181 --replication-factor 1 --partitions 1 --topic kdc_pdp

kerberos KDC Proxy

https://medium.com/@Santii/how-to-setup-kerberos-behind-a-reverse-proxy-nginx-d692f609191c