Go：給expvarmon插上數(shù)據(jù)持久化的“翅膀”
Go在標準庫中為暴露Go應用內(nèi)部指標數(shù)據(jù)提供了標準的對外接口，這就是expvar包[1]。expvar包通過init函數(shù)將內(nèi)置的expvarHandler(一個標準http HandlerFunc)注冊到http包ListenAndServe創(chuàng)建的默認Server上。
// $GOROOT/src/expvar/expvar.go
func init() {
http.HandleFunc("/debug/vars", expvarHandler)
Publish("cmdline", Func(cmdline))
Publish("memstats", Func(memstats))
}
這樣如果一個Go應用要想利用expvar默認暴露的內(nèi)部指標數(shù)據(jù)，僅需做到兩點：
以副作用方式導入expvar包
import _ "expvar"
啟動默認HTTP Server
http.ListenAndServe("localhost:8080", nil)
我們來建立的使用expvar包暴露指標的最簡單的例子：
// expvar_demo1.go
package main
import (
_ "expvar"
"net/http"
)
func main() {
http.ListenAndServe(":8080", nil)
}
這樣expvar包的expvarHandler會自動響應到localhost:8080/debug/vars上的http請求：
$go build expvar_demo1.go
$./expvar_demo1 -w=1 -r=2
$curl localhost:8080/debug/vars
{
"cmdline": ["./expvar_demo1","-w=1","-r=2"],
"memstats": {"Alloc":227088,"TotalAlloc":227088,"Sys":71650320,"Lookups":0,"Mallocs":730,"Frees":13,"HeapAlloc":227088,"HeapSys":66715648,"HeapIdle":65937408,"HeapInuse":778240,"HeapReleased":65937408,"HeapObjects":717,"StackInuse":393216,"StackSys":393216,"MSpanInuse":37536,"MSpanSys":49152,"MCacheInuse":9600,"MCacheSys":16384,"BuckHashSys":3769,"GCSys":3783272,"OtherSys":688879,"NextGC":4473924,"LastGC":0,"PauseTotalNs":0,"PauseNs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":0,"NumForcedGC":0,"GCCPUFraction":0,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":14,"Frees":0},{"Size":16,"Mallocs":297,"Frees":0},{"Size":24,"Mallocs":32,"Frees":0},{"Size":32,"Mallocs":20,"Frees":0},{"Size":48,"Mallocs":105,"Frees":0},{"Size":64,"Mallocs":31,"Frees":0},{"Size":80,"Mallocs":9,"Frees":0},{"Size":96,"Mallocs":13,"Frees":0},{"Size":112,"Mallocs":2,"Frees":0},{"Size":128,"Mallocs":7,"Frees":0},{"Size":144,"Mallocs":3,"Frees":0},{"Size":160,"Mallocs":16,"Frees":0},{"Size":176,"Mallocs":5,"Frees":0},{"Size":192,"Mallocs":0,"Frees":0},{"Size":208,"Mallocs":33,"Frees":0},{"Size":224,"Mallocs":3,"Frees":0},{"Size":240,"Mallocs":0,"Frees":0},{"Size":256,"Mallocs":10,"Frees":0},{"Size":288,"Mallocs":8,"Frees":0},{"Size":320,"Mallocs":2,"Frees":0},{"Size":352,"Mallocs":10,"Frees":0},{"Size":384,"Mallocs":24,"Frees":0},{"Size":416,"Mallocs":7,"Frees":0},{"Size":448,"Mallocs":0,"Frees":0},{"Size":480,"Mallocs":1,"Frees":0},{"Size":512,"Mallocs":0,"Frees":0},{"Size":576,"Mallocs":3,"Frees":0},{"Size":640,"Mallocs":3,"Frees":0},{"Size":704,"Mallocs":5,"Frees":0},{"Size":768,"Mallocs":0,"Frees":0},{"Size":896,"Mallocs":7,"Frees":0},{"Size":1024,"Mallocs":7,"Frees":0},{"Size":1152,"Mallocs":10,"Frees":0},{"Size":1280,"Mallocs":4,"Frees":0},{"Size":1408,"Mallocs":1,"Frees":0},{"Size":1536,"Mallocs":0,"Frees":0},{"Size":1792,"Mallocs":5,"Frees":0},{"Size":2048,"Mallocs":1,"Frees":0},{"Size":2304,"Mallocs":2,"Frees":0},{"Size":2688,"Mallocs":2,"Frees":0},{"Size":3072,"Mallocs":0,"Frees":0},{"Size":3200,"Mallocs":0,"Frees":0},{"Size":3456,"Mallocs":0,"Frees":0},{"Size":4096,"Mallocs":4,"Frees":0},{"Size":4864,"Mallocs":0,"Frees":0},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":1,"Frees":0},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":0,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":1,"Frees":0},{"Size":9472,"Mallocs":0,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":8,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0}]}
}
如果我們不使用http.ListenAndServe建立的默認Server呢？expvar包也提供了相應的方法幫助你在自定義http server以及自定義請求路徑上使用expvarHandler，我們看看下面示例：
// expvar_demo2.go
package main
import (
"expvar"
"net/http"
)
func main() {
mux := http.NewServeMux()
mux.Handle("/mydebug/myvars", expvar.Handler())
var server = &http.Server{
Addr: "localhost:8081",
Handler: mux,
}
server.ListenAndServe()
}
在這個示例中，我們利用http.ServeMux建立了expvarHandler響應的自定義路徑(/mydebug/myvars)，并自定義了一個http.Server，這樣當expvar_demo2運行起來后，我們就可以在localhost:8081/mydebug/myvars上獲取該應用暴露的指標數(shù)據(jù)了。
通過expvar_demo1的輸出結果，我們看到expvar默認將命令行字段和runtime包的MemStats結構[2]暴露給外部。我們也可以自定義要暴露到外部的數(shù)據(jù)，expvar包提供了常用指標類型的便捷接口以幫助我們更容易的自定義要暴露到外部的數(shù)據(jù)，看下面示例：
// expvar_demo3.go
package main
import (
"expvar"
_ "expvar"
"net/http"
)
var (
total *expvar.Int
)
func init() {
total = expvar.NewInt("TotalRequest")
}
func main() {
http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
total.Add(1)
w.Write([]byte("hello, go\n"))
}))
http.ListenAndServe(":8080", nil)
}
在這個示例中，我們自定義了一個公開指標TotalRequest，用于描述該Server總共處理了多少個請求。我們用*expvar.Int作為TotalRequest的類型，expvar.Int類型提供了并發(fā)安全的Add方法，利用該方法我們可以對指標做運算。運行上面示例后，我們就可以獲取TotalRequest這個指標了：
$curl localhost:8080/debug/vars
{
"TotalRequest": 2,
... ..
}
expvar包提供了對外的數(shù)據(jù)接口，但觀測方式卻是你決定的。圖形化的觀測方式是對人類友好的，一位名為divan[3]的gopher開發(fā)了expvarmon[4]工具，該工具可以在命令行終端以圖形化的方式實時展示特定的指標數(shù)據(jù)的變化，我們可以執(zhí)行如下命令實時查看應用指標變化；
$expvarmon -ports="http://localhost:8080/debug/vars" -i 1s
命令執(zhí)行的效果如下：
如果不指定指標，那么expvarmon默認展示上述圖中的memstats的幾個指標！
expvarmon支持實時獲取數(shù)據(jù)并展示數(shù)據(jù)的實時變動趨勢。但有些時候，我們不僅要看實時趨勢，可能還需要將數(shù)據(jù)存儲起來便于事后分析。但expvarmon不支持將數(shù)據(jù)序列化到磁盤并做歷史數(shù)據(jù)查看和分析。筆者曾提過issue[5]，但作者似乎認為這個項目完成度已經(jīng)很高了，該項目到2019年后就沒有更新了。因此自然也沒人理我的issue。我也只能自己動手豐衣足食了。
2. expvarmon的大致原理
要想基于expvarmon二次開發(fā)出支持數(shù)據(jù)持久化的版本，我們首先需要大致弄清楚expvarmon的工作原理，這里我將其工作原理大致總結為下面這幅示意圖：
expvarmon執(zhí)行時的兩個命令行標志參數(shù)很重要：-ports和-vars。前者決定了expvarmon啟動多少個Service(每個Service一個goroutine承載)：
// https://github.com/divan/expvarmon/blob/master/main.go
for _, port := range ports {
service := NewService(port, vars)
data.Services = append(data.Services, service)
}
后者用于指定expvarmon要實時顯示的數(shù)據(jù)項：
// https://github.com/divan/expvarmon/blob/master/service.go
// NewService returns new Service object.
func NewService(url url.URL, vars []VarName) *Service {
values := make(map[VarName]*Stack)
for _, name := range vars { //根據(jù)vars建立存儲對應var數(shù)據(jù)的Stack
values[VarName(name)] = NewStack()
}
... ...
}
expvar定時采集各個目標app的指標數(shù)據(jù)
// https://github.com/divan/expvarmon/blob/master/service.go
func main() {
... ...
UpdateAll(ui, data)
for {
select {
case <-tick.C:
UpdateAll(ui, data)
case e := <-termui.PollEvents():
if e.Type == termui.KeyboardEvent && e.ID == "q" {
return
}
if e.Type == termui.ResizeEvent {
ui.Update(*data)
}
}
}
}
// UpdateAll collects data from expvars and refreshes UI.
func UpdateAll(ui UI, data *UIData) {
var wg sync.WaitGroup
for _, service := range data.Services {
wg.Add(1)
go service.Update(&wg) // 每個服務單獨獲取對應port的數(shù)據(jù)
}
wg.Wait()
data.LastTimestamp = time.Now()
ui.Update(*data) // 更新并刷新命令行終端ui
}
3. 持久化到csv文件中
大致了解expvarmon的運作原理后，我們就來設計和實現(xiàn)將expvarmon啟動后針對每個port得到的指標數(shù)據(jù)存儲到磁盤文件中留待后續(xù)分析之用，這里選擇持久化到csv文件中，csv文件不僅便于直接打開并肉眼查看，也便于后續(xù)轉(zhuǎn)換為其他文件格式，比如：Microsoft的excel文件。
下面是對expvarmon的設計與實現(xiàn)改動點：
增加-w命令行標志參數(shù)(布爾型)，如果為true，則持久化獲取到的指標數(shù)據(jù)
// https://github.com/bigwhite/expvarmon/blob/master/main.go
var (
... ...
serialize = flag.Bool("w", false, "Serialize the data into a disk file")
)
在Service結構中增加持久化數(shù)據(jù)所需字段
// https://github.com/bigwhite/expvarmon/blob/master/service.go
// Service represents constantly updating info about single service.
type Service struct {
... ...
vars []VarName
// for serializing the data
// controlled by cmd option: serialize
f *os.File
w *csv.Writer // csv writer
}
vars用于存儲該Service對應的指標名；f為文件名；w是創(chuàng)建的csv.Writer結構。
在創(chuàng)建Service的時候，根據(jù)-w的值來決定是否創(chuàng)建持久化文件：
// https://github.com/bigwhite/expvarmon/blob/master/service.go
func NewService(url url.URL, vars []VarName) *Service {
... ...
if *serialize {
f, err := os.Create(s.Name + ".csv")
if err != nil {
panic(err)
}
s.f = f
s.w = csv.NewWriter(f)
// write first record: category line
record := []string{"time"}
for _, v := range vars {
record = append(record, string(v))
}
s.w.Write(record)
s.w.Flush()
}
... ...
}
我們看到：當-w為true時，NewService創(chuàng)建了持久化文件，并用Service的Name+.csv后綴為其命名。文件創(chuàng)建成功后，我們將寫入第一行csv數(shù)據(jù)，這一行數(shù)據(jù)為數(shù)據(jù)類別，就像下面這樣：
// 10.10.195.133:8080.csv
time,mem:memstats.Alloc,mem:memstats.Sys,mem:memstats.HeapAlloc,mem:memstats.HeapInuse,duration:memstats.PauseNs,duration:memstats.PauseTotalNs
除了第一列為時間(time)外，其余列都是以指標名命名的，如：mem:memstats.Alloc。
我們在Service的Update方法中定時寫入指標數(shù)據(jù)
// https://github.com/bigwhite/expvarmon/blob/master/service.go
// Update updates Service info from Expvar variable.
func (s *Service) Update(wg *sync.WaitGroup) {
... ...
if *serialize {
// serialize the values to csv
tm := time.Now().Format("2006-01-02 15:04:05")
values := []string{tm}
for _, name := range s.vars {
values = append(values, s.Value(name))
}
s.w.Write(values)
s.w.Flush()
}
}
增加Service的Close方法以優(yōu)雅關閉csv文件
和原expvarmon不同的是，我們二次開發(fā)的expvarmon在-w為true時會為每個Service創(chuàng)建一個磁盤文件，這樣我們就需要記著在適當?shù)臅r候優(yōu)雅的關閉這些csv格式的磁盤文件。
// https://github.com/bigwhite/expvarmon/blob/master/service.go
// Close does some cleanup before service exit
func (s *Service) Close() {
if *serialize {
if s.f != nil {
s.f.Close()
}
}
}
我們在程序退出前通過defer來調(diào)用Service的關閉方法：
// https://github.com/bigwhite/expvarmon/blob/master/main.go
func main() {
... ...
// Init UIData
data := NewUIData(vars)
for _, port := range ports {
service := NewService(port, vars)
data.Services = append(data.Services, service)
}
defer func() {
// close service before program exit
for _, service := range data.Services {
service.Close()
}
}()
... ...
}
按照上述這幾點改造后，我們再執(zhí)行如下命令：
$expvarmon -ports="http://10.10.195.133:8080/debug/vars" -i 1s -w=true
我們將得到10.10.195.133:8080.csv文件(如果-ports由多個值組成，那么將生成多個.csv文件)，內(nèi)容如下：
$cat 10.10.195.133:8080.csv
time,mem:memstats.Alloc,mem:memstats.Sys,mem:memstats.HeapAlloc,mem:memstats.HeapInuse,duration:memstats.PauseNs,duration:memstats.PauseTotalNs
2021-04-09 16:55:58,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:55:59,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:00,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:01,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:02,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:03,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:04,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:05,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:06,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:07,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:08,16MB,88MB,16MB,25MB,159μs,1m50s
2021-04-09 16:56:09,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:10,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:11,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:12,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:13,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:14,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:15,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:16,15MB,88MB,15MB,25MB,159μs,1m50s
2021-04-09 16:56:17,15MB,88MB,15MB,25MB,159μs,1m50s
... ...
4. 將csv數(shù)據(jù)轉(zhuǎn)換為excel圖表
csv存儲了各個應用暴露給外部的分時指標數(shù)據(jù)，但要對這些數(shù)據(jù)進行分析，我們需要將csv中的數(shù)據(jù)以可視化的形式展示出來，而excel圖表是一個不錯的選擇。
為此，我建立了一個csv2xls[6]的工具項目，專門用來將expvarmon生成的csv文件轉(zhuǎn)換為excel圖表。
csv2xls的用法如下：
$./csv2xls -h
Usage of ./csv2xls:
-col int
the column which we draw a chart based on, default: 1 (range 0~max-1) (default 1)
-i string
the name of csv file
-o string
the name of xls file
Examples:
./csv2xls -i xxx.csv
./csv2xls -i xxx.csv -o yyy.xlsx
csv2xls將csv文件中的數(shù)據(jù)讀取并存儲在xls中，并支持基于其中某列數(shù)據(jù)生成對應的折線圖。以上面的10.10.195.133:8080.csv為例，我們通過命令：csv2xls -i 10.10.195.133:8080.csv即可生成如下excel圖表文件：
csv2xls使用p著名的excelize(go語言execl操作庫)](github.com/360EntSecGroup-Skylar/excelize)生成excel文件。
5. 小結
至此，我們給expvarmon插上數(shù)據(jù)持久化的“翅膀”的目的算是初步達到了。但是由于app指標數(shù)據(jù)千變?nèi)f化，expvarmon使用的byten包[7]又給解析指標數(shù)據(jù)單位帶來了一些復雜性，因此csv2xls還不完善，后續(xù)還有很大的改進的空間。
支持公開指標持久化的expvarmon的代碼在這里[8](https://github.com/bigwhite/expvarmon)。 csv2xls的代碼在這里[9](https://github.com/bigwhite/csv2xls)。
參考資料
expvar包: https://tip.golang.org/pkg/expvar/
[2]runtime包的MemStats結構: https://tip.golang.org/pkg/runtime/#MemStats
[3]divan: https://github.com/divan
[4]expvarmon: https://github.com/divan/expvarmon
[5]issue: https://github.com/divan/expvarmon/issues/37
[6]csv2xls: https://github.com/bigwhite/csv2xls
[7]byten包: github.com/pyk/byten
[8]這里: https://github.com/bigwhite/expvarmon
[9]這里: https://github.com/bigwhite/csv2xls
[10]改善Go語?編程質(zhì)量的50個有效實踐: https://www.imooc.com/read/87
[11]Kubernetes實戰(zhàn)：高可用集群搭建、配置、運維與應用: https://coding.imooc.com/class/284.html
[12]我愛發(fā)短信: https://51smspush.com/
[13]鏈接地址: https://m.do.co/c/bff6eed92687
推薦閱讀