通過實例理解Go標準庫http包是如何處理keep-alive連接的
HTTP是如今互聯(lián)網(wǎng)的基礎(chǔ)協(xié)議，承載了互聯(lián)網(wǎng)上的絕大部分應(yīng)用層流量，并且從目前趨勢來看，在未來10年，http仍然會是互聯(lián)網(wǎng)應(yīng)用的主要協(xié)議。Go語言自帶“電池”[1]，基于Go標準庫我們可以輕松建立起一個http server處理客戶端http請求，或創(chuàng)建一個http client向服務(wù)端發(fā)送http請求。
最初早期的http 1.0協(xié)議只支持短連接，即客戶端每發(fā)送一個請求，就要和服務(wù)器端建立一個新TCP連接，請求處理完畢后，該連接將被拆除。顯然每次tcp連接握手和拆除都將帶來較大損耗，為了能充分利用已建立的連接，后來的http 1.0更新版和http 1.1支持在http請求頭中加入Connection: keep-alive來告訴對方這個請求響應(yīng)完成后不要關(guān)閉鏈接，下一次還要復(fù)用這個連接以繼續(xù)傳輸后續(xù)請求和響應(yīng)。后HTTP協(xié)議規(guī)范明確規(guī)定了HTTP/1.0版本如果想要保持長連接，需要在請求頭中加上Connection: keep-alive，而HTTP/1.1版本將支持keep-alive長連接作為默認選項，有沒有這個請求頭都可以。
本文我們就來一起看看Go標準庫中net/http包的http.Server和http.Client對keep-alive長連接的處理以及如何在Server和Client側(cè)關(guān)閉keep-alive機制。
1. http包默認啟用keep-alive
按照HTTP/1.1的規(guī)范，Go http包的http server和client的實現(xiàn)默認將所有連接視為長連接，無論這些連接上的初始請求是否帶有Connection: keep-alive。
下面分別是使用go http包的默認機制實現(xiàn)的一個http client和一個http server：
默認開啟keep-alive的http client實現(xiàn)：
//github.com/bigwhite/experiments/http-keep-alive/client-keepalive-on/client.go
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
c := &http.Client{}
req, err := http.NewRequest("Get", "http://localhost:8080", nil)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", *req)
for i := 0; i < 5; i++ {
resp, err := c.Do(req)
if err != nil {
fmt.Println("http get error:", err)
return
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("read body error:", err)
return
}
fmt.Println("response body:", string(b))
}
}
默認開啟keep-alive的http server實現(xiàn)：
//github.com/bigwhite/experiments/http-keep-alive/server-keepalive-on/server.go
package main
import (
"fmt"
"net/http"
)
func Index(w http.ResponseWriter, r *http.Request) {
fmt.Println("receive a request from:", r.RemoteAddr, r.Header)
w.Write([]byte("ok"))
}
func main() {
http.HandleFunc("/", Index)
var s = http.Server{
Addr: ":8080",
Handler: http.HandlerFunc(Index),
}
s.ListenAndServe()
}
現(xiàn)在我們啟動上面的http server：
// server-keepalive-on目錄下
$go run server.go
我們使用上面的client向該server發(fā)起5次http請求：
// client-keepalive-on目錄下
$go run client.go
http.Request{Method:"Get", URL:(*url.URL)(0xc00016a000), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.ReadCloser(nil), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"localhost:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.emptyCtx)(0xc00012c008)}
response body: ok
response body: ok
response body: ok
response body: ok
response body: ok
這期間server端輸出的日志如下：
receive a request from: [::1]:55238 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
receive a request from: [::1]:55238 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
receive a request from: [::1]:55238 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
receive a request from: [::1]:55238 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
receive a request from: [::1]:55238 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
我們簡單分析一下兩端的輸出結(jié)果：
從server端打印的請求的頭部字段來看，客戶端發(fā)來的請求header中并沒有顯式包含Connection: keep-alive，而僅有Accept-Encoding和User-Agent兩個header字段； server端處理的5個請求均來自同一個連接“[::1]:55238”，Server端默認保持了該連接，而不是在處理完一個請求后將連接關(guān)閉，說明兩端均復(fù)用了第一個請求創(chuàng)建的http連接。
即便我們的client端每間隔5秒發(fā)送一次請求，server端默認也不會關(guān)閉連接(我們將fmt包緩沖log包，輸出帶有時間戳的日志)：
// client-keepalive-on目錄下
$go run client-with-delay.go
http.Request{Method:"Get", URL:(*url.URL)(0xc00016a000), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.ReadCloser(nil), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"localhost:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.emptyCtx)(0xc00012c008)}
2021/01/03 12:25:21 response body: ok
2021/01/03 12:25:26 response body: ok
2021/01/03 12:25:31 response body: ok
2021/01/03 12:25:36 response body: ok
2021/01/03 12:25:41 response body: ok
// server-keepalive-on目錄下
$go run server.go
2021/01/03 12:25:21 receive a request from: [::1]:58419 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:25:26 receive a request from: [::1]:58419 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:25:31 receive a request from: [::1]:58419 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:25:36 receive a request from: [::1]:58419 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:25:41 receive a request from: [::1]:58419 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2. http client端基于非keep-alive連接發(fā)送請求
有時候http client在一條連接上的數(shù)據(jù)請求密度并不高，因此client端并不想長期保持這條連接(占用端口資源)，那么client端如何協(xié)調(diào)Server端在處理完請求返回應(yīng)答后就關(guān)閉這條連接呢？我們看看在Go中如何實現(xiàn)這一場景需求：
//github.com/bigwhite/experiments/http-keep-alive/client-keepalive-off/client.go
... ...
func main() {
tr := &http.Transport{
DisableKeepAlives: true,
}
c := &http.Client{
Transport: tr,
}
req, err := http.NewRequest("Get", "http://localhost:8080", nil)
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
resp, err := c.Do(req)
if err != nil {
fmt.Println("http get error:", err)
return
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("read body error:", err)
return
}
log.Println("response body:", string(b))
time.Sleep(5 * time.Second)
}
}
http.Client底層的數(shù)據(jù)連接建立和維護是由http.Transport實現(xiàn)的，http.Transport結(jié)構(gòu)有一個DisableKeepAlives字段，其默認值為false，即啟動keep-alive。這里我們將其置為false，即關(guān)閉keep-alive，然后將該Transport實例作為初值，賦值給http Client實例的Transport字段。
接下來，我們使用這個client向上面那個http server發(fā)送五個請求，請求間間隔5秒(模擬連接空閑的狀態(tài))，我們得到如下結(jié)果(從server端打印信息觀察)：
// 在client-keepalive-off下面
$go run client.go
2021/01/03 12:42:38 response body: ok
2021/01/03 12:42:43 response body: ok
2021/01/03 12:42:48 response body: ok
2021/01/03 12:42:53 response body: ok
2021/01/03 12:42:58 response body: ok
// 在server-keepalive-on下面
$go run server.go
2021/01/03 12:42:38 receive a request from: [::1]:62287 map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:42:43 receive a request from: [::1]:62301 map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:42:48 receive a request from: [::1]:62314 map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:42:53 receive a request from: [::1]:62328 map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Go-http-client/1.1]]
2021/01/03 12:42:58 receive a request from: [::1]:62342 map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Go-http-client/1.1]]
從Server的輸出結(jié)果來看，來自客戶端的請求中增加了**Connection:[close]**的頭字段，當收到這樣的請求后，Server端便不再保持這一連接了。我們也看到上面日志中，每個請求都是通過不同的客戶端端口發(fā)送出來的，顯然這是五條不同的連接。
3. 建立一個不支持keep-alive連接的http server
假設(shè)我們有這樣的一個需求，server端完全不支持keep-alive的連接，無論client端發(fā)送的請求header中是否顯式帶有Connection: keep-alive，server端都會在返回應(yīng)答后關(guān)閉連接。那么在Go中，我們?nèi)绾蝸韺崿F(xiàn)這一需求呢？我們來看下面代碼：
//github.com/bigwhite/experiments/http-keep-alive/server-keepalive-off/server.go
package main
import (
"log"
"net/http"
)
func Index(w http.ResponseWriter, r *http.Request) {
log.Println("receive a request from:", r.RemoteAddr, r.Header)
w.Write([]byte("ok"))
}
func main() {
http.HandleFunc("/", Index)
var s = http.Server{
Addr: ":8080",
Handler: http.HandlerFunc(Index),
}
s.SetKeepAlivesEnabled(false)
s.ListenAndServe()
}
我們看到在ListenAndServe前，我們調(diào)用了http.Server的SetKeepAlivesEnabled方法，并傳入false參數(shù)，這樣我們就在全局層面關(guān)閉了該Server對keep-alive連接的支持，我們用前面client-keepalive-on下面的client向該Server發(fā)送五個請求：
// 在client-keepalive-on下面
$go run client.go
http.Request{Method:"Get", URL:(*url.URL)(0xc000174000), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{}, Body:io.ReadCloser(nil), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"localhost:8080", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.emptyCtx)(0xc00013a008)}
2021/01/03 13:30:08 response body: ok
2021/01/03 13:30:08 response body: ok
2021/01/03 13:30:08 response body: ok
2021/01/03 13:30:08 response body: ok
2021/01/03 13:30:08 response body: ok
// 在server-keepalive-off下面
$go run server.go
2021/01/03 13:30:08 receive a request from: [::1]:53005 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 13:30:08 receive a request from: [::1]:53006 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 13:30:08 receive a request from: [::1]:53007 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 13:30:08 receive a request from: [::1]:53008 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 13:30:08 receive a request from: [::1]:53009 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
我們看到該Server在處理完每個請求后就關(guān)閉了傳輸該請求的連接，這導(dǎo)致client測不得不為每個請求建立一個新連接(從server輸出的客戶端地址和端口看出)。
4. 支持長連接閑置超時關(guān)閉的http server
顯然上面的server處理方式“太過霸道”，對于想要復(fù)用連接，提高請求和應(yīng)答傳輸效率的client而言，上面的“一刀切”機制并不合理。那么是否有一種機制可以讓http server即可以對高密度傳輸數(shù)據(jù)的連接保持keep-alive，又可以及時清理掉那些長時間沒有數(shù)據(jù)傳輸?shù)膇dle連接，釋放占用的系統(tǒng)資源呢？我們來看下面這個go實現(xiàn)的server：
//github.com/bigwhite/experiments/http-keep-alive/server-keepalive-with-idletimeout/server.go
package main
import (
"log"
"net/http"
"time"
)
func Index(w http.ResponseWriter, r *http.Request) {
log.Println("receive a request from:", r.RemoteAddr, r.Header)
w.Write([]byte("ok"))
}
func main() {
http.HandleFunc("/", Index)
var s = http.Server{
Addr: ":8080",
Handler: http.HandlerFunc(Index),
IdleTimeout: 5 * time.Second,
}
s.ListenAndServe()
}
從代碼中我們看到，我們僅在創(chuàng)建http.Server實例時顯式為其字段IdleTimeout做了一次顯式賦值，設(shè)置idle連接的超時時間為5s。下面是Go標準庫中關(guān)于http.Server的字段IdleTimeout的注釋：
// $GOROOT/src/net/server.go
// IdleTimeout是當啟用keep-alive時等待下一個請求的最大時間。
// 如果IdleTimeout為零，則使用ReadTimeout的值。如果兩者都是
// 零，則沒有超時。
IdleTimeout time.Duration
我們來看看效果如何，是否是我們期望那樣的。為了測試效果，我們改造了client端，放在client-keepalive-on-with-idle下面：
//github.com/bigwhite/experiments/http-keep-alive/client-keepalive-on-with-idle/client.go
... ...
func main() {
c := &http.Client{}
req, err := http.NewRequest("Get", "http://localhost:8080", nil)
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
log.Printf("round %d begin:\n", i+1)
for j := 0; j < i+1; j++ {
resp, err := c.Do(req)
if err != nil {
fmt.Println("http get error:", err)
return
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("read body error:", err)
return
}
log.Println("response body:", string(b))
}
log.Printf("round %d end\n", i+1)
time.Sleep(7 * time.Second)
}
}
client端請求分為5輪，輪與輪之間間隔7秒，下面是通信過程與結(jié)果：
// 在client-keepalive-on-with-idle下
$go run client.go
2021/01/03 14:17:05 round 1 begin:
2021/01/03 14:17:05 response body: ok
2021/01/03 14:17:05 round 1 end
2021/01/03 14:17:12 round 2 begin:
2021/01/03 14:17:12 response body: ok
2021/01/03 14:17:12 response body: ok
2021/01/03 14:17:12 round 2 end
2021/01/03 14:17:19 round 3 begin:
2021/01/03 14:17:19 response body: ok
2021/01/03 14:17:19 response body: ok
2021/01/03 14:17:19 response body: ok
2021/01/03 14:17:19 round 3 end
2021/01/03 14:17:26 round 4 begin:
2021/01/03 14:17:26 response body: ok
2021/01/03 14:17:26 response body: ok
2021/01/03 14:17:26 response body: ok
2021/01/03 14:17:26 response body: ok
2021/01/03 14:17:26 round 4 end
2021/01/03 14:17:33 round 5 begin:
2021/01/03 14:17:33 response body: ok
2021/01/03 14:17:33 response body: ok
2021/01/03 14:17:33 response body: ok
2021/01/03 14:17:33 response body: ok
2021/01/03 14:17:33 response body: ok
2021/01/03 14:17:33 round 5 end
// 在server-keepalive-with-idletimeout下
$go run server.go
2021/01/03 14:17:05 receive a request from: [::1]:64071 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:12 receive a request from: [::1]:64145 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:12 receive a request from: [::1]:64145 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:19 receive a request from: [::1]:64189 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:19 receive a request from: [::1]:64189 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:19 receive a request from: [::1]:64189 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:26 receive a request from: [::1]:64250 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:26 receive a request from: [::1]:64250 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:26 receive a request from: [::1]:64250 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:26 receive a request from: [::1]:64250 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:33 receive a request from: [::1]:64304 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:33 receive a request from: [::1]:64304 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:33 receive a request from: [::1]:64304 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:33 receive a request from: [::1]:64304 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:17:33 receive a request from: [::1]:64304 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
我們看到：
在每輪內(nèi)，client端的所有請求都是復(fù)用已建立的連接； 但每輪之間，由于Sleep了7秒，超出了server端idletimeout的時長，上一輪的連接被拆除，新一輪只能重建連接。
我們期望的效果實現(xiàn)了！
5. 一個http client可管理到多個server的連接
Go標準庫的http.Client與一個server可不是一對一的關(guān)系，它可以實現(xiàn)一對多的http通信，也就是說一個http client可管理到多個server的連接，并優(yōu)先復(fù)用到同一server的連接(keep-alive)，而不是建立新連接，就像我們上面看到的那樣。我們來創(chuàng)建一個向多個server發(fā)送請求的client：
//github.com/bigwhite/experiments/http-keep-alive/client-keepalive-on-to-multiple-servers/client.go
... ...
func main() {
c := &http.Client{}
req1, err := http.NewRequest("Get", "http://localhost:8080", nil)
if err != nil {
panic(err)
}
req2, err := http.NewRequest("Get", "http://localhost:8081", nil)
if err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
resp1, err := c.Do(req1)
if err != nil {
fmt.Println("http get error:", err)
return
}
defer resp1.Body.Close()
b1, err := ioutil.ReadAll(resp1.Body)
if err != nil {
fmt.Println("read body error:", err)
return
}
log.Println("response1 body:", string(b1))
resp2, err := c.Do(req2)
if err != nil {
fmt.Println("http get error:", err)
return
}
defer resp2.Body.Close()
b2, err := ioutil.ReadAll(resp2.Body)
if err != nil {
fmt.Println("read body error:", err)
return
}
log.Println("response2 body:", string(b2))
time.Sleep(5 * time.Second)
}
}
我們建立兩個默認的http server，分別監(jiān)聽8080和8081，運行上面client：
$go run client.go
2021/01/03 14:52:20 response1 body: ok
2021/01/03 14:52:20 response2 body: ok
2021/01/03 14:52:25 response1 body: ok
2021/01/03 14:52:25 response2 body: ok
2021/01/03 14:52:30 response1 body: ok
2021/01/03 14:52:30 response2 body: ok
2021/01/03 14:52:35 response1 body: ok
2021/01/03 14:52:35 response2 body: ok
2021/01/03 14:52:40 response1 body: ok
2021/01/03 14:52:40 response2 body: ok
server端的輸出結(jié)果如下：
// server1(8080):
2021/01/03 14:52:20 receive a request from: [::1]:63871 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:25 receive a request from: [::1]:63871 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:30 receive a request from: [::1]:63871 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:35 receive a request from: [::1]:63871 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:40 receive a request from: [::1]:63871 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
// server2(8081):
2021/01/03 14:52:20 receive a request from: [::1]:63872 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:25 receive a request from: [::1]:63872 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:30 receive a request from: [::1]:63872 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:35 receive a request from: [::1]:63872 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2021/01/03 14:52:40 receive a request from: [::1]:63872 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
我們看到client同時支持與多個server進行通信，并針對每個server可以使用keep-alive的連接進行高效率通信。
本文涉及源代碼可以在這里[2](https://github.com/bigwhite/experiments/tree/master/http-keep-alive)下載。
參考資料
Go語言自帶“電池”: https://www.imooc.com/read/87/article/2341
[2]這里: https://github.com/bigwhite/experiments/tree/master/http-keep-alive
[3]改善Go語?編程質(zhì)量的50個有效實踐: https://www.imooc.com/read/87
[4]Kubernetes實戰(zhàn)：高可用集群搭建、配置、運維與應(yīng)用: https://coding.imooc.com/class/284.html
[5]我愛發(fā)短信: https://51smspush.com/
[6]鏈接地址: https://m.do.co/c/bff6eed92687
推薦閱讀
評論
圖片
表情