客戶端禁用Keep-Alive， 服務(wù)端開(kāi)啟Keep-Alive，會(huì)怎么樣？

最近部署的web程序，服務(wù)器上出現(xiàn)不少time_wait的tcp連接狀態(tài)，占用了tcp端口，花費(fèi)幾天時(shí)間排查。

? ? ? ?之前我有結(jié)論：HTTP keep-alive 是在應(yīng)用層對(duì)TCP連接的滑動(dòng)續(xù)約復(fù)用，如果客戶端、服務(wù)器穩(wěn)定續(xù)約，就成了名副其實(shí)的長(zhǎng)連接。

有關(guān)[Http持久連接]的一切，卷給你看

HTTP1.1 Keep-Alive到底算不算長(zhǎng)連接？

目前所有的HTTP網(wǎng)絡(luò)庫(kù)(不論是客戶端、服務(wù)端)都默認(rèn)開(kāi)啟了HTTP Keep-Alive，通過(guò)Request/Response的Connection標(biāo)頭來(lái)協(xié)商復(fù)用連接。

? ? ? ? 我手上有個(gè)項(xiàng)目，由于歷史原因，客戶端禁用了Keep-Alive，服務(wù)端默認(rèn)開(kāi)啟了Keep-Alive，如此一來(lái)協(xié)商復(fù)用連接失敗， 客戶端每次請(qǐng)求會(huì)使用新的TCP連接， 也就是回退為短連接。

客戶端強(qiáng)制禁用Keep-Alive

package?main
import?(
????"fmt"
????"io/ioutil"
????"log"
????"net/http"
????"time"
)

func?main()?{
????tr?:=?http.Transport{
????????DisableKeepAlives:?true,
????}
????client?:=?&http.Client{
????????Timeout:???10?*?time.Second,
????????Transport:?&tr,
????}
????for?{
????????requestWithClose(client)
????????time.Sleep(time.Second?*?1)
????}
}

func?requestWithClose(client?*http.Client)?{
????resp,?err?:=?client.Get("http://10.100.219.9:8081")
????if?err?!=?nil?{
????????fmt.Printf("error?occurred?while?fetching?page,?error:?%s",?err.Error())
????????return
????}
????defer?resp.Body.Close()
????c,?err?:=?ioutil.ReadAll(resp.Body)
????if?err?!=?nil?{
????????log.Fatalf("Couldn't?parse?response?body.?%+v",?err)
????}

????fmt.Println(string(c))
}

web服務(wù)端默認(rèn)開(kāi)啟Keep-Alive

package?main

import?(
????"fmt"
????"log"
????"net/http"
)

//?根據(jù)RemoteAddr?知道客戶端使用的持久連接
func?IndexHandler(w?http.ResponseWriter,?r?*http.Request)?{
????fmt.Println("receive?a?request?from:",?r.RemoteAddr,?r.Header)
????w.Write([]byte("ok"))
}

func?main()?{
????fmt.Printf("Starting?server?at?port?8081\n")
????//?net/http?默認(rèn)開(kāi)啟持久連接
????if?err?:=?http.ListenAndServe(":8081",?http.HandlerFunc(IndexHandler));?err?!=?nil?{
????????log.Fatal(err)
????}
}

從服務(wù)端的日志看，確實(shí)是短連接。

receive?a?request?from:?10.22.34.48:54722?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54724?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54726?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54728?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54731?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54733?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54734?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54738?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from:?10.22.34.48:54740?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54741?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54743?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54744?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]
receive?a?request?from: 10.22.34.48:54746?map[Accept-Encoding:[gzip]?Connection:[close]?User-Agent:[Go-http-client/1.1]]

我想當(dāng)然的以為 客戶端是主動(dòng)斷開(kāi)方，被現(xiàn)實(shí)啪啪打臉。

某一天服務(wù)器上超過(guò)300的time_wait報(bào)警，告訴我這tmd是服務(wù)器主動(dòng)終斷連接。

常規(guī)的TCP4次揮手， 主動(dòng)斷開(kāi)方會(huì)進(jìn)入time_wait狀態(tài)，等待2MSL后釋放占用的SOCKET

以下是從服務(wù)器上tcpdump抓取的tcp連接信息。

2,3紅框顯示：

? ? ? Server端先發(fā)起TCP的FIN消息, 之后Client回應(yīng)ACK確認(rèn)收到Server的關(guān)閉通知；?之后Client再發(fā)FIN消息，告知現(xiàn)在可以關(guān)閉了， Server端最后發(fā)ACK確認(rèn)收到，并進(jìn)入time_wait狀態(tài)，等待2MSL的時(shí)間關(guān)閉Socket。

特意指出，紅框1表示TCP雙端同時(shí)關(guān)閉^[1]，此時(shí)會(huì)在Client,Server同時(shí)留下time_wait痕跡，發(fā)生概率較小。

此種情況是服務(wù)端主動(dòng)關(guān)閉，我們翻一翻golang httpServer的源碼

?http.ListenAndServe(":8081")?server.ListenAndServe()?srv.Serve(ln)?go c.serve(connCtx) 使用go協(xié)程來(lái)處理每個(gè)請(qǐng)求

服務(wù)器連接處理請(qǐng)求的簡(jiǎn)略源碼如下：

func?(c?*conn)?serve(ctx?context.Context)?{
????c.remoteAddr?=?c.rwc.RemoteAddr().String()
????ctx?=?context.WithValue(ctx,?LocalAddrContextKey,?c.rwc.LocalAddr())
????defer?func()?{
????if?!c.hijacked()?{
????????????c.close()? ?// go協(xié)程conn處理請(qǐng)求的協(xié)程退出時(shí)，主動(dòng)關(guān)閉底層的TCP連接
????????????c.setState(c.rwc,?StateClosed,?runHooks)
????????}
????}()

??......
????//?HTTP/1.x?from?here?on.

????ctx,?cancelCtx?:=?context.WithCancel(ctx)
????c.cancelCtx?=?cancelCtx
????defer?cancelCtx()

????c.r?=?&connReader{conn:?c}
????c.bufr?=?newBufioReader(c.r)
????c.bufw?=?newBufioWriterSize(checkConnErrorWriter{c},?4<<10)

????for?{
????????w,?err?:=?c.readRequest(ctx)
.....
????????serverHandler{c.server}.ServeHTTP(w,?w.req)
????????w.cancelCtx()
????????if?c.hijacked()?{
????????????return
????????}
????????w.finishRequest()
????????if?!w.shouldReuseConnection()?{
????????????if?w.requestBodyLimitHit?||?w.closedRequestBodyEarly()?{
????????????????c.closeWriteAndWait()
????????????}
????????????return
????????}
????????c.setState(c.rwc,?StateIdle,?runHooks)
????????c.curReq.Store((*response)(nil))

????????if?!w.conn.server.doKeepAlives()?{
????????????//?We're?in?shutdown?mode.?We?might've?replied
????????????//?to?the?user?without?"Connection:?close"?and
????????????//?they?might?think?they?can?send?another
????????????//?request,?but?such?is?life?with?HTTP/1.1.
????????????return
????????}

????????if?d?:=?c.server.idleTimeout();?d?!=?0?{
????????????c.rwc.SetReadDeadline(time.Now().Add(d))
????????????if?_,?err?:=?c.bufr.Peek(4);?err?!=?nil?{
????????????????return
????????????}
????????}
????????c.rwc.SetReadDeadline(time.Time{})
????}
}

我們需要關(guān)注

①for循環(huán)，表示嘗試復(fù)用該conn，用于處理迎面而來(lái)的請(qǐng)求②w.shouldReuseConnection() = false， 表明讀取到ClientConnection：Close標(biāo)頭，設(shè)置closeAfterReply=true，跳出for循環(huán)，協(xié)程即將結(jié)束，結(jié)束之前執(zhí)行defer函數(shù)，defer函數(shù)內(nèi)close該連接? c.close()
......
//?Close?the?connection.
func?(c?*conn)?close()?{
?c.finalFlush()
?c.rwc.Close()
}③如果 w.shouldReuseConnection() = true，則將該連接狀態(tài)置為idle， 并繼續(xù)走for循環(huán)，復(fù)用連接，處理后續(xù)請(qǐng)求。

1. TCP 4次揮手的八股文2. ?短連接的效應(yīng)：主動(dòng)關(guān)閉方會(huì)在機(jī)器上產(chǎn)生 time_wait狀態(tài)，需要等待2MSL時(shí)間才會(huì)關(guān)閉SOCKET3.golang http keep-alive復(fù)用tcp連接的源碼級(jí)分析4.tcpdump抓包的姿勢(shì)

引用鏈接

[1]?TCP雙端同時(shí)關(guān)閉:?https://blog.csdn.net/q1007729991/article/details/69950255