Spring Boot + Filter 實(shí)現(xiàn) Gzip 壓縮超大 json 對(duì)象,傳輸耗時(shí)大大減少!

1. 業(yè)務(wù)背景

是這樣的，業(yè)務(wù)背景是公司的內(nèi)部系統(tǒng)有一個(gè)廣告保存接口，需要ADX那邊將投放的廣告數(shù)據(jù)進(jìn)行保存供后續(xù)使用。 廣告數(shù)據(jù)大概長(zhǎng)這樣：

• adName是廣告名字
• adTag是廣告渲染的HTML代碼，超級(jí)大數(shù)據(jù)庫(kù)中都是用text類(lèi)型來(lái)存放的，我看到最大的adTag足足有60kb大小…

{
????"adName":"",
????"adTag":""
}


因此，對(duì)與請(qǐng)求數(shù)據(jù)那么大的接口我們肯定是需要作一個(gè)優(yōu)化的否則太大的數(shù)據(jù)傳輸有以下幾個(gè)弊端：

• 占用網(wǎng)絡(luò)帶寬，而有些云產(chǎn)品就是按照帶寬來(lái)計(jì)費(fèi)的，間接浪費(fèi)了錢(qián)
• 傳輸數(shù)據(jù)大導(dǎo)致網(wǎng)絡(luò)傳輸耗時(shí)

為了克服這幾個(gè)問(wèn)題團(tuán)隊(duì)中的老鳥(niǎo)產(chǎn)生一個(gè)想法：

請(qǐng)求廣告保存接口時(shí)先將Json對(duì)象字符串進(jìn)行GZIP壓縮，那請(qǐng)求時(shí)傳入的就是壓縮后的數(shù)據(jù)，而GZIP的壓縮效率是很高的，因此可以大大減小傳輸數(shù)據(jù)，而當(dāng)數(shù)據(jù)到達(dá)廣告保存接口前再將傳來(lái)的數(shù)據(jù)進(jìn)行解壓縮，還原成JSON對(duì)象就完成了整個(gè)GZIP壓縮數(shù)據(jù)的請(qǐng)求以及處理流程。

其實(shí)這樣做也存在著弊端：

• 請(qǐng)求變復(fù)雜了
  • 接口調(diào)用方那邊需要對(duì)數(shù)據(jù)進(jìn)行壓縮
  • 接口執(zhí)行方那邊需要對(duì)拿到的數(shù)據(jù)進(jìn)行解壓
• 需要額外占用更多的CPU計(jì)算資源
• 可能會(huì)影響到原有的其他接口

對(duì)于以上幾點(diǎn)基于我們公司當(dāng)前的業(yè)務(wù)可以這樣解決：

• 對(duì)與需要占用而外的CPU計(jì)算資源來(lái)說(shuō)，公司的內(nèi)部系統(tǒng)屬于IO密集型應(yīng)用，因此用一些CPU資源來(lái)?yè)Q取更快的網(wǎng)絡(luò)傳輸其實(shí)是很劃算的
• 使用過(guò)濾器在請(qǐng)求數(shù)據(jù)到達(dá)Controller之前對(duì)數(shù)據(jù)進(jìn)行解壓縮處理后重新寫(xiě)回到Body中，避免影響Controller的邏輯，代碼零侵入
• 而對(duì)于改造接口的同時(shí)是否會(huì)影響到原來(lái)的接口這一點(diǎn)可以通過(guò) HttpHeader 的Content-Encoding=gzip屬性來(lái)區(qū)分是否需要對(duì)請(qǐng)求數(shù)據(jù)進(jìn)行解壓縮

那廢話少說(shuō)，下面給出實(shí)現(xiàn)方案

2. 實(shí)現(xiàn)思路

前置知識(shí)：

• Http 請(qǐng)求結(jié)構(gòu)以及Content-Encoding 屬性
• gzip壓縮方式
• Servlet Filter
• HttpServletRequestWrapper
• Spring Boot
• Java 輸入輸出流

實(shí)現(xiàn)流程圖：

核心代碼：

創(chuàng)建一個(gè)SpringBoot項(xiàng)目，先編寫(xiě)一個(gè)接口，功能很簡(jiǎn)單就是傳入一個(gè)Json對(duì)象并返回，以模擬將廣告數(shù)據(jù)保存到數(shù)據(jù)庫(kù)

/**
?*?@ClassName:?ProjectController
?*?@Author?zhangjin
?*?@Date?2022/3/24?20:41
?*?@Description:
?*/
@Slf4j
@RestController
public?class?AdvertisingController?{

????@PostMapping("/save")
????public?Advertising?saveProject(@RequestBody?Advertising?advertising)?{
????????log.info("獲取內(nèi)容"+?advertising);
????????return?advertising;
????}
}

/**
?*?@ClassName:?Project
?*?@Author?zhangjin
?*?@Date?2022/3/24?20:42
?*?@Description:
?*/
@Data
public?class?Advertising?{
????private?String?adName;
????private?String?adTag;
}


編寫(xiě)并注冊(cè)一個(gè)攔截器

/**
?*?@ClassName:?GZIPFilter
?*?@Author?zhangjin
?*?@Date?2022/3/26?0:36
?*?@Description:
?*/
@Slf4j
@Component
public?class?GZIPFilter?implements?Filter?{

????private?static?final?String?CONTENT_ENCODING?=?"Content-Encoding";
????private?static?final?String?CONTENT_ENCODING_TYPE?=?"gzip";

????@Override
????public?void?init(FilterConfig?filterConfig)?throws?ServletException?{
????????log.info("init?GZIPFilter");
????}

????@Override
????public?void?doFilter(ServletRequest?servletRequest,?ServletResponse?servletResponse,?FilterChain?filterChain)?throws?IOException,?ServletException?{
????????long?start?=?System.currentTimeMillis();
????????HttpServletRequest?httpServletRequest?=?(HttpServletRequest)servletRequest;

????????String?encodeType?=?httpServletRequest.getHeader(CONTENT_ENCODING);
????????if?(CONTENT_ENCODING_TYPE.equals(encodeType))?{
????????????log.info("請(qǐng)求:{}?需要解壓",?httpServletRequest.getRequestURI());
????????????UnZIPRequestWrapper?unZIPRequestWrapper?=?new?UnZIPRequestWrapper(httpServletRequest);
????????????filterChain.doFilter(unZIPRequestWrapper,servletResponse);
????????}
????????else?{
????????????log.info("請(qǐng)求:{}?無(wú)需解壓",?httpServletRequest.getRequestURI());
????????????filterChain.doFilter(servletRequest,servletResponse);
????????}
????????log.info("耗時(shí)：{}ms",?System.currentTimeMillis()?-?start);
????}

????@Override
????public?void?destroy()?{
????????log.info("destroy?GZIPFilter");
????}
}

/**
?*?@ClassName:?FilterRegistration
?*?@Author?zhangjin
?*?@Date?2022/3/26?0:36
?*?@Description:
?*/
@Configuration
public?class?FilterRegistration?{

????@Resource
????private?GZIPFilter?gzipFilter;

????@Bean
????public?FilterRegistrationBean?gzipFilterRegistrationBean()?{
????????FilterRegistrationBean?registration?=?new?FilterRegistrationBean<>();
????????//Filter可以new，也可以使用依賴注入Bean
????????registration.setFilter(gzipFilter);
????????//過(guò)濾器名稱(chēng)
????????registration.setName("gzipFilter");
????????//攔截路徑
????????registration.addUrlPatterns("/*");
????????//設(shè)置順序
????????registration.setOrder(1);
????????return?registration;
????}
}


實(shí)現(xiàn)RequestWrapper實(shí)現(xiàn)解壓和寫(xiě)回Body的邏輯

/**
?*?@ClassName:?UnZIPRequestWrapper
?*?@Author?zhangjin
?*?@Date?2022/3/26?11:02
?*?@Description:?JsonString經(jīng)過(guò)壓縮后保存為二進(jìn)制文件?->?解壓縮后還原成JsonString轉(zhuǎn)換成byte[]?寫(xiě)回body中
?*/
@Slf4j
public?class?UnZIPRequestWrapper?extends?HttpServletRequestWrapper?{

????private?final?byte[]?bytes;

????public?UnZIPRequestWrapper(HttpServletRequest?request)?throws?IOException?{
????????super(request);
????????try?(BufferedInputStream?bis?=?new?BufferedInputStream(request.getInputStream());
?????????????ByteArrayOutputStream?baos?=?new?ByteArrayOutputStream())?{
????????????final?byte[]?body;
????????????byte[]?buffer?=?new?byte[1024];
????????????int?len;
????????????while?((len?=?bis.read(buffer))?>?0)?{
????????????????baos.write(buffer,?0,?len);
????????????}
????????????body?=?baos.toByteArray();
????????????if?(body.length?==?0)?{
????????????????log.info("Body無(wú)內(nèi)容，無(wú)需解壓");
????????????????bytes?=?body;
????????????????return;
????????????}
????????????this.bytes?=?GZIPUtils.uncompressToByteArray(body);
????????}?catch?(IOException?ex)?{
????????????log.info("解壓縮步驟發(fā)生異常！");
????????????ex.printStackTrace();
????????????throw?ex;
????????}
????}

????@Override
????public?ServletInputStream?getInputStream()?throws?IOException?{
????????final?ByteArrayInputStream?byteArrayInputStream?=?new?ByteArrayInputStream(bytes);
????????return?new?ServletInputStream()?{

????????????@Override
????????????public?boolean?isFinished()?{
????????????????return?false;
????????????}

????????????@Override
????????????public?boolean?isReady()?{
????????????????return?false;
????????????}

????????????@Override
????????????public?void?setReadListener(ReadListener?readListener)?{

????????????}

????????????public?int?read()?throws?IOException?{
????????????????return?byteArrayInputStream.read();
????????????}
????????};
????}

????@Override
????public?BufferedReader?getReader()?throws?IOException?{
????????return?new?BufferedReader(new?InputStreamReader(this.getInputStream()));
????}

}


附上壓縮工具類(lèi)

public?class?GZIPUtils?{
?
????public?static?final?String?GZIP_ENCODE_UTF_8?=?"UTF-8";

????/**
?????*?字符串壓縮為GZIP字節(jié)數(shù)組
?????*?@param?str
?????*?@return
?????*/
????public?static?byte[]?compress(String?str)?{
????????return?compress(str,?GZIP_ENCODE_UTF_8);
????}
?
????/**
?????*?字符串壓縮為GZIP字節(jié)數(shù)組
?????*?@param?str
?????*?@param?encoding
?????*?@return
?????*/
????public?static?byte[]?compress(String?str,?String?encoding)?{
????????if?(str?==?null?||?str.length()?==?0)?{
????????????return?null;
????????}
????????ByteArrayOutputStream?out?=?new?ByteArrayOutputStream();
????????GZIPOutputStream?gzip;
????????try?{
????????????gzip?=?new?GZIPOutputStream(out);
????????????gzip.write(str.getBytes(encoding));
????????????gzip.close();
????????}?catch?(IOException?e)?{
????????????e.printStackTrace();
????????}
????????return?out.toByteArray();
????}
?
????/**
?????*?GZIP解壓縮
?????*?@param?bytes
?????*?@return
?????*/
????public?static?byte[]?uncompress(byte[]?bytes)?{
????????if?(bytes?==?null?||?bytes.length?==?0)?{
????????????return?null;
????????}
????????ByteArrayOutputStream?out?=?new?ByteArrayOutputStream();
????????ByteArrayInputStream?in?=?new?ByteArrayInputStream(bytes);
????????try?{
????????????GZIPInputStream?ungzip?=?new?GZIPInputStream(in);
????????????byte[]?buffer?=?new?byte[256];
????????????int?n;
????????????while?((n?=?ungzip.read(buffer))?>=?0)?{
????????????????out.write(buffer,?0,?n);
????????????}
????????}?catch?(IOException?e)?{
????????????e.printStackTrace();
????????}
????????return?out.toByteArray();
????}
?
????/**
?????*?解壓并返回String
?????*?@param?bytes
?????*?@return
?????*/
????public?static?String?uncompressToString(byte[]?bytes)?throws?IOException?{
????????return?uncompressToString(bytes,?GZIP_ENCODE_UTF_8);
????}

????/**
?????*
?????*?@param?bytes
?????*?@return
?????*/
????public?static?byte[]?uncompressToByteArray(byte[]?bytes)?throws?IOException?{
????????return?uncompressToByteArray(bytes,?GZIP_ENCODE_UTF_8);
????}
?
????/**
?????*?解壓成字符串
?????*?@param?bytes?壓縮后的字節(jié)數(shù)組
?????*?@param?encoding?編碼方式
?????*?@return?解壓后的字符串
?????*/
????public?static?String?uncompressToString(byte[]?bytes,?String?encoding)?throws?IOException?{
????????byte[]?result?=?uncompressToByteArray(bytes,?encoding);
????????return?new?String(result);
????}

????/**
?????*?解壓成字節(jié)數(shù)組
?????*?@param?bytes
?????*?@param?encoding
?????*?@return
?????*/
????public?static?byte[]?uncompressToByteArray(byte[]?bytes,?String?encoding)?throws?IOException?{
????????if?(bytes?==?null?||?bytes.length?==?0)?{
????????????return?null;
????????}
????????ByteArrayOutputStream?out?=?new?ByteArrayOutputStream();
????????ByteArrayInputStream?in?=?new?ByteArrayInputStream(bytes);
????????try?{
????????????GZIPInputStream?ungzip?=?new?GZIPInputStream(in);
????????????byte[]?buffer?=?new?byte[256];
????????????int?n;
????????????while?((n?=?ungzip.read(buffer))?>=?0)?{
????????????????out.write(buffer,?0,?n);
????????????}
????????????return?out.toByteArray();
????????}?catch?(IOException?e)?{
????????????e.printStackTrace();
????????????throw?new?IOException("解壓縮失敗！");
????????}
????}

????/**
?????*?將字節(jié)流轉(zhuǎn)換成文件
?????*?@param?filename
?????*?@param?data
?????*?@throws?Exception
?????*/
????public?static?void?saveFile(String?filename,byte?[]?data)throws?Exception{
????????if(data?!=?null){
????????????String?filepath?="/"?+?filename;
????????????File?file??=?new?File(filepath);
????????????if(file.exists()){
????????????????file.delete();
????????????}
????????????FileOutputStream?fos?=?new?FileOutputStream(file);
????????????fos.write(data,0,data.length);
????????????fos.flush();
????????????fos.close();
????????????System.out.println(file);
????????}
????}


3. 測(cè)試效果

注意一個(gè)大坑：千萬(wàn)不要直接將壓縮后的byte[]當(dāng)作字符串進(jìn)行傳輸，否則你會(huì)發(fā)現(xiàn)壓縮后的請(qǐng)求數(shù)據(jù)竟然比沒(méi)壓縮后的要大得多??！一般有兩種傳輸壓縮后的byte[]的方式：

• 將壓縮后的byet[]進(jìn)行base64編碼再傳輸字符串，這種方式會(huì)損失掉一部分GZIP的壓縮效果，適用于壓縮結(jié)果要存儲(chǔ)在Redis中的情況
• 將壓縮后的byte[]以二進(jìn)制的形式寫(xiě)入到文件中，請(qǐng)求時(shí)直接在body中帶上文件即可，用這種方式可以不損失壓縮效果

Postman測(cè)試Gzip壓縮數(shù)據(jù)請(qǐng)求：

• 請(qǐng)求頭指定數(shù)據(jù)壓縮方式：

• Body帶上壓縮后的byte[]寫(xiě)入的二進(jìn)制文件

• 執(zhí)行請(qǐng)求，服務(wù)端正確處理了請(qǐng)求并且請(qǐng)求size縮小了將近一半，效果還是很不錯(cuò)的，這樣GZIP壓縮數(shù)據(jù)的請(qǐng)求的處理就完成了，完整的項(xiàng)目代碼在下方??

4. Demo地址

• https://gitee.com/wx_1bceb446a4/gziptest