大文件上傳下載實現(xiàn)思路,分片、斷點續(xù)傳代碼實現(xiàn)
????
關(guān)注后回復(fù)?“進(jìn)群”?，拉你進(jìn)程序員交流群
????
?
大文件上傳
實現(xiàn)思路：
-
分片：按照自定義緩沖區(qū)大小，將大文件分成多個小文件片段。
-
斷點續(xù)傳：根據(jù)分片數(shù)量，給每個小文件通過循環(huán)起對應(yīng)名稱，當(dāng)文件下載中斷在續(xù)傳時，判斷小文件名稱若存在則不存了，此時還需要判斷文件若不是最后一個分片則大小為緩沖區(qū)固定大小，若沒達(dá)到則證明小文件沒傳完需要重新傳輸。
-
合并：下載時通過線程池創(chuàng)建任務(wù)進(jìn)行下載或上傳、當(dāng)判斷最后一個分片傳完時，調(diào)用合并方法，根據(jù)之前定義的文件名稱順序進(jìn)行合并，肯能出現(xiàn)最后一個分片傳完，之前分片未傳完的情況，需要使用while循環(huán)進(jìn)行判斷，多文件未傳輸完，則等待一會繼續(xù)判斷。
-
大文件秒傳：實際上是根據(jù)文件名稱區(qū)一個唯一的md5值存儲，傳文件時進(jìn)行判斷，若存在則不傳。
創(chuàng)建springboot項目，添加依賴
?<dependencies>
????????<dependency>
????????????<groupId>org.springframework.boot</groupId>
????????????<artifactId>spring-boot-starter</artifactId>
????????</dependency>
?
????????<dependency>
????????????<groupId>org.springframework.boot</groupId>
????????????<artifactId>spring-boot-starter-test</artifactId>
????????????<scope>test</scope>
????????</dependency>
????????<dependency>
????????????<groupId>org.springframework.boot</groupId>
????????????<artifactId>spring-boot-autoconfigure</artifactId>
????????</dependency>
????????<dependency>
????????????<groupId>commons-fileupload</groupId>
????????????<artifactId>commons-fileupload</artifactId>
????????????<version>1.3.1</version>
????????</dependency>
????????<dependency>
????????????<groupId>commons-io</groupId>
????????????<artifactId>commons-io</artifactId>
????????????<version>2.4</version>
????????</dependency>
<!--????????做斷點下載使用-->
????????<dependency>
????????????<groupId>org.apache.httpcomponents</groupId>
????????????<artifactId>httpcore</artifactId>
????????</dependency>
????????<dependency>
????????????<groupId>org.apache.httpcomponents</groupId>
????????????<artifactId>httpclient</artifactId>
????????</dependency>
????????<dependency>
????????????<groupId>org.springframework.boot</groupId>
????????????<artifactId>spring-boot-starter-web</artifactId>
????????</dependency>
????</dependencies>
編寫測試環(huán)境看環(huán)境有沒有搭建成功
@Controller
public?class?UploadController?{
????@RequestMapping("/up")
????@ResponseBody
????public?String?upload(HttpServletRequest?request,?HttpServletResponse?response){
????????return?"搭建成功";
????}
}
頁面主要代碼
<body>
<div?id="upload-container">
????<span>上傳</span>
</div>
<div?id="upload-list"></div>
<button?id="picker">點擊上傳</button>
</body>
<script>
????$('#upload-container').click(function?(event){
????????$("#picker").find('input').click();
????});
????var?uploader?=?WebUploader.create({
????????auto:?true,
????????swf?:?'Uploader.swf',?//swf文件路徑
????????server:?'http://localhost:8080/upload',
????????dnd:?'#upload-container',
????????pick:?'#picker',??//內(nèi)部根據(jù)當(dāng)前運行創(chuàng)建
????????multiple:?true,?????//選擇多個
????????chunked:?true,??????//開啟分片
????????threads:?20,????????//并發(fā)數(shù)
????????method:?'POST',
????????fileSizeLimit:?1024*1024*1024*10,?//單個文件大小限制
????????fileSingleSizeLimit:?1024*1024*1024,??//總文件大小
????????fileVal:?'upload'
????});
????uploader.on("beforeFileQueued",function?(file){
????????console.log(file);?//獲取文件后綴
????});
????uploader.on('fileQueued',function?(file){
????????//選中文件要做的事
????????console.log(file.ext);
????????console.log(file.size);
????????console.log(file.name);
????????var?html?=?'<div?class="upload-item"><span>文件名：'+file.name+'</span><span data-file_id="'+file.id+'"class="btn-delete">刪除</span><span data-file_id="'+file.id+'"class="btn-retry">重試</span><div?class="percentage?'+file.id+'"?style="width:?0%;"></div></div>'
????????$('#upload-list').append(html);
????????uploader.md5File(file)??//給文件定義唯一的md5值，當(dāng)再次上傳相同文件時，就不用傳了??大文件秒傳實際上是沒傳，直接拷貝之前文件地址
????????//顯示進(jìn)度
????????.progress(function?(percentage){
????????????console.log('Percentage:',percentage);
????????})
????????//完成
????????.then(function?(val){
????????????console.log('md5?result',val);
????????});
????});
webUpload組件支持分片上傳：利用多進(jìn)程并發(fā)上傳，將大文件拆分成一個一個的小文件，每一個小文件屬于大文件的一個分片
斷點續(xù)傳實現(xiàn)：后端代碼
@Controller
public?class?UploadController?{
????private?final?static?String?utf8?=?"utf-8";
????@RequestMapping("/up")
????@ResponseBody
????public?void?upload(HttpServletRequest?request,?HttpServletResponse?response)?throws?Exception?{
???????response.setCharacterEncoding(utf8);
???????//長傳時候會有多個分片，需要記錄當(dāng)前為那個分片
???????Integer?schunk?=?null;
???????//總分片數(shù)
????????Integer?schunks?=?null;
????????//名字
????????String?name?=?null;
????????//文件目錄
????????String?path?=?"D:\\file";
????????BufferedOutputStream?os?=?null;
????????try?{
????????????//設(shè)置緩沖區(qū)大小??先讀到內(nèi)存里在從內(nèi)存寫
????????????DiskFileItemFactory?factory?=?new?DiskFileItemFactory();
????????????factory.setSizeThreshold(1024);
????????????factory.setRepository(new?File(path));
????????????//解析
????????????ServletFileUpload?upload?=?new?ServletFileUpload(factory);
????????????//設(shè)置單個大小與最大大小
????????????upload.setFileSizeMax(5l*1024l*1024l*1024l);
????????????upload.setSizeMax(10l*1024l*1024l*1024l);
????????????List<FileItem>?items?=?upload.parseRequest(request);
????????????for?(FileItem?item?:?items){
????????????????if?(item.isFormField()){
????????????????????//獲取分片數(shù)賦值給遍量
????????????????????if?("chunk".equals(item.getFieldName())){
????????????????????????schunk?=?Integer.parseInt(item.getString(utf8));
????????????????????}
????????????????????if?("chunks".equals(item.getFieldName())){
????????????????????????schunks?=?Integer.parseInt(item.getString(utf8));
????????????????????}
????????????????????if?("name".equals(item.getFieldName())){
????????????????????????name?=?item.getString(utf8);
????????????????????}
????????????????}
????????????}
????????????//取出文件基本信息后
????????????for?(FileItem?item?:?items){
????????????????if?(!item.isFormField()){
????????????????????//有分片需要臨時目錄
????????????????????String?temFileName?=?name;
????????????????????if?(name?!=?null){
????????????????????????if?(schunk?!=?null){
????????????????????????????temFileName?=?schunk+"_"+name;
????????????????????????}
????????????????????????//判斷文件是否存在
????????????????????????File?temfile?=?new?File(path,?temFileName);
????????????????????????//斷點續(xù)傳??判斷文件是否存在，若存在則不傳
????????????????????????if?(!temfile.exists()){
????????????????????????????item.write(temfile);
????????????????????????}
????????????????????}
????????????????}
????????????}
????????????//文件合并??當(dāng)前分片為最后一個就合并
????????????if?(schunk?!=?null?&&?schunk.intValue()==?schunks.intValue()-1){
????????????????File?tempFile?=?new?File(path,?name);
????????????????os?=?new?BufferedOutputStream(new?FileOutputStream(tempFile));
????????????????//根據(jù)之前命名規(guī)則找到所有分片
????????????????for?(int?i?=?0;?i?<?schunks;?i++)?{
????????????????????File?file?=?new?File(path,?i?+?"_"?+?name);
????????????????????//并發(fā)情況?需要判斷所有??因為可能最后一個分片傳完，之前有的還沒傳完
????????????????????while?(!file.exists()){
????????????????????????//不存在休眠100毫秒后在從新判斷
????????????????????????Thread.sleep(100);
????????????????????}
????????????????????//分片存在??讀入數(shù)組中
????????????????????byte[]?bytes?=?FileUtils.readFileToByteArray(file);
????????????????????os.write(bytes);
????????????????????os.flush();
????????????????????file.delete();
????????????????}
????????????????os.flush();
????????????}
????????????response.getWriter().write("上傳成功");
????????}finally?{
????????????try?{
????????????????if?(os?!=?null){
????????????????????os.close();
????????????????}
????????????}catch?(IOException?e){
????????????????e.printStackTrace();
????????????}
????????}
????}
}
文件分片下載服務(wù)端
@Controller
public?class?DownLoadController?{
????private?final?static?String?utf8?=?"utf-8";
????@RequestMapping("/down")
????public?void?downLoadFile(HttpServletRequest?request,?HttpServletResponse?response)?throws?IOException?{
????????response.setCharacterEncoding(utf8);
????????//定義文件路徑
????????File?file?=?new?File("D:\\File\\a.mp4");
????????InputStream?is?=?null;
????????OutputStream?os?=?null;
????????try?{
????????????//分片下載
????????????long?fSize?=?file.length();//獲取長度
????????????response.setContentType("application/x-download");
????????????String?fileName?=?URLEncoder.encode(file.getName(),utf8);
????????????response.addHeader("Content-Disposition","attachment;filename="+fileName);
????????????//根據(jù)前端傳來的Range??判斷支不支持分片下載
????????????response.setHeader("Accept-Range","bytes");
????????????//獲取文件大小
????????????response.setHeader("fSize",String.valueOf(fSize));
????????????response.setHeader("fName",fileName);
????????????//定義斷點
????????????long?pos?=?0,last?=?fSize-1,sum?=?0;
????????????//判斷前端需不需要分片下載
????????????if?(null?!=?request.getHeader("Range")){
????????????????response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
????????????????String?numRange?=?request.getHeader("Range").replaceAll("bytes=","");
????????????????String[]?strRange?=?numRange.split("-");
????????????????if?(strRange.length?==?2){
????????????????????pos?=?Long.parseLong(strRange[0].trim());
????????????????????last?=?Long.parseLong(strRange[1].trim());
????????????????????//若結(jié)束字節(jié)超出文件大小?取文件大小
????????????????????if?(last>fSize-1){
????????????????????????last?=?fSize-1;
????????????????????}
????????????????}else?{
????????????????????//若只給一個長度??開始位置一直到結(jié)束
????????????????????pos?=?Long.parseLong(numRange.replaceAll("-","").trim());
????????????????}
????????????}
????????????long?rangeLenght?=?last-pos+1;
????????????String?contentRange?=?new?StringBuffer("bytes").append(pos).append("-").append(last).append("/").append(fSize).toString();
????????????response.setHeader("Content-Range",contentRange);
????????????response.setHeader("Content-Lenght",String.valueOf(rangeLenght));
????????????os?=?new?BufferedOutputStream(response.getOutputStream());
????????????is?=?new?BufferedInputStream(new?FileInputStream(file));
????????????is.skip(pos);//跳過已讀的文件
????????????byte[]?buffer?=?new?byte[1024];
????????????int?lenght?=?0;
????????????//相等證明讀完
????????????while?(sum?<?rangeLenght){
????????????????lenght?=?is.read(buffer,0,?(rangeLenght-sum)<=buffer.length??(int)?(rangeLenght?-?sum)?:buffer.length);
????????????????sum?=?sum+lenght;
????????????????os.write(buffer,0,lenght);
?
????????????}
????????????System.out.println("下載完成");
????????}finally?{
????????????if?(is!=?null){
????????????????is.close();
????????????}
????????????if?(os!=null){
????????????????os.close();
????????????}
????????}
????}
}
客戶端分片下載，指定固定文件
@RestController
public?class?DownloadClient?{
????private?final?static?long?per_page?=?1024l*1024l*50l;
????//分片存儲臨時目錄?當(dāng)分片下載完后在目錄中找到文件合并
????private?final?static?String?down_path="D:\\File";
????//多線程下載
????ExecutorService?pool?=??Executors.newFixedThreadPool(10);
????//文件大小?分片數(shù)量?文件名稱
????//使用探測?獲取變量
????//使用多線程分片下載
????//最后一個分片下載完?開始合并
????@RequestMapping("/downloadFile")
????public?String?downloadFile()?throws?IOException?{
????????FileInfo?fileInfo?=?download(0,10,-1,null);
????????if?(fileInfo!=?null){
????????????long?pages?=?fileInfo.fSize/per_page;
????????????for?(int?i?=?0;?i?<=?pages;?i++)?{
????????????????pool.submit(new?Download(i*per_page,(i+1)*per_page-1,i,fileInfo.fName));
????????????}
????????}
?
????????return?"成功";
????}
????class?Download?implements?Runnable{
????????long?start;
????????long?end;
????????long?page;
????????String?fName;
?
????????public?Download(long?start,?long?end,?long?page,?String?fName)?{
????????????this.start?=?start;
????????????this.end?=?end;
????????????this.page?=?page;
????????????this.fName?=?fName;
????????}
?
????????@Override
????????public?void?run()?{
????????????try?{
????????????????FileInfo?fileInfo?=?download(start,end,page,fName);
????????????}?catch?(IOException?e)?{
????????????????e.printStackTrace();
????????????}
????????}
????}
????//返回文件名?跟大小
????private?FileInfo?download(long?start,long?end,long?page,String?fName)?throws?IOException?{
????????//斷點下載?文件存在不需要下載
????????File?file?=?new?File(down_path,?page?+?"-"?+?fName);
????????//探測必須放行?若下載分片只下載一半就鍛煉需要重新下載所以需要判斷文件是否完整
????????if?(file.exists()&&page?!=?-1&&file.length()==per_page){
????????????return?null;
????????}
????????//需要知道??開始-結(jié)束?=?分片大小
????????HttpClient?client?=?HttpClients.createDefault();
????????//httpclient進(jìn)行請求
????????HttpGet?httpGet?=?new?HttpGet("http://127.0.0.1:8080/down");
????????//告訴服務(wù)端做分片下載
????????httpGet.setHeader("Range","bytes="+start+"-"+end);
????????HttpResponse?response?=?client.execute(httpGet);
????????String?fSize?=?response.getFirstHeader("fSize").getValue();
????????fName=?URLDecoder.decode(response.getFirstHeader("fName").getValue(),"utf-8");
????????HttpEntity?entity?=?response.getEntity();//獲取文件流對象
????????InputStream?is?=?entity.getContent();
????????//臨時存儲分片文件
????????FileOutputStream?fos?=?new?FileOutputStream(file);
????????byte[]?buffer?=?new?byte[1024];//定義緩沖區(qū)
????????int?ch;
????????while?((ch?=?is.read(buffer))?!=?-1){
????????????fos.write(buffer,0,ch);
????????}
????????is.close();
????????fos.flush();
????????fos.close();
????????//判斷是不是最后一個分片
????????if?(end-Long.valueOf(fSize)>0){
????????????//合并
????????????try?{
????????????????mergeFile(fName,page);
????????????}?catch?(Exception?e)?{
????????????????e.printStackTrace();
????????????}
????????}
????????return?new?FileInfo(Long.valueOf(fSize),fName);
????}
?
????private?void?mergeFile(String?fName,?long?page)?throws?Exception?{
????????//歸并文件位置
????????File?file?=?new?File(down_path,?fName);
????????BufferedOutputStream?os?=?new?BufferedOutputStream(new?FileOutputStream(file));
????????for?(int?i?=?0;?i?<=?page;?i++)?{
????????????File?tempFile?=?new?File(down_path,?i?+?"-"?+?fName);
????????????//分片沒下載或者沒下載完需要等待
????????????while?(!file.exists()||(i!=page&&tempFile.length()<per_page)){
????????????????Thread.sleep(100);
????????????}
????????????byte[]?bytes?=?FileUtils.readFileToByteArray(tempFile);
????????????os.write(bytes);
????????????os.flush();
????????????tempFile.delete();
????????}
????????File?file1?=?new?File(down_path,?-1?+?"-null");
????????file1.delete();
????????os.flush();
????????os.close();
????}
?
????//使用內(nèi)部類實現(xiàn)
????class?FileInfo{
????????long?fSize;
????????String?fName;
?
????????public?FileInfo(long?fSize,?String?fName)?{
????????????this.fSize?=?fSize;
????????????this.fName?=?fName;
????????}
????}
}
感謝閱讀，希望對你有所幫助?:)?
來源：https://blog.csdn.net/weixin_52210557
-End-
最近有一些小伙伴，讓我?guī)兔φ乙恍?面試題?資料，于是我翻遍了收藏的 5T 資料后，匯總整理出來，可以說是程序員面試必備！所有資料都整理到網(wǎng)盤了，歡迎下載！
面試題】即可獲取
在看點這里
好文分享給更多人↓↓
評論
圖片
表情