Java upload large file

SpringBoot: Large Streaming File Upload Using Apache Commons FileUpload

Am trying to upload a large file using the ‘streaming’ Apache Commons File Upload API. The reason I am using the Apache Commons File Uploader and not the default Spring Multipart uploader is that it fails when we upload very large file sizes (~2GB). I working on a GIS application where such file uploads are pretty common. The full code for my file upload controller is as follows:

@Controller public class FileUploadController < @RequestMapping(value="/upload", method=RequestMethod.POST) public void upload(HttpServletRequest request) < boolean isMultipart = ServletFileUpload.isMultipartContent(request); if (!isMultipart) < // Inform user about invalid request return; >//String filename = request.getParameter("name"); // Create a new file upload handler ServletFileUpload upload = new ServletFileUpload(); // Parse the request try < FileItemIterator iter = upload.getItemIterator(request); while (iter.hasNext()) < FileItemStream item = iter.next(); String name = item.getFieldName(); InputStream stream = item.openStream(); if (item.isFormField()) < System.out.println("Form field " + name + " with value " + Streams.asString(stream) + " detected."); >else < System.out.println("File field " + name + " with file name " + item.getName() + " detected."); // Process the input stream OutputStream out = new FileOutputStream("incoming.gz"); IOUtils.copy(stream, out); stream.close(); out.close(); >> >catch (FileUploadException e)< e.printStackTrace(); >catch (IOException e) < e.printStackTrace(); >> @RequestMapping(value = "/uploader", method = RequestMethod.GET) public ModelAndView uploaderPage() < ModelAndView model = new ModelAndView(); model.setViewName("uploader"); return model; >> 

The trouble is that the getItemIterator(request) always returns an iterator that does not have any items (i.e. iter.hasNext() ) always returns false . My application.properties file is as follows:

spring.datasource.driverClassName=org.postgresql.Driver spring.datasource.url=jdbc:postgresql://localhost:19095/authdb spring.datasource.username=georbis spring.datasource.password=asdf123 logging.level.org.springframework.web=DEBUG spring.jpa.hibernate.ddl-auto=update multipart.maxFileSize: 128000MB multipart.maxRequestSize: 128000MB server.port=19091 

   
File to upload:  
Name:  
 
Press here to upload the file! " value="$" />
   

Источник

Uploading big size file on server?

I need to upload the files on server. It can be done either thru webservice/or UI. I just need to store that file content in DB. File can be of any size upto 2 to 4 GB as well. I am not sure whats the way to upload big size file on server without getting out of memory exception? System Configuration :- 8 GB ,java 7 64 bit processor.

1 Answer 1

I am not sure whats the way to upload big size file on server without getting out of memory exception?

That part is easy. Don’t buffer the entire file in memory. Stream it straight to disk.

(Pseudo code . ignoring exception handling and resource management)

InputStream in = . OutputStrean out = . // the place you want to ultimately store the file byte[] buffer = new byte[8192]; int bytesRead; while ((bytesRead = in.read(buffer)) > 0) < out.write(buffer, 0, bytesRead); >// close streams. 

You seem to be confused about how to get the input stream.

• If you are using the Servlet APIs, then use you can get the request’s input stream using ServletRequest.getInputStream() .
• If you are using different APIs, be specific . and I’ll look into it for you.

The more difficult part is dealing with the various issues to do with uploaded file encoding, encapsulation and so on. For that, the best approach is to look for an existing solution. But that depends on the context in which you are doing the uploads; e.g. what web container you are using, etc.

Источник

how to upload >50gb file using java

We have a requirement to upload a >=50GB file to a server, probably to a file system. To achieve this we can split the file into chunks and merge them after upload.This is what I came up with after long googling. Is there any best way to upload such big files? (keeping in mind performance issues).

The best (performance oriented) solution would be to use the native tools which come with the platform rather than implementing the upload in Java.

2 Answers 2

Using the FTP protocol may prove to have only advantages:

-I’m sure there is a lot of libraries in java (my answer is based on what exists for .NET)

-it has an «append» method which allows you to restart your transfer if it was interrupted (you can first check the size of your partially uploaded file, and then know where to re-start reading to complete the missing part)

The biggest problem/catch with FTP, however, is the higher probability of firewall restrictions. HTTP is almost always accessible. FTP is more restricted.

If you have 50GB files to transfer, I think you can make the extra effort of getting a port opened. FTP is, as of today, a still so widely spread standard that not allowing it is just. Apart from that, FTP also allows binary transfer, while HTTP does not. HTTP would result in significant data size overhead, especially if the transfert has to go through a public pipe.

No disagreement. FTP or SFTP would definitely be the choice for simplicity. However, the OP didn’t specify what the target audience/client is, however, so having a port opened may or not be an option. Was just something to keep in mind when choosing a less accessible protocol.

Источник

Upload Large File in a Spring Boot 2 Application Using Swagger UI

Join the DZone community and get the full member experience.

In this article, we are going to create a sample Spring Boot application for uploading large files using Swagger UI. The API created for uploading large files can receive an HTTP multi-part file upload request.

What We Will Need to Build the Application

• JDK 1.8 or later
• Maven 3.2+
• A favorite text editor or IDE. You can also import this example code straight into your IntelliJ IDEA from GitHub. The GitHub link is provided at the end of the article.
• Spring Boot 2.0.1. RELEASE
• Spring Framework 5.0.5. RELEASE
• Spring Cloud Consul 2.0.0.M5
• Swagger 2.8.0

Consul is used to hold the externalized properties for our Spring Boot application. Read about Consul integration with Spring Boot 2

Step 1: Download Consul Agent. Once the download is complet, go to the specific directory and use the following command to start the agent.

consul agent -dev -config-dir=C:\\consul_0.7.3_windows_amd64

Once the agent is started, the Consul Agent UI can be accessed via this URL: http://localhost:8500/ui/#

Step 2: Now, create your application’s YAML file with configurations under the key-value section of Consul UI (config/spring-boot-file-upload.yml).

The content of this file is :

spring: application: name: spring-boot-file-upload servlet: multipart: enabled: false max-file-size: 10MB server: servlet: contextPath: /spring-boot-file-upload port: 8090 logging: level: ROOT: DEBUG

Now let us revisit the following configurations from the above file:

servlet.multipart.enabled: false servlet.multipart.max-file-size: 10MB

For large files, we cannot use Spring Boot’s default StandardServletMultipartResolver or CommonsMultipartResolver , since the server has limited resources (disk space) or memory for buffering. So we need to disable the default MultipartResolver and define our own MultipartResolver , which is present in the main application class.

We need to provide the Consul information to our Spring Boot application via the bootstrap.yml file, which is present in the resources folder of the project.

spring: application: name: spring-boot-file-upload cloud: consul: host: localhost port: 8500 discovery: tags: spring-boot-file-upload enabled: true config: enabled: true format: files fail-fast: true

Project Structure:

The standard project structure is as follows:

Create an Application Class

To start a Spring Boot MVC application, we first need a starter. Thanks to Spring Boot, everything is auto-configured for you. All you need to get started with this application is the following SpringBoot2FileUpload class.

package com.tuturself; import lombok.extern.slf4j.Slf4j; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.client.discovery.EnableDiscoveryClient; import org.springframework.context.annotation.Bean; import org.springframework.web.multipart.commons.CommonsMultipartResolver; /** * Created by Arpan Das on 4/17/2018. */ @Slf4j @EnableDiscoveryClient @SpringBootApplication @EnableAutoConfiguration public class SpringBoot2FileUpload < public static void main(String[] args) < log.info("SpringBoot2FileUpload Application is Starting. "); try < SpringApplication.run(SpringBoot2FileUpload.class, args); >catch (Exception e) < log.error("Error occurred while starting SpringBoot2FileUpload"); >log.info("SpringBoot2FileUpload Application Started.."); > @Bean(name = "multipartResolver") public CommonsMultipartResolver multipartResolver() < CommonsMultipartResolver multipartResolver = new CommonsMultipartResolver(); multipartResolver.setMaxUploadSize(-1); return multipartResolver; >>

As described earlier, we have defined our multipartResolver in our main application class SpringBoot2FileUpload.java and set the MaxUploadSize property to -1. Now it will read the maximum file size from the spring-boot-file-upload.yml file in Consul, which is set to 10MB.

Create a File Upload Controller

Now let us create the Controller class, FileUploadRestApi.java , where we have defined the REST API to upload the file. The API is documented using Swagger UI.

package com.tuturself.webservice; import io.swagger.annotations.*; import org.apache.commons.io.FileUtils; import org.springframework.http.HttpStatus; import org.springframework.http.MediaType; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestPart; import org.springframework.web.bind.annotation.RestController; import org.springframework.web.multipart.MultipartFile; import java.io.File; import java.io.IOException; import java.util.List; @RestController @RequestMapping("/upload") @Api(value = "Guidelines", description = "Describes the guidelines for " + " Spring boot 2.0.1 for uploading large file using Swagger UI") public class FileUploadRestApi < @PostMapping @ApiOperation(value = "Make a POST request to upload the file", produces = "application/json", consumes = MediaType.MULTIPART_FORM_DATA_VALUE) @ApiResponses(value = < @ApiResponse(code = 200, message = "The POST call is Successful"), @ApiResponse(code = 500, message = "The POST call is Failed"), @ApiResponse(code = 404, message = "The API could not be found") >) public ResponseEntity uploadFile( @ApiParam(name = "file", value = "Select the file to Upload", required = true) @RequestPart("file") MultipartFile file) < try < File testFile = new File("test"); FileUtils.writeByteArrayToFile(testFile, file.getBytes()); Listlines = FileUtils.readLines(testFile); lines.forEach(line -> System.out.println(line)); > catch (IOException e) < e.printStackTrace(); return new ResponseEntity("Failed", HttpStatus.INTERNAL_SERVER_ERROR); > return new ResponseEntity("Done", HttpStatus.OK); > >

Creating the Swagger Configuration Class

We need to configure Swagger UI with our Spring Boot 2 application.

package com.tuturself.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import springfox.documentation.builders.ApiInfoBuilder; import springfox.documentation.builders.PathSelectors; import springfox.documentation.builders.RequestHandlerSelectors; import springfox.documentation.service.ApiInfo; import springfox.documentation.service.Contact; import springfox.documentation.spi.DocumentationType; import springfox.documentation.spring.web.plugins.Docket; import springfox.documentation.swagger2.annotations.EnableSwagger2; /** * @author Arpan Das */ @Configuration @EnableSwagger2 public class SwaggerConfig < @Bean public Docket api() < return new Docket(DocumentationType.SWAGGER_2) .select() .apis(RequestHandlerSelectors.basePackage("com.tuturself.webservice")) .paths(PathSelectors.any()) .build().apiInfo(metaData()) .useDefaultResponseMessages(false); >private ApiInfo metaData() < return new ApiInfoBuilder() .title("Spring Boot 2.0 File Upload example with Consul Integration & Swagger 2.8.0") .description("Upload file Swagger-ui 2.8.0 and Spring Boot 2 Spring Cloud Consul") .version("version 1.0") .contact(new Contact("Tutu'rself", "https://www.tuturself.com", "arpan.kgp@gmail.com")).build(); >>

Now the application is ready. Just check for all the required dependencies in the pom.xml file.

  4.0.0 com.tuturself spring-boot-file-upload 1.0-SNAPSHOT org.springframework.boot spring-boot-starter-parent 2.0.1.RELEASE  1.16.20 5.0.5.RELEASE 2.0.0.M5 2.8.0 UTF-8 UTF-8 1.8    org.springframework.boot spring-boot-starter-web  org.springframework spring-context $ org.springframework.boot spring-boot-starter-test test  org.springframework.cloud spring-cloud-starter-consul-all $ org.springframework.cloud spring-cloud-consul-discovery $   io.springfox springfox-swagger2 $ io.springfox springfox-swagger-ui $   org.projectlombok lombok $ provided  commons-fileupload commons-fileupload 1.3.3   org.apache.commons commons-io 1.3.2    org.apache.maven.plugins maven-compiler-plugin 3.5.1 $ $ $  org.springframework.boot spring-boot-maven-plugin true     src/main/resources true     spring-milestones Spring Milestones http://repo.spring.io/milestone false      org.springframework.cloud spring-cloud-dependencies Finchley.M7 pom import    

Now run the application. The Swagger UI app can be accessed via the URL, http://localhost:8080/spring-boot-consul/swagger-ui.html#

To try the POST API for uploading a file, click on it. Once it is expanded, click on Try Now and then choose a file to upload and Execute.

The project can be downloaded from GitHub: spring-boot-file-upload.

Published at DZone with permission of Arpan Das . See the original article here.

Opinions expressed by DZone contributors are their own.

Источник

Upload large files : Spring Boot

So guys, I was dealing with a problem recently. I was getting OutOfMemoryError when trying to upload and save large files (like 2/3 gbs). I was trying to deal it with HttpServletRequest but didn’t end so well. But after spending some time thinking about the universe, mens style, water pond (road? seriously. ) on dhaka city after a heavy rain and how to make life easier doing absolutely nothing, figured out the nicest way to do it.

We’ll use apache commons IO to copy inputstream (and write) to a file. But it has nothing to do with OutOfMemoryError. It’s just a convenient and simple way to write inputstream to a file

1. Dependency

2. Create a multipart form

 
 
Image
 
 

3. Controller

@RequestMapping(value = "/create", method = RequestMethod.POST) private String create(@RequestParam("image") MultipartFile multipartFile) throws IOException< /*** * Don't do this. * it loads all of the bytes in java heap memory that leads to OutOfMemoryError. We'll use stream instead. **/ // byte[] fileBytes = multipartFile.getBytes(); InputStream fileStream = multipartFile.getInputStream(); File targetFile = new File("src/main/resources/targetFile.tmp"); FileUtils.copyInputStreamToFile(fileStream, targetFile); return "redirect:/?message=Successful!"; >

Nice. Everything should be alright and your file should stay in right place without braking of corrupting your data.

Share this:

Источник