How can I read a large text file line by line using Java?
I need to read a large text file of around 5-6 GB line by line using Java. How can I do this quickly?
@kamaci et. al. This question should not be marked as a duplicate. «Quickly read the last line» is not an alternative, and its debatable whether «Quickest way to read text-file line by line» is. The quickest way to do something is not necessarily the common way. Furthermore, the answers below include code, the most relevant alternative you list does not. This question is useful. It is currently the top google search result for «java read file line by line». Finally, its off putting to arrive at stack overflow and find that 1 in every 2 question is flagged for disposal.
Event though I have been reading comments arguing that SO’s close policy sucks, SO persists in it. It’s such a narrow minded developer perspective to want to avoid redundancy at all costs! Just let it be! The cream will rise to the top and the sh*t will sink to the bottom just fine all by itself. Even though a question may have been asked before (which question isn’t??), that does not mean that a new question may not be able to phrase it better, get better answers, rank higher in search engines etc. Interestingly, this question is now ‘protected’.
After Shog’s edit this is indeed a duplicate of stackoverflow.com/q/5800361/103167 but this one has gotten far more activity.
22 Answers 22
A common pattern is to use
try (BufferedReader br = new BufferedReader(new FileReader(file))) < String line; while ((line = br.readLine()) != null) < // process the line. >> You can read the data faster if you assume there is no character encoding. e.g. ASCII-7 but it won’t make much difference. It is highly likely that what you do with the data will take much longer.
EDIT: A less common pattern to use which avoids the scope of line leaking.
try(BufferedReader br = new BufferedReader(new FileReader(file))) < for(String line; (line = br.readLine()) != null; ) < // process the line. >// line is not visible here. > UPDATE: In Java 8 you can do
try (Stream stream = Files.lines(Paths.get(fileName)))
NOTE: You have to place the Stream in a try-with-resource block to ensure the #close method is called on it, otherwise the underlying file handle is never closed until GC does it much later.
What does this pattern look like with proper exception handling? I note that br.close() throws IOException, which seems surprising — what could happen when closing a file that is opened for read, anyway? FileReader’s constructor might throw a FileNotFound exception.
If I have a 200MB file and it can read at 90MB/s then I expect it to take ~3s? Mine seem to take minutes, with this «slow» way of reading. I am on an SSD so read speeds should not be a problem?
@JiewMeng SO I would suspect something else you are doing is taking time. Can you try just reading the lines of the file and nothing else.
Why not for(String line = br.readLine(); line != null; line = br.readLine()) Btw, in Java 8 you can do try( Stream
@AleksandrDubinsky The problem I have with closures in Java 8 is that it very easily makes the code more complicated to read (as well as being slower) I can see lots of developers overusing it because it is «cool».
The buffer size may be specified, or the default size may be used. The default is large enough for most purposes.
// Open the file FileInputStream fstream = new FileInputStream("textfile.txt"); BufferedReader br = new BufferedReader(new InputStreamReader(fstream)); String strLine; //Read File Line By Line while ((strLine = br.readLine()) != null) < // Print the content on the console System.out.println (strLine); >//Close the input stream fstream.close(); Downvoted for poor quality link. There is a completely pointless DataInputStream , and the wrong stream is closed. Nothing wrong with the Java Tutorial, and no need to cite arbitrary third-party Internet rubbish like this.
Once Java 8 is out (March 2014) you’ll be able to use streams:
try (Stream lines = Files.lines(Paths.get(filename), Charset.defaultCharset())) < lines.forEachOrdered(line ->process(line)); > Printing all the lines in the file:
try (Stream lines = Files.lines(file, Charset.defaultCharset()))
Use StandardCharsets.UTF_8 , use Stream
@steventrouble Take a look at: stackoverflow.com/questions/16635398/… It’s not bad if you pass a short function reference like forEach(this::process) , but it gets ugly if you write blocks of code as lambdas inside forEach() .
@msayag, You’re right, you need forEachOrdered in order to execute in-order. Be aware that you won’t be able to parallelize the stream in that case, although I’ve found that parallelization doesn’t turn on unless the file has thousands of lines.
Here is a sample with full error handling and supporting charset specification for pre-Java 7. With Java 7 you can use try-with-resources syntax, which makes the code cleaner.
If you just want the default charset you can skip the InputStream and use FileReader.
InputStream ins = null; // raw byte-stream Reader r = null; // cooked reader BufferedReader br = null; // buffered for readLine() try < String s; if (true) < String data = "#foobar\t1234\n#xyz\t5678\none\ttwo\n"; ins = new ByteArrayInputStream(data.getBytes()); >else < ins = new FileInputStream("textfile.txt"); >r = new InputStreamReader(ins, "UTF-8"); // leave charset out for default br = new BufferedReader(r); while ((s = br.readLine()) != null) < System.out.println(s); >> catch (Exception e) < System.err.println(e.getMessage()); // handle exception >finally < if (br != null) < try < br.close(); >catch(Throwable t) < /* ensure close happens */ >> if (r != null) < try < r.close(); >catch(Throwable t) < /* ensure close happens */ >> if (ins != null) < try < ins.close(); >catch(Throwable t) < /* ensure close happens */ >> > Here is the Groovy version, with full error handling:
File f = new File("textfile.txt"); f.withReader("UTF-8") < br ->br.eachLine < line ->println line; > > absolutely useless closes. There is zero reason to close every stream. If you close any of those streams you automatically close all other streams.
I documented and tested 10 different ways to read a file in Java and then ran them against each other by making them read in test files from 1KB to 1GB. Here are the fastest 3 file reading methods for reading a 1GB test file.
Note that when running the performance tests I didn’t output anything to the console since that would really slow down the test. I just wanted to test the raw reading speed.
Tested in Java 7, 8, 9. This was overall the fastest method. Reading a 1GB file was consistently just under 1 second.
import java.io..File; import java.io.IOException; import java.nio.file.Files; public class ReadFile_Files_ReadAllBytes < public static void main(String [] pArgs) throws IOException < String fileName = "c:\\temp\\sample-1GB.txt"; File file = new File(fileName); byte [] fileBytes = Files.readAllBytes(file.toPath()); char singleChar; for(byte b : fileBytes) < singleChar = (char) b; System.out.print(singleChar); >> > This was tested successfully in Java 8 and 9 but it won’t work in Java 7 because of the lack of support for lambda expressions. It took about 3.5 seconds to read in a 1GB file which put it in second place as far as reading larger files.
import java.io.File; import java.io.IOException; import java.nio.file.Files; import java.util.stream.Stream; public class ReadFile_Files_Lines < public static void main(String[] pArgs) throws IOException < String fileName = "c:\\temp\\sample-1GB.txt"; File file = new File(fileName); try (Stream linesStream = Files.lines(file.toPath())) < linesStream.forEach(line ->< System.out.println(line); >); > > > Tested to work in Java 7, 8, 9. This took about 4.5 seconds to read in a 1GB test file.
import java.io.BufferedReader; import java.io.FileReader; import java.io.IOException; public class ReadFile_BufferedReader_ReadLine < public static void main(String [] args) throws IOException < String fileName = "c:\\temp\\sample-1GB.txt"; FileReader fileReader = new FileReader(fileName); try (BufferedReader bufferedReader = new BufferedReader(fileReader)) < String line; while((line = bufferedReader.readLine()) != null) < System.out.println(line); >> > You can find the complete rankings for all 10 file reading methods here.
How can I print to the same line?
But these should all be printed to the same line in the terminal instead of a new one. What I mean by that is that each new line should replace the previous, it’s not about using print() instead of println() . How can I do that in Java?
8 Answers 8
Format your string like so:
Note the \r character. It is the so-called carriage return that will move the cursor back to the beginning of the line.
Finally, make sure you use
The carriage return escape sequence \r doesn’t work in Eclipse because of an old bug reported in 2004 and not fixed yet in the last version of Eclipse (4.3.2 Kepler)
@Krige: The same applies with the output pane in Netbeans 8, at least on my Mac. The \r character is interpreted as a newline. VT100 codes and backspaces don’t work correctly either, unfortunately.
@Marco Lackovic: also tried it on eclipse neon 4.6.0, still doesn’t work (interpreted as a new line.)
@MarcoLackovic The bug in Eclipse is fixed (since 2020-03). As pointed out in stackoverflow.com/a/62108112/3427883, you basically need to Window ➺ Preferences ➺ Run/Debug ➺ Console, then check the two check-boxes ‘Interpret ASCII. ‘ and ‘Interpret Carriage Return (\r) as’
In Linux, there is different escape sequences for control terminal. For example, there is special escape sequence for erase whole line: \33[2K and for move cursor to previous line: \33[1A . So all you need is to print this every time you need to refresh the line. Here is the code which prints Line 1 (second variant) :
System.out.println("Line 1 (first variant)"); System.out.print("\33[1A\33[2K"); System.out.println("Line 1 (second variant)"); There are codes for cursor navigation, clearing screen and so on.
I think there are some libraries which helps with it ( ncurses ?).
I’ve written a couple utility classes for automating this task for you; they include colourising the output as well: bitbucket.org/dainkaplan/tempura-utils
First, I’d like to apologize for bringing this question back up, but I felt that it could use another answer.
Derek Schultz is kind of correct. The ‘\b’ character moves the printing cursor one character backwards, allowing you to overwrite the character that was printed there (it does not delete the entire line or even the character that was there unless you print new information on top). The following is an example of a progress bar using Java though it does not follow your format, it shows how to solve the core problem of overwriting characters (this has only been tested in Ubuntu 12.04 with Oracle’s Java 7 on a 32-bit machine, but it should work on all Java systems):
public class BackSpaceCharacterTest < // the exception comes from the use of accessing the main thread public static void main(String[] args) throws InterruptedException < /* Notice the user of print as opposed to println: the '\b' char cannot go over the new line char. */ System.out.print("Start[ ]"); System.out.flush(); // the flush method prints it to the screen // 11 '\b' chars: 1 for the ']', the rest are for the spaces System.out.print("\b\b\b\b\b\b\b\b\b\b\b"); System.out.flush(); Thread.sleep(500); // just to make it easy to see the changes for(int i = 0; i < 10; i++) < System.out.print("."); //overwrites a space System.out.flush(); Thread.sleep(100); >System.out.print("] Done\n"); //overwrites the ']' + adds chars System.out.flush(); > > Printing Java ArrayList contents line by line to console
I’m very new to Java programming, and this is actually part of a problem I need to solve for homework: I am reading the contents of a file line by line as a String into an ArrayList for later processing. I need the program to print out to console the contents of the ArrayList on separate lines, but the output after I run the compiled file prints the first line of the file, then prints the first and second lines together on the next line, then prints the first, second and third lines of the program. My understanding of how this is supposed to work is that the program will take my file, the FileReader and BufferedReader will grab the lines of text in the file as Strings, which are then placed in the ArrayList with each String at a different position in the ArrayList right? Can someone please tell me where in the while loop I’m going wrong? Thanks! Code:
public class ArrayListDemo < public static void main (String[]args) < try < ArrayListdemo= new ArrayList (); FileReader fr= new FileReader("hi.tpl"); BufferedReader reader= new BufferedReader(fr); String line; while ((line=reader.readLine()) !=null) < //Add to ArrayList demo.add(line); System.out.println(demo); >reader.close(); >catch (Exception e) < System.out.println("Error: "+e.getMessage()); System.exit(0); >> > 9 Answers 9
The output you are seeing each time is the result of ArrayList.toString() being printed to standard out. Instead, loop through the ArrayList after you’ve finished reading the contents of the file into the ArrayList:
while ((line=reader.readLine()) !=null) < //Add to ArrayList demo.add(line); >reader.close(); //do fun stuff with the demo ArrayList here for (String s : demo)
This will, however, do both the reading and printing in one loop. You may be required to do the following after the first loop building the array: