Python read line large file

Read a file line by line in Python

In this Python tutorial, we will learn, how to read a file line by line in Python with a few examples.

There are five methods to read a file line by line in Python, which are shown below:

• Using for loop
• Using readline()
• Using readlines() method
• Using a For Loop and List Comprehension
• Using the iter() with the next() Function

Python read a file line by line

Reading a file line by line is a common task in many Python programs, especially those that deal with large files. There are several methods that can be used to read a file line by line in Python.

Method-1: Python read a file line by line using for loop

This is the simplest method to read a file line by line in Python. It uses a for loop to iterate through the file object, which automatically reads the file one line at a time. This method is easy to understand and use, and it’s a good choice if you just need to process each line of the file one by one.

Open any text editor and write the lines as shown in the above picture and save the file with an extension (.txt). Now use the below code to read the file line by line.

# Open a file named 'python.txt' using a 'with' statement to ensure it's properly closed with open('python.txt') as f: # Iterate over each line in the file and print it to the console for line in f: print(line) 

The above code opens a file named ‘python.txt’ and reads its content line by line.

• It does so using the built-in ‘open’ function in Python, with a ‘with’ statement to ensure that the file is properly closed when the block is exited.
• The code then iterates over each line in the file using a ‘for’ loop and prints each line to the console using the ‘print’ function.

Method-2: Python read a file line by line using readline()

This method reads each line of the file using the readline() method, which returns a single line of the file at a time. The method then uses a while loop to process each line until the readline() method returns an empty string, indicating the end of the file.

# Open a file named 'python.txt' in read mode using a 'with' statement with open('python.txt', 'r') as file: # Read the first line of the file line = file.readline() # Continue reading the file line by line using a 'while' loop while line: line = file.readline() # Print each line to the console print(line) 

The above code opens a file named ‘python.txt’ in read mode and reads its content line by line.

• It does so using the built-in ‘open’ function in Python, with a ‘with’ statement to ensure that the file is properly closed when the block is exited.
• The code then reads the first line of the file using the ‘readline’ method and enters a ‘while’ loop that continues to read the file line by line until it reaches the end. For each line, the code prints it to the console using the ‘print’ function.

Method-3: Python read a file line by line using readlines() method

This method reads the entire file into a list of strings using the readlines() method.

# Define a list of strings Line = ["Welcome\n","to\n","Pythonguides\n"] # Open a file named 'line.txt' in write mode file = open('line.txt', 'w') # Write the strings from the list to the file using the 'writelines' method file.writelines(Line) # Close the file file.close() # Re-open the file in read mode file = open('line.txt', 'r') # Read the contents of the file into a list of strings Lines = file.readlines() # Iterate over the list of strings and print each one to the console for line in Lines: print(line) 

The above code creates a list of strings and writes its contents to a file named ‘line.txt’ using the built-in ‘open’ function in Python.

• It writes the strings from the list to the file using the ‘writelines’ method and then closes the file.
• The code then reopens the file in read mode, reads its contents into a list of strings using the ‘readlines’ method, and iterates over the list to print each line to the console using the ‘print’ function.

Method-4: Python read a file line by line using a For Loop and List Comprehension

This method reads the entire file into a list of stripped strings using a combination of a for loop and list comprehension. The for loop is used to iterate through the file object and the list comprehension is used to create a new list of stripped lines.

# Open a file named 'python.txt' in read mode using a 'with' statement with open('python.txt', 'r') as file: # Read the contents of the file into a list of strings using a list comprehension lines = [line.strip() for line in file] 

The above code opens a file named ‘python.txt’ in read mode using a ‘with’ statement.

• It then reads the contents of the file into a list of strings using list comprehension. The ‘strip’ method is used to remove any leading or trailing whitespace from each line in the file.
• The resulting list contains one string element for each line in the file, with the strings separated by ‘\n’ characters.

Method-5: Python read a file line by line using the iter() with the next() Function

This method uses the iter() function to create an iterator object from the file object and then uses the next() function to read each line of the file one at a time. You can use a while loop to read each line of the file until the end of the file is reached.

# Open a file named 'python.txt' in read mode using a 'with' statement to ensure it's properly closed with open('python.txt', 'r') as file: # Create an iterator over the file's lines lines = iter(file) # Loop over the iterator until there are no more lines to read while True: try: # Get the next line from the iterator and print it after removing any leading or trailing whitespace line = next(lines) print(line.strip()) except StopIteration: # Exit the loop when there are no more lines to read break 

The above code opens a file in read mode, creates an iterator over the file’s lines, and loops over the iterator to read and print each line of the file after removing any leading or trailing whitespace. The ‘with’ statement ensures that the file is properly closed when the block is exited.

You may also like to read the following Python tutorials.

Conclusion

In this tutorial, we have learned about Python read a file line by line example, and also we have covered these methods:

• Using for loop
• Using readline()
• Using readlines() method
• Using a For Loop and List Comprehension
• Using the iter() with the next() Function

I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.

Источник

How to Read Large Text Files in Python

While we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial.

Python File object provides various ways to read a text file. The popular way is to use the readlines() method that returns a list of all the lines in the file. However, it’s not suitable to read a large text file because the whole file content will be loaded into the memory.

Reading Large Text Files in Python

We can use the file object as an iterator. The iterator will return each line one by one, which can be processed. This will not read the whole file into memory and it’s suitable to read large files in Python. Here is the code snippet to read large file in Python by treating it as an iterator.

import resource import os file_name = "/Users/pankaj/abcdef.txt" print(f'File Size is MB') txt_file = open(file_name) count = 0 for line in txt_file: # we can process file line by line here, for simplicity I am taking count of lines count += 1 txt_file.close() print(f'Number of Lines in the file is ') print('Peak Memory Usage =', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) print('User Mode Time =', resource.getrusage(resource.RUSAGE_SELF).ru_utime) print('System Mode Time =', resource.getrusage(resource.RUSAGE_SELF).ru_stime) 

File Size is 257.4920654296875 MB Number of Lines in the file is 60000000 Peak Memory Usage = 5840896 User Mode Time = 11.46692 System Mode Time = 0.09655899999999999 

• I am using os module to print the size of the file.
• The resource module is used to check the memory and CPU time usage of the program.

We can also use with statement to open the file. In this case, we don’t have to explicitly close the file object.

with open(file_name) as txt_file: for line in txt_file: # process the line pass 

What if the Large File doesn’t have lines?

The above code will work great when the large file content is divided into many lines. But, if there is a large amount of data in a single line then it will use a lot of memory. In that case, we can read the file content into a buffer and process it.

with open(file_name) as f: while True: data = f.read(1024) if not data: break print(data) 

The above code will read file data into a buffer of 1024 bytes. Then we are printing it to the console. When the whole file is read, the data will become empty and the break statement will terminate the while loop. This method is also useful in reading a binary file such as images, PDF, word documents, etc. Here is a simple code snippet to make a copy of the file.

with open(destination_file_name, 'w') as out_file: with open(source_file_name) as in_file: for line in in_file: out_file.write(line) 

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Источник

How to read large text files in Python?

In this article, we will try to understand how to read a large text file using the fastest way, with less memory usage using Python.

To read large text files in Python, we can use the file object as an iterator to iterate over the file and perform the required task. Since the iterator just iterates over the entire file and does not require any additional data structure for data storage, the memory consumed is less comparatively. Also, the iterator does not perform expensive operations like appending hence it is time-efficient as well. Files are iterable in Python hence it is advisable to use iterators.

Problem with readline() method to read large text files

In Python, files are read by using the readlines() method. The readlines() method returns a list where each item of the list is a complete sentence in the file. This method is useful when the file size is small. Since readlines() method appends each line to the list and then returns the entire list it will be time-consuming if the file size is extremely large say in GB. Also, the list will consume a large chunk of the memory which can cause memory leakage if sufficient memory is unavailable.

Read large text files in Python using iterate

In this method, we will import fileinput module. The input() method of fileinput module can be used to read large files. This method takes a list of filenames and if no parameter is passed it accepts input from the stdin, and returns an iterator that returns individual lines from the text file being scanned.

Note: We will also use it to calculate the time taken to read the file using Python time.

Источник