Python open() Function Explained: How to Open, Read, and Write Files

In the previous tutorial, you used console to take input. Now, we will be taking input using a file. That means, we will read from and write into files. To do so, we need to maintain some steps. Those are-

1. Open a file
2. Take input from that file / write output to that file
3. Close the file

We will also learn some useful operations such as copy file and delete file.

When working with large datasets in machine learning problems, working with files is a basic necessity. Since Python is a majorly used language for data science, you need to be proficient with the different file operations that Python offers.

So, let’s explore some of the Python file operations here.

1. Open a file in Python with the open() function

The first step to working with files in Python is to learn how to open a file. You can open files using the open() method.

The open() function in Python accepts two arguments. The first one is the file name along with the complete path and the second one is the file open mode.

Here is a detailed breakdown of the common reading modes for files:

Primary Modes:

Modifier Modes: These characters can be combined with the primary modes to add functionality.

Combined Modes (with +)

Suppose, we place a text file name file.txt in the same directory where our code is placed.

Now we want to open that file. The open(filename, mode) function returns a file object. With that file object you can proceed your further operation.

#directory:   /home/imtiaz/code.py
text_file = open('file.txt','r')

#Another method using full location
text_file2 = open('/home/imtiaz/file.txt','r')
print('First Method')
print(text_file)

print('Second Method')
print(text_file2)

The output of the following code will be

================== RESTART: /home/imtiaz/code.py ==================
First Method

Second Method

>>>

2. Read and write to files in Python

Python offers various methods to read and write to files where each functions behaves differently. One important thing to note is the file operations mode. To read a file, you need to open the file in the read or write mode. While to write to a file in Python, you need the file to be open in write mode.

Here are some of the functions in Python that allow you to read and write to files:

• read() : This function reads the entire file and returns a string
• readline() : This function reads lines from that file and returns as a string. It fetch the line n, if it is been called nth time.
• readlines() : This function returns a list where each element is single line of that file.
• write() : This function writes a fixed sequence of characters to a file.
• writelines() : This function writes a list of string.

Let’s take an example file “abc.txt”, and read individual lines from the file with a for loop:

#open the file
text_file = open('/Users/pankaj/abc.txt','r')

#get the list of line
line_list = text_file.readlines();

#for each line from the list, print the line
for line in line_list:
    print(line)

text_file.close() #don't forget to close the file

Output:

Now, that we know how to read a file in Python, let’s move ahead and perform a write operation here with the writelines() function.

#open the file
text_file = open('/Users/pankaj/file.txt','w')

#initialize an empty list
word_list= []

#iterate 4 times
for i in range (1, 5):
    print("Please enter data: ")
    line = input() #take input
    word_list.append(line + '\n') #append to the list


text_file.writelines(word_list) #write 4 words to the file

text_file.close() #don’t forget to close the file

Output:

3. Copy files in Python using the shutil() method

We can use the shutil module to copy files in Python. This utility allows us to perform copy and move operations in Python on different files. Let’s work on this with an example:

import shutil

shutil.copy2('/Users/pankaj/abc.txt', '/Users/pankaj/abc_copy2.txt')

#another way to copy file

shutil.copyfile('/Users/pankaj/abc.txt', '/Users/pankaj/abc_copyfile.txt')

print("File Copy Done")

4. Delete files in Python with the os.remove() method

Python’s os module offers the remove() method to delete files from the file system. Let’s take a look at how we can perform a delete operation in Python.

import os

os.remove('/Users/pankaj/abc_copy2.txt')

If you want to delete an entire folder, you can use the shutil.rmtree() method.

import shutil

shutil.remove('/Users/pankaj/test') # this will delete the test folder 
                                    # and all the contents inside

5. Close an open file in Python with the close() method

When you open a file in Python, it’s extremely important to close the file after you make the changes. This saves any changes that you’ve previously made, removes the file from the memory, and prevents any further reads or writes within the program.

Syntax to close an open file in Python:

fileobject.close()

If we continue on from our previous examples where we read files, here’s how you’d close the file:

text_file = open('/Users/pankaj/abc.txt','r')
# some file operations here

text_file.close()

6. Open and close a file using with open()

When it comes to file handling in Python, the with open() statement is the recommended way. You can use with open() to automatically close the file once the block is executed. This way, you don’t have to remember to call the close() method everytime.

with open('text_file.txt', 'r') as f:
  content = f.read()
  print(content)

If an error occurs while you are working with the file, Python will still make sure the file is closed before the program stops. This prevents files from being left open by accident, which can cause bugs.

try:
  with open('log.txt', 'w') as f:
    print('Writing to log file...')
    f.write('Log entry started.\n')
    # Some error occurs
    result = 10 / 0
    f.write('This line will not be written.\n')
except ZeroDivisionError:
  print('An error occurred, but the file was closed automatically!')

In the code above, a ZeroDivisionError occurs. The with statement detects that the block is being exited due to an error, and it still closes the file properly before the except block is handled. If you were using a manual open() without a finally block, the file could have been left open and in a corrupted state.

The with statement also works perfectly with all file modes and options, including specifying encodings and handling binary files. You simply include the parameters in the open() function as you normally would. Also, when working with files like images, audio, or executables, always open them in binary mode. This reads the data as bytes, not text, preventing corruption.

6. Python FileNotFoundError

It’s common to receive the FileNotFoundError when working with files in Python. It can be easily avoided by providing complete file paths when creating the file object.

  File "/Users/pankaj/Desktop/string1.py", line 2, in <module>
    text_file = open('/Users/pankaj/Desktop/abc.txt','r')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/pankaj/Desktop/abc.txt'

To fix the FileNotFoundError, you simply need to verify that the path you’ve mentioned for the file open method is correct.

Handling text data from various sources is a common requirement in software development. One of the most common challenge in this process is the UnicodeDecodeError, which arises from a misunderstanding between how a text file was stored and how it is being read.

A file encoding is a standardized ruleset that defines the mapping between human-readable characters and their corresponding byte representations.

• ASCII: An early, limited encoding standard designed for the English alphabet and basic control characters.
• UTF-8: The current global standard for text encoding. Its comprehensive design can represent nearly every character and symbol from all modern languages, making it the recommended encoding for most applications.
• Legacy Encodings (cp1252, iso-8859-1, etc.): Numerous other encodings exist, often found in files generated by legacy systems or software specific to a particular region (e.g., cp1252 was common on older Windows systems).

When you write text to a file, you are “encoding” characters into bytes. When you read it, you are “decoding” those bytes back into characters. A UnicodeDecodeError occurs when an application attempts to decode a file using a different ruleset than the one used to create it.

When you encounter a file that is not UTF-8 encoded:

• Indentify the correct encoding. Based on the file’s origin, you can make an informed assumption. If you don’t know the origin, a third-party library like chardet can programmatically inspect the raw bytes and suggest the most probable encoding.

• Once the correct encoding has been identified, provide it to the encoding parameter of the open() function. For example:

  try:
    with open('legacy_file.txt', 'r', encoding='utf-8') as f:
      content = f.read()
      # Process the file content
  except (UnicodeDecodeError, FileNotFoundError) as e:
    print(f"An error occurred: {e}")
  

• In some cases, a file may contain byte sequences that are invalid even for the specified encoding. By default, this (errors=‘strict’) will raise an error. The errors parameter provides control over this behavior.

  • errors='replace' substitutes any problematic byte sequence with a replacement character (). It allows the file to be read completely while visually indicating where decoding issues occurred.
  • errors='ignore' will discard any undecodable bytes. It should be used with caution, as it can result in silent data loss.

  Here’s an example using errors='replace' to handle unexpected characters:

  with open('data_with_errors.txt', 'r', encoding='utf-8', errors='replace') as f:
      content = f.read()
      # The program proceeds without crashing, and 'content' will
      # contain '' where invalid byte sequences were found.
  

The pathlib module is Python’s modern system for handling filesystem paths. The difference between pathlib and os.path is that pathlib treats every path as an object, not just a string. This object-oriented approach allows you to call methods and access attributes directly on the path itself. By treating paths as objects, pathlib makes your code more readable, easier to maintain, and less prone to errors across different operating systems like Windows, macOS, and Linux.

A primary advantage of pathlib is using the / operator to build paths, which is simpler than calling os.path.join(). Once you create a Path object, you can access its components directly using attributes. This allows you to get the parent directory, filename, or file extension without using separate functions. This allows for a more fluid and object-oriented style of path manipulation.

from pathlib import Path

config_path = Path.home() / 'Documents' / 'settings.ini'

print(f"Full Path: {config_path}")
print(f"Parent Directory: {config_path.parent}")
print(f"File Name: {config_path.name}")
print(f"File Suffix: {config_path.suffix}")

Path objects also have built-in methods for common filesystem operations. You can check if a file exists with .exists(), read its contents with .read_text(), or write to it with .write_text(). To find files matching a pattern, you can use the .glob() method. This consolidation of functionality makes pathlib an efficient and powerful tool for all file system tasks.

1. How do I check if a file exists in Python?

You can check if a file exists in Python using several methods. The most common and recommended approaches are using the os and pathlib modules.

• Using the os.path.exists() function is a straightforward way to check for the existence of a file or directory.

  import os
  
  if os.path.exists("my_file.txt"):
      print("File exists!")
  else:
      print("File does not exist.")
  

• The pathlib module (recommended for modern Python) offers a more object-oriented approach to filesystem paths.

  from pathlib import Path
  
  file_path = Path("my_file.txt")
  if file_path.is_file():
      print("File exists!")
  else:
      print("File does not exist.")
  

• You can also try to open the file and catch the FileNotFoundError that occurs if it doesn’t exist. This is useful if you intend to read the file immediately after checking for its existence.

  try:
      with open("my_file.txt") as f:
          # File exists and is open for reading
          content = f.read()
          print("File exists and was read.")
  except FileNotFoundError:
    print("File does not exist.")
  

2. What is the difference between ‘w’ and ‘a’ mode?

The 'w' and 'a' modes are both used for writing to files, but they behave differently:

• In 'w' (Write) mode, if the file exists, its contents are truncated (i.e., completely erased) before writing the new content. And if the file does not exist, a new file is created.
• In 'a' (Append) mode, if the file exists, the new data is appended to the end of the file, preserving the existing content. If the file does not exist, a new file is created.

Here’s a simple illustration:

# Using 'w' mode
with open("data.txt", "w") as f:
    f.write("This is the first line.\n")

with open("data.txt", "w") as f:
    f.write("This overwrites the previous content.\n")
# The file "data.txt" will only contain "This overwrites the previous content."

# Using 'a' mode
with open("log.txt", "a") as f:
    f.write("Log entry 1.\n")

with open("log.txt", "a") as f:
    f.write("Log entry 2.\n")
# The file "log.txt" will contain both "Log entry 1." and "Log entry 2."

3. Why use with open() instead of just open()?

Using with open() is the recommended way to work with files in Python because it provides automatic resource management.

When you use with open(), the file is closed automatically when you exit the with block, even if errors occur within the block. This prevents resource leaks, where a file might remain open unintentionally, consuming system resources and potentially leading to issues.

Without with open(), you are responsible for explicitly closing the file using the close() method. Forgetting to do so can lead to problems, especially in larger applications.

4. How do I avoid overwriting a file?

To avoid accidentally overwriting a file, you can use the 'x' mode or check for the file’s existence before opening it in write mode.

This mode will create and write to a new file, but if the file already exists, it will raise a FileExistsError.

try:
    with open("new_file.txt", "x") as f:
        f.write("This is a new file.")
except FileExistsError:
    print("File already exists and was not overwritten.")

You can use the methods described in the first FAQ to check if the file exists before deciding whether to write to it.

import os

if not os.path.exists("my_data.txt"):
    with open("my_data.txt", "w") as f:
        f.write("Initial data.")
else:
    print("File already exists. Not writing to it.")

5. How to open a file in a different encoding?

To open a file with a specific character encoding, use the encoding parameter in the open() function. This is crucial for working with text files that contain non-ASCII characters, as different operating systems may use different default encodings. UTF-8 is a widely used and recommended encoding.

# Writing to a file with UTF-8 encoding
with open("data_utf8.txt", "w", encoding="utf-8") as f:
    f.write("こんにちは世界") # "Hello, World" in Japanese

# Reading a file with a specific encoding
with open("data_utf8.txt", "r", encoding="utf-8") as f:
    content = f.read()
    print(content)

6. Can I open a file for reading and writing?

Yes, you can open a file for both reading and writing by using one of the following modes:

• 'r+': Opens the file for both reading and writing. The file pointer is at the beginning of the file. If the file does not exist, it will raise a FileNotFoundError.
• 'w+': Opens the file for both writing and reading. If the file exists, its contents are truncated. If the file does not exist, a new file is created.
• 'a+': Opens the file for both appending and reading. The file pointer is at the end of the file for writing. If the file does not exist, a new file is created.

with open("read_write_example.txt", "w+") as f:
    f.write("Line 1\n")
    f.write("Line 2\n")
    f.seek(0) # Move the file pointer back to the beginning
    content = f.read()
    print("Content after writing:", content)

7. What’s the best way to read a large file in Python?

Reading a large file into memory all at once can be inefficient and may lead to memory errors. The best way to read a large file is to process it line by line or in chunks.

Iterating over the file object (most common and Pythonic): This is the most memory-efficient way to read a text file line by line.

with open("large_log_file.txt", "r") as f:
    for line in f:
        # Process each line individually
        print(line, end='')

Using a while loop with readline(): This achieves a similar result to the for loop.

with open("large_log_file.txt", "r") as f:
    while True:
        line = f.readline()
        if not line:
            break
        # Process the line
        print(line, end='')

Reading in chunks (for binary files or specific processing): If you need to process the file in fixed-size chunks, you can use the read() method with a specified size.

with open("large_binary_file.bin", "rb") as f:
    chunk_size = 4096 # 4KB
    while True:
        chunk = f.read(chunk_size)
        if not chunk:
            break
        # Process the chunk of data

We have now covered the essentials of file handling in Python. You now know how to open, read, and write to files using various modes. You have also learned how to manage files and directories and handle common errors like FileNotFoundError and UnicodeDecodeError.

As you continue to build projects, remember to use modern best practices. Always prefer the with open() statement to ensure files are closed safely, and use the pathlib module for a cleaner, more reliable way to work with filesystem paths.

For more detailed guides, you can refer to the following tutorials:

• Learn how to handle plain text files in Python to understand the basics of file operations.
• Understand the usage of raw strings in Python for better handling of file paths and regular expressions.
• Explore methods to read large text files in Python efficiently to manage memory usage and performance.

Additionally, here’s an article on how you can use the Pandas module to read CSV datasets in Python.