Python change default encoding

Python: Use the UTF-8 mode on Windows!

Summary: Set the PYTHONUTF8=1 environment variable. On macOS and Linux, UTF-8 is the standard encoding already.
But Windows still uses legacy encoding (e.g. cp1252, cp932, etc. ) as system encoding. Python works very well about file names and console IO (e.g. use ~W APIs). But the legacy system encoding is used for the default encoding of text files and pipes. It is a very common mistake that omits the encoding=»utf-8″ option. Developers who use macOS or Linux doesn’t have any trouble by the mistake. For example, even the packaging tutorial in the packaging.python.org uses this code snippet:

with open("README.md", "r") as fh: long_description = fh.read() 

• The default encoding of Python source code is UTF-8
• UTF-8 is the standard encoding of the Web
• Modern text editors like VS Code use UTF-8 by default. And even the notepad.exe chose UTF-8 for the default encoding!

But it is difficult to change the default encoding of text files because it is backward incompatible change. It will break some legacy applications which relying on the legacy encoding.

But there is good news: Python 3.7 introduced the «UTF-8 mode». (thanks to Victor Stinner!!)

When UTF-8 mode is enabled, Python uses UTF-8 as default encoding for text files instead of system encoding.
You can live in the world «UTF-8 is the default. Other legacy encodings are used only when explicitly specified.» like macOS and Linux.

Источник

PEP 686 – Make UTF-8 mode default

This PEP proposes enabling UTF-8 mode by default.

With this change, Python consistently uses UTF-8 for default encoding of files, stdio, and pipes.

Motivation

UTF-8 becomes de facto standard text encoding.

• The default encoding of Python source files is UTF-8.
• JSON, TOML, YAML use UTF-8.
• Most text editors, including Visual Studio Code and Windows Notepad use UTF-8 by default.
• Most websites and text data on the internet use UTF-8.
• And many other popular programming languages, including Node.js, Go, Rust, and Java uses UTF-8 by default.

Changing the default encoding to UTF-8 makes it easier for Python to interoperate with them.

Additionally, many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify encoding=»utf-8″ when they read text files encoded in UTF-8 (e.g. JSON, TOML, Markdown, and Python source files). Inconsistent default encoding causes many bugs.

Specification

Enable UTF-8 mode by default

Python will enable UTF-8 mode by default from Python 3.15.

Users can still disable UTF-8 mode by setting PYTHONUTF8=0 or -X utf8=0 .

locale.getencoding()

Since UTF-8 mode affects locale.getpreferredencoding(False) , we need an API to get locale encoding regardless of UTF-8 mode.

locale.getencoding() will be added for this purpose. It returns locale encoding too, but ignores UTF-8 mode.

When warn_default_encoding option is specified, locale.getpreferredencoding() will emit EncodingWarning like open() (see also PEP 597).

This API was added in Python 3.11.

Fixing encoding=»locale» option

PEP 597 added the encoding=»locale» option to the TextIOWrapper . This option is used to specify the locale encoding explicitly. TextIOWrapper should use locale encoding when the option is specified, regardless of default text encoding.

But TextIOWrapper uses «UTF-8″ in UTF-8 mode even if encoding=»locale» is specified for now. This behavior is inconsistent with the PEP 597 motivation. It is because we didn’t expect making UTF-8 mode default when Python changes its default text encoding.

This inconsistency should be fixed before making UTF-8 mode default. TextIOWrapper should use locale encoding when encoding=»locale» is passed even in UTF-8 mode.

This issue was fixed in Python 3.11.

Backward Compatibility

Most Unix systems use UTF-8 locale and Python enables UTF-8 mode when its locale is C or POSIX. So this change mostly affects Windows users.

When a Python program depends on the default encoding, this change may cause UnicodeError , mojibake, or even silent data corruption. So this change should be announced loudly.

This is the guideline to fix this backward compatibility issue:

1. Disable UTF-8 mode.
2. Use EncodingWarning (PEP 597) to find every places UTF-8 mode affects.
   • If encoding option is omitted, consider using encoding=»utf-8″ or encoding=»locale» .
   • If locale.getpreferredencoding() is used, consider using «utf-8» or locale.getencoding() .
3. Test the application with UTF-8 mode.

Preceding examples

• Ruby changed the default external_encoding to UTF-8 on Windows in Ruby 3.0 (2020).
• Java changed the default text encoding to UTF-8 in JDK 18. (2022).

Both Ruby and Java have an option for backward compatibility. They don’t provide any warning like PEP 597’s EncodingWarning in Python for use of the default encoding.

Rejected Alternative

Deprecate implicit encoding

Deprecating the use of the default encoding is considered.

But there are many cases that the default encoding is used for reading/writing only ASCII text. Additionally, such warnings are not useful for non-cross platform applications run on Unix.

So forcing users to specify the encoding everywhere is too painful. Emitting a lot of DeprecationWarning will lead users ignore warnings.

PEP 387 requires adding a warning for backward incompatible changes. But it doesn’t require using DeprecationWarning . So using optional EncodingWarning doesn’t violate the PEP 387.

Java also rejected this idea in JEP 400.

Use PYTHONIOENCODING for PIPEs

To ease backward compatibility issue, using PYTHONIOENCODING as the default encoding of PIPEs in the subprocess module is considered.

With this idea, users can use legacy encoding for subprocess.Popen(text=True) even in UTF-8 mode.

But this idea makes “default encoding” complicated. And this idea is also backward incompatible.

So this idea is rejected. Users can disable UTF-8 mode until they replace text=True with encoding=»utf-8″ or encoding=»locale» .

How to teach this

For new users, this change reduces things that need to teach. Users don’t need to learn about text encoding in their first year. They should learn it when they need to use non-UTF-8 text files.

For existing users, see the Backward compatibility section.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

Источник

How do I change the default encoding in Python?

Under Eclipse, run dialog settings (“run configurations”, if I remember correctly); you can choose the default encoding on the common tab. Change it to US-ASCII if you want to have these errors ‘early’ (in other words: in your PyDev environment).

What is the default encoding for Python?

The default encoding in Python-2 is ASCII, don’t change it just to run your code. For python portability, every string in Python(3+) is now Unicode.

Does Python use UTF-8 by default?

1. Python 2 uses str type to store bytes and unicode type to store unicode code points. All strings by default are str type — which is bytes~ And Default encoding is ASCII. Default encoding is UTF-8 instead of ASCII.

How do I change encoding in python 3?

Python String encode() In this tutorial, we will learn about the Python String encode() method with the help of examples. The encode() method returns an encoded version of the given string.

How do you specify encoding in python?

In the source header you can declare: #!/usr/bin/env python # -*- coding: utf-8 -*- …. This declaration is not needed in Python 3 as UTF-8 is the default source encoding (see PEP 3120). In addition, it may be worth verifying that your text editor properly encodes your code in UTF-8.

How do I change text encoding in Python?

How to Convert a String to UTF-8 in Python?

1. string1 = “apple” string2 = “Preeti125” string3 = “12345” string4 = “[email protected]”
2. string. encode(encoding = ‘UTF-8’, errors = ‘strict’)
3. # unicode string string = ‘pythön!’ # default encoding to utf-8 string_utf = string. encode() print(‘The encoded version is:’, string_utf)

What is encoding UTF-8 in Python?

UTF-8 is one of the most commonly used encodings, and Python often defaults to using it. UTF stands for “Unicode Transformation Format”, and the ‘8’ means that 8-bit values are used in the encoding. UTF-8 uses the following rules: If the code point is < 128, it’s represented by the corresponding byte value.

How do I change the encoding of a file in python?

“convert file encoding to utf-8 python” Code Answer

1. with open(ff_name, ‘rb’) as source_file:
2. with open(target_file_name, ‘w+b’) as dest_file:
3. contents = source_file. read()
4. dest_file. write(contents. decode(‘utf-16’). encode(‘utf-8’))

How do I fix encoding in Python?

The best way to attack the problem, as with many things in Python, is to be explicit. That means that every string that your code handles needs to be clearly treated as either Unicode or a byte sequence. The most systematic way to accomplish this is to make your code into a Unicode-only clean room.

How do I change encoding in Python 3?

What does encode () do in Python?

Python String encode() Method The encode() method encodes the string, using the specified encoding. If no encoding is specified, UTF-8 will be used.

What does encoding do normal Python strings use?

What is encoding in Python?

What is a text file in Python?

Источник