Convert Bytes to String in Python: A Tutorial for Beginners

Image by Author

In Python, strings are immutable sequences of characters that are human-readable and typically encoded in a specific character encoding, such as UTF-8. While bytes represent raw binary data. A byte object is immutable and consists of an array of bytes (8-bit values). In Python 3, string literals are Unicode by default, while byte literals are prefixed with a b.

Converting bytes to strings is a common task in Python, particularly when working with data from network operations, file I/O, or responses from certain APIs. This is a tutorial on how to convert bytes to strings in Python.

1. Convert Bytes to String Using the decode() Method

The most straightforward way to convert bytes to a string is using the decode() method on the byte object (or the byte string). This method requires specifying the character encoding used.

Note: Strings do not have an associated binary encoding and bytes do not have an associated text encoding. To convert bytes to string, you can use the decode() method on the bytes object. And to convert string to bytes, you can use the encode() method on the string. In either case, specify the encoding to be used.

Example 1: UTF-8 Encoding

Here we convert byte_data to a UTF-8-encoded string using the decode() method:

<code># Sample byte object
byte_data = b'Hello, World!'

# Converting bytes to string 
string_data = byte_data.decode('utf-8')

print(string_data)  </code>

You should get the following output:

You can verify the data types before and after the conversion like so:

<code>print(type(bytes_data))
print(type(string_data))
</code>

The data types should be as expected:

<code>Output >>>
<class 'bytes'>
<class 'str'></code>

Example 2: Handling Other Encodings

Sometimes, the bytes sequence may contain encodings other than UTF-8. You can handle this by specifying the corresponding encoding scheme used when you call the decode() method on the bytes object.

Here’s how you can decode a byte string with UTF-16 encoding:

<code># Sample byte object 
byte_data_utf16 = b'\xff\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00W\x00o\x00r\x00l\x00d\x00!\x00'

# Converting bytes to string 
string_data_utf16 = byte_data_utf16.decode('utf-16')

print(string_data_utf16)  
</code>

And here’s the output:

Using Chardet to Detect Encoding

In practice, you may not always know the encoding scheme used. And mismatched encodings can lead to errors or garbled text. So how do you get around this?

You can use the chardet library (install chardet using pip: pip install chardet) to detect the encoding. And then use it in the `decode()` method call. Here’s an example:

<code>import chardet

# Sample byte object with unknown encoding
byte_data_unknown = b'\xe4\xbd\xa0\xe5\xa5\xbd'

# Detecting the encoding
detected_encoding = chardet.detect(byte_data_unknown)
encoding = detected_encoding['encoding']
print(encoding)

# Converting bytes to string using detected encoding
string_data_unknown = byte_data_unknown.decode(encoding)

print(string_data_unknown) 
</code>

You should get a similar output:

Error Handling in Decoding

The bytes object that you’re working with may not always be valid; it may sometimes contain invalid sequences for the specified encoding. This will lead to errors.

Here, byte_data_invalid
contains the invalid sequence \xff:

<code># Sample byte object with invalid sequence for UTF-8
byte_data_invalid = b'Hello, World!\xff'

# try converting bytes to string 
string_data = byte_data_invalid.decode('utf-8')

print(string_data) 
</code>

When you try to decode it, you’ll get the following error:

<code>Traceback (most recent call last):
  File "/home/balapriya/bytes2str/main.py", line 5, in <module>
	string_data = byte_data_invalid.decode('utf-8')
              	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 13: invalid start byte
</module></code>

But there are a couple of ways you can handle these errors. You can ignore such errors when decoding or you can replace invalid sequences with a placeholder.

Ignoring Errors

To ignore invalid sequences when decoding, you can set the errors you can set errors to ignore in the decode() method call:

<code># Sample byte object with invalid sequence for UTF-8
byte_data_invalid = b'Hello, World!\xff'

# Converting bytes to string while ignoring errors
string_data = byte_data_invalid.decode('utf-8', errors="ignore")

print(string_data) 
</code>

You’ll now get the following output without any errors:

Replacing Errors

You can as well replace invalid sequences with the placeholder. To do this, you can set errors to replace as shown:

<code># Sample byte object with invalid sequence for UTF-8
byte_data_invalid = b'Hello, World!\xff'

# Converting bytes to string while replacing errors with a placeholder
string_data_replace = byte_data_invalid.decode('utf-8', errors="replace")

print(string_data_replace)  
</code>

Now the invalid sequence (at the end) is replaced by a placeholder:

<code>Output >>>
Hello, World!�
</code>

2. Convert Bytes to String Using the str() Constructor

The decode() method is the most common way to convert bytes to string. But you can also use the str() constructor to get a string from a bytes object. You can pass in the encoding scheme to str() like so:

<code># Sample byte object
byte_data = b'Hello, World!'

# Converting bytes to string
string_data = str(byte_data,'utf-8')

print(string_data)
</code>

This outputs:

3. Convert Bytes to String Using the Codecs Module

Yet another method to convert bytes to string in Python is using the decode() function from the built-in codecs module. This module provides convenience functions for encoding and decoding.

You can call the decode() function with the bytes object and the encoding scheme as shown:

<code>import codecs

# Sample byte object
byte_data = b'Hello, World!'

# Converting bytes to string
string_data = codecs.decode(byte_data,'utf-8')

print(string_data)  
</code>

As expected, this also outputs:

Summary

In this tutorial, we learned how to convert bytes to strings in Python while also handling different encodings and potential errors gracefully. Specifically, we learned how to:

• Use the decode() method to convert bytes to a string, specifying the correct encoding.
• Handle potential decoding errors using the errors parameter with options like ignore or replace.
• Use the str() constructor to convert a valid bytes object to a string.
• Use the decode() function from the codecs module that is built into the Python standard library to convert a valid bytes object to a string.

Happy coding!

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she’s working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more. Bala also creates engaging resource overviews and coding tutorials.