How to Get JSON from a URL in Python
Being able to retrieve data from remote servers is a fundamental requirement for most projects in web development. JSON is probably one of the most popular formats for data exchange due to its lightweight and easy to understand structure that is fairly easy to parse. Python, being a versatile language, offers a variety of ways to fetch JSON data from a URL in your web project.
In this article, we’ll explore how to use Python to retrieve JSON data from a URL. We’ll cover two popular libraries — requests and urllib , and show how to extract and parse the JSON data using Python’s built-in json module. Additionally, we’ll discuss common errors that may occur when fetching JSON data, and how to handle them in your code.
Using the requests Library
One popular library for fetching data from URLs in Python is requests . It provides an easy-to-use interface for sending HTTP requests to retrieve data from remote servers. To use requests , you’ll first need to install it by using pip in your terminal:
Once we have requests installed, we can use it to fetch JSON data from a URL using the get() method. Say we want to fetch posts from the dummy API called jsonplaceholder.typicode.com/posts :
# Import the requests module import requests # Send a GET request to the desired API URL response = requests.get('https://jsonplaceholder.typicode.com/posts') # Parse the response and print it data = response.json() print(data) We used the get() method to fetch JSON data from the URL https://jsonplaceholder.typicode.com/posts , we extracted the JSON data using the json() method, and printed it to the console. And that’s pretty much it! You will get the JSON response stored as a Python list, with each post represented by one dictionary in that list. For example, one post will be represented as the following dictionary:
But, what if the API request returns an error? Well, we’ll handle that error by checking the status code we got from the API when sending a GET request:
# Import the requests module import requests # Send a GET request to the desired API URL response = requests.get('https://jsonplaceholder.typicode.com/posts') # If everything went well, parse the response # and print it if response.status_code == 200: data = response.json() print(data) else: # Print an error message print('Error fetching data') In addition to what we have already done, we checked the status code of the response to ensure that the request was successful. If the status code is 200 , we print the extracted JSON in the same fashion as before, and if the status code is not 200 we are prompting an error message.
Note: The requests library automatically handles decoding JSON responses, so you don’t need to use the json module to parse the response. Instead, you can use the json() method of the response object to extract the JSON data as a Python dictionary or list:
This method will raise a ValueError if the response body does not contain valid JSON.
Using the urllib Library
Python’s built-in urllib library provides a simple way to fetch data from URLs. To fetch JSON data from a URL, you can use the urllib.request.urlopen() method:
# Import the required modules import json from urllib.request import urlopen # Open the URL that contains JSON response = urlopen('https://jsonplaceholder.typicode.com/posts') if response.getcode() == 200: # Parse JSON in Python data = json.loads(response.read().decode('utf-8')) # Print the title of each post for post in data: print(post['title']) else: print('Error fetching data') After fetching the JSON from the API URL of choice, we checked the status code of the response to ensure that the request was successful. If the status code is 200 , we extract the JSON data using the json.loads() method and print the title of each post.
It’s worth noting that urllib does not automatically decode response bodies, so we need to use the decode() method to decode the response into a string. We then use the json.loads() method to parse the JSON data:
data = json.loads(response.read().decode('utf-8')) Advice: If you want to know more about parsing JSON objects in Python, you should definitely read our guide Reading and Writing JSON to a File in Python».
Using the aiohttp Library
In addition to urllib and requests , there is another library that is commonly used for making HTTP requests in Python — aiohttp . It’s an asynchronous HTTP client/server library for Python that allows for more efficient and faster requests by using asyncio .
To use aiohttp , you’ll need to install it using pip :
Once installed, you can start using it. Let’s fetch JSON data from a URL using the aiohttp library:
Free eBook: Git Essentials
Check out our hands-on, practical guide to learning Git, with best-practices, industry-accepted standards, and included cheat sheet. Stop Googling Git commands and actually learn it!
# Import the required modules import aiohttp import asyncio import json # Define async function that fetches JSON from a desired URL async def fetch_json(url): async with aiohttp.ClientSession() as session: async with session.get(url) as response: data = await response.json() return data async def main(): url = 'https://jsonplaceholder.typicode.com/posts' data = await fetch_json(url) print(json.dumps(data, indent=4)) asyncio.run(main()) We defined an async function fetch_json that takes a URL as input and uses aiohttp to make an HTTP GET request to that URL. We then used the response.json() method to convert the response data to a Python object.
We also defined an async function main that simply calls fetch_json with a URL and prints the resulting JSON data.
Finally, we used the asyncio.run() function to run the main function and fetch the JSON data asynchronously.
Overall, aiohttp can be a great choice for applications that need to make a large number of HTTP requests or require faster response times. However, it may have a steeper learning curve compared to urllib and requests due to its asynchronous nature and the use of asyncio .
Which Library to Choose?
When choosing a library for getting JSON data from a URL in Python, the decision often comes down to the specific needs of your project. Here are some general guidelines to consider:
- For simple requests or legacy code: If you’re making simple requests or working with legacy code, urllib may be a good choice due to its built-in nature and compatibility with older Python versions.
- For ease of use: If ease of use and simplicity are a priority, requests is often the preferred choice. It has a user-friendly syntax and offers many useful features that make it easy to fetch JSON data from a URL.
- For high-performance and scalability: If your application needs to make a large number of HTTP requests or requires faster response times, aiohttp may be the best choice. It offers asynchronous request handling and is optimized for performance.
- For compatibility with other asyncio -based code: If you’re already using asyncio in your project or if you need compatibility with other asyncio -based code, aiohttp may be the best choice due to its built-in support for asyncio .
Conclusion
Getting JSON data from a URL is a common task in Python, and there are several libraries available for this purpose. In this article, we have explored three popular libraries for making HTTP requests: urllib , requests , and aiohttp .
urllib is built-in and suitable for simpler requests or legacy code, while requests offers a more user-friendly and robust interface. aiohttp is optimized for high-performance asynchronous apps and scalability, and is particularly useful for applications that need to make a large number of HTTP requests or require faster response times.