Asynchronous programming is a feature of modern programming languages that allows applications to perform multiple operations without waiting for the others. Using aiohttp allows you to create multiple network connections and move on to the next one instead of waiting for the previous connection's response. This tutorial shows you how to use async code, updated for Python 3.10+.
When to use async requests?
Asynchronous requests are best used if you have multiple requests to run and they do not depend on each other. Also, if you have non I/O blocking operations and do not need the result of the request immediately, you can use asynchronous requests. Asynchronous code does not block other code from running
asyncio and aiohttp Python libraries
asyncio is a Python library that includes support for running and managing coroutines. A coroutine is a Python generator function that can put a pause on its execution before reaching the return statement. It can also pass control to another coroutine. asyncio uses the async / await syntax. You can read more about asyncio here.
For Python to make asynchronous HTTP calls, we will use asyncio with aiohttp for this article.
By default, asyncio and aiohttp are not installed. To install them using pip, run this:
pip install asyncio
pip install aiohttp
Making HTTP calls to get 100 random catfacts using requests
There are a number of free API resources on the Internet. Cat Facts API at https://catfact.ninja/fact is a good resource. Please do not overload the server or you will be blocked or get 429 timeout errors.
We will first write this program using the synchronous requests module. We will time the execution to see how many seconds have elapsed.
import requests
import time
def print_cat_facts():
for number in range(100):
url = f'https://catfact.ninja/fact'
response = requests.get(url)
cat_fact = response.json()
print(cat_fact['fact'])
if __name__ == '__main__':
start_time = time.perf_counter()
print_cat_facts()
end_time = time.perf_counter()
print(f'Total Time: {round(end_time - start_time, 5)} seconds.')
When you run this program, it makes 100 synchronous calls to the API, parses the received JSON string and prints the cat facts. This is the last line:
Total Time: 25.56755 seconds.
Making HTTP calls to get 100 random catfacts using aiohttp
Now, let us use async / aiohttp code to get the same information asynchronously. This code works for Python 3.10 and greater (the latest current version is 3.12.1).
import asyncio
import time
from aiohttp import ClientSession, ClientResponseError
async def fetch_url_data(session, url):
try:
async with session.get(url, timeout=60) as response:
resp = await response.read()
except Exception as e:
print(e)
else:
return resp
return
async def fetch_page(session, url):
async with session.get(url) as response:
return await response.json()
async def main():
async with ClientSession() as session:
tasks = [asyncio.ensure_future(fetch_page(session, url)) for url in urls]
responses = await asyncio.gather(*tasks)
for response in responses:
print(response['fact'])
if __name__ == '__main__':
start_time = time.perf_counter()
urls = [f'https://catfact.ninja/fact' for _ in range(100)]
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(main())
end_time = time.perf_counter()
print(f'Total Time with async: {round(end_time - start_time, 5)} seconds.')
When we run this rogram using async / aiohttp, it makes the same 100 HTTP calls to the same API, but asynchronously. Each HTTP call is independent of the other, and there is no I/O blocking or waiting for the other call to complete.
This program prints the random 100 cat facts. The execution time, however, is greatly reduced!
This is the last line:
Total Time with async: 2.56992 seconds.
You may or may not get the same output, but asynchronous code is definitely way faster than synchronous IO-blocking code.
DeprecationWarning: There is no current event loop [warning]
Sometimes, you may have an older codebase and keep getting this warning:
DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
You are probably using legacy code, written for Python 3.9 and previous versions. If your code looks like this:
import asyncio
import time
from aiohttp import ClientSession, ClientResponseError
async def fetch_url_data(session, url):
try:
async with session.get(url, timeout=60) as response:
resp = await response.read()
except Exception as e:
print(e)
else:
return resp
return
async def fetch_page(session, url):
async with session.get(url) as response:
return await response.json()
async def main():
async with ClientSession() as session:
tasks = [asyncio.ensure_future(fetch_page(session, url)) for url in urls]
responses = await asyncio.gather(*tasks)
for response in responses:
print(response['fact'])
if __name__ == '__main__':
start_time = time.perf_counter()
urls = [f'https://catfact.ninja/fact' for _ in range(100)]
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
end_time = time.perf_counter()
print(f'Total Time with async: {round(end_time - start_time, 5)} seconds.')
This DeprecationWarning message is a warning for deprecated features. When you call asyncio.get_event_loop() without a running event loop, the warning is triggered. It worked fine until Python 3.9, but will keep showing up in 3.10 and above.
To update your code, you first have to create the new event loop object, and then set the event loop with the loop object you just created.
This would be changed:
loop = asyncio.get_event_loop()
to this:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
Now, when you run the updated code, it will run flawlessly without any warnings, unless you have other issues with your code.
Errors in the output
This should probably be due to the API server temporarily or permanently blocking your IP for abuse of resources.
Please first read the updated terms and conditions of use.
Make your API requests reasonably. Do not make too many requests per second or within a short interval of time. It is a good habit to wait for a few seconds or minutes between running asynchronous programs.
The most common HTTP error code you will get is 429, for Too Many Requests. Some API servers can also wrongly return 401 for Unauthorized Error or 403 for Forbidden Error.
If your program has crashed the API server, you may get a 503 for Service Unavailable Error. It is also possible for a really annoyed API server to return 404 Not Found error. Error codes do not always have to mean what they say, but all are possible.
Conclusion
This is the first blog post in a series of concurrent programming with Python. We have more coming up.
Related Posts
If you have any questions, please contact me at arulbOsutkNiqlzziyties@gNqmaizl.bkcom. You can also post questions in our Facebook group. Thank you.