Asynchronous Programming with httpx

The relatively new httpx module is an HTTP client for Python and allows both synchronous and asynchronous HTTP requests. It also has support for HTTP/1.1 and HTTP/2. It requires Python 3.6 and above. This article talks briefly about httpx and how to use it for asynchronous programming.

What is httpx?

httpx is a new and modern module and has asynchronous capabilities by using async / await syntax for non-blocking code. It works in tandem with asyncio.

httpx supports HTTP/1.1, HTTP/2 and WebSocket. It includes rich customization features and has a lot of security-related features with built-in HTTPS support, authentication and cookie-handling.

httpx can be used as a replacement to requests on many levels.

How to install the httpx module

We will install httpx with pip:

pip install httpx

Make an HTTP GET request

We will first performa simple GET request to print the IP address. We will use our IP address lookup API at https://api.aruljohn.com/ip

import httpx
response = httpx.get('https://api.aruljohn.com/ip')
print(response.text)

The output will be your IP address.

Make an HTTP HEAD request

If you want it to print the HTTP status (200 for successful, 400-500 for error):

import httpx
response = httpx.head('https://api.aruljohn.com/ip')
print(response.status_code)

OUTPUT:

200

The server returned a 200 error code.

Make a GET request with query parameters

Let us use the API from animechan.xyz to get a random quote by anime title.

This is the endpoint: https://animechan.xyz/api/random/anime and the parameter is title.

To make the request, we use this code:

import httpx 

title = 'naruto'
response = httpx.get('https://animechan.xyz/api/random/anime?title=' + title)
print(response.text)

OUTPUT:

{"anime":"Naruto","character":"Gaara","quote":"If love is just a word, then why does it hurt so much if you realize it isn't there?"}

Make a POST request with query parameters

Let us create a POST request at this free endpoint https://jsonplaceholder.typicode.com/posts which allows GET and POST requests.

The POST request has to include these three:

• title: string
• body: string
• userId: int

To make a POST request:

import httpx 

payload = {"title": "Testing the POST request", "body": "Nothing in the POST body", "userId": 100}

response = httpx.post('https://jsonplaceholder.typicode.com/posts', data=payload)
print(response.text)

OUTPUT:

{
  "title": "Testing the POST request",
  "body": "Nothing in the POST body",
  "userId": "100",
  "id": 101
}

Make an httpx async GET request

Now, let us get into async code with httpx. We will make an async call to the endpoint https://animechan.xyz/api/random to get random Anime info.

import httpx
import asyncio

async def main():
    async with httpx.AsyncClient() as client:
        response = await client.get('https://animechan.xyz/api/random')
        print(response.text)

asyncio.run(main())

OUTPUT:

{"anime":"Demon Slayer","character":"Zenitsu Agatsuma","quote":"I won the battle but lost the war!"}

We retrieved the JSON response asynchronously. This is how it worked.

We first imported the asyncio and httpx modules. The asyncio module helps us write code with concurrency using the async / await syntax [used first by NodeJS], and is great for IO tasks.

We then created a coroutine using async def main(); this helps with non-preemptive (or cooperative) multitasking. In cooperative multitasking, all processes must cooperate for the scheduling scheme to work. The cooperative scheduler starts the processes and lets them return control back to it voluntarily.

We created an asynchronous HTTP client with async with httpx.AsyncClient() as client: and then call the get method with await:

async with httpx.AsyncClient() as client:
    response = await client.get('https://animechan.xyz/api/random')

When it encounters await, the .get() coroutine launches and the program resumes execution, which goes back to the event loop. When it gets the response, the event loop resumes where the coroutine had the await keyword.

Finally, when the asyncio.run(main()) method is called, the run() method starts the event loop and calls the main coroutine.

Make multiple concurrent httpx async GET requests

Let us make multiple concurrent HTTP calls to this endpoint using httpx. In this example, we will retrieve zip code information from this endpoint, where :ZIPCODE is the parameter and a valid US zip code, between 00XXX to 1XXXX (X=1 through 9). Replace :ZIPCODE with the actual zip code.

https://api.zippopotam.us/us/:ZIPCODE

In this case, we will write a program to find the zip code information for all these ten zip codes: 20002, 33162, 33902, 14702, 20159, 55460, '05001', '06001', 82005, 57717. We will make concurrent, parallel calls to the API. Our code will also time the speed, for comparison with synchronous API calls in the next section.

Please do not make more or run this program too frequently out of consideration for the owner of the API.

httpx_async_zip_codes.py

import asyncio
import time
import httpx

# Vars
zip_codes = [20002, 33162, 33902, 14702, 20159, 55460, '05001', '06001', 82005, 57717]

async def fetch_page(url):
    async with httpx.AsyncClient() as client:
        return await client.get(url)

async def main():
    urls = [f'https://api.zippopotam.us/us/{zip}' for zip in zip_codes]
    response = await asyncio.gather(*map(fetch_page, urls))
    results = [r.json() for r in response]
    # Print city, state
    for info in results:
        print(f"{info['post code']} : {info['places'][0]['place name']}, {info['places'][0]['state']}")

if __name__ == '__main__':
    start_time = time.perf_counter()
    asyncio.run(main())
    end_time = time.perf_counter()
    print(f'Total Time with httpx async: {round(end_time - start_time, 5)} seconds.')

OUTPUT:

$ python httpx_async_zip_codes.py 
20002 : Washington, District of Columbia
33162 : Miami, Florida
33902 : Fort Myers, Florida
14702 : Jamestown, New York
20159 : Hamilton, Virginia
55460 : Minneapolis, Minnesota
05001 : White River Junction, Vermont
06001 : Avon, Connecticut
82005 : Fe Warren Afb, Wyoming
57717 : Belle Fourche, South Dakota
Total Time with httpx async: 0.48245 seconds.

These 10 URLs ran independent of each other. The sequence of the output may or may not ordered the same way as in the list urls.

Make multiple httpx synchronous GET requests

Let us make multiple one-by-one synchronous HTTP calls to this endpoint using httpx. In this example, we will retrieve zip code information from the previous zip code endpoint, one after another.

https://api.zippopotam.us/us/:ZIPCODE

We will write a synchronous program to find the zip code information for all these ten zip codes: 20002, 33162, 33902, 14702, 20159, 55460, '05001', '06001', 82005, 57717. We will make concurrent, parallel calls to the API. Our code will time the speed of execution, just to see whether asynchronous or synchronous is faster.

Please do not make more or run this program too frequently out of consideration for the owner of the API.

httpx_sync_zip_codes.py

import asyncio
import time
import httpx

# Vars
zip_codes = [20002, 33162, 33902, 14702, 20159, 55460, '05001', '06001', 82005, 57717]

def main():
    urls = [f'https://api.zippopotam.us/us/{zip}' for zip in zip_codes]
    for url in urls:
        info = httpx.get(url).json()
        # Print city, state
        print(f"{info['post code']} : {info['places'][0]['place name']}, {info['places'][0]['state']}")

if __name__ == '__main__':
    start_time = time.perf_counter()
    main()
    end_time = time.perf_counter()
    print(f'Total Time with httpx sync: {round(end_time - start_time, 5)} seconds.')

OUTPUT:

$ python code/httpx_sync_zip_codes.py 
20002 : Washington, District of Columbia
33162 : Miami, Florida
33902 : Fort Myers, Florida
14702 : Jamestown, New York
20159 : Hamilton, Virginia
55460 : Minneapolis, Minnesota
05001 : White River Junction, Vermont
06001 : Avon, Connecticut
82005 : Fe Warren Afb, Wyoming
57717 : Belle Fourche, South Dakota
Total Time with httpx sync: 1.54338 seconds.

Asynchronous vs Synchronous

The asynchronous code took 0.48245 seconds to run.

The synchronous code took 1.54338 seconds to run.

Where speed is a factor and limits and usage are permissible, it is best to go with asynchronous code. For all other reasons, stick with synchronous.

httpx streaming responses

httpx includes the stream() interface to allow streaming download. In requests, we use streaming=True.

For this example, let us download a ~ 1 GB file with Seascapes Data and Documentation from northeastoceandata.org. The download link is here: https://services.northeastoceandata.org/downloads/Habitat/SeascapesData_Documentation.zip

This httpx code will stream the download to our local storage.

url = 'https://services.northeastoceandata.org/downloads/Habitat/SeascapesData_Documentation.zip'
with open('Seascapes.zip', 'wb') as f:
    with httpx.stream('GET', url) as s:
        for chunk in s.iter_bytes():
            f.write(chunk)

The above code will download the zip file as Seascapes.zip.

Conclusion

This was an introduction to httpx modules. Future articles will include more complex programs with httpx.

Related Posts

If you have any questions, please contact me at arulbOsutkNiqlzziyties@gNqmaizl.bkcom. You can also post questions in our Facebook group. Thank you.