Tutorial

Python HTTP Client Request - GET, POST

Published on August 3, 2022
Default avatar

By Shubham

Python HTTP Client Request - GET, POST

While we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial.

Python HTTP module defines the classes which provide the client-side of the HTTP and HTTPS protocols. In most of the programs, the HTTP module is not directly used and is clubbed with the urllib module to handle URL connections and interaction with HTTP requests. Today we will learn how to use a Python HTTP client to fire HTTP request and then parse response status and get response body data.

Python HTTP Client

In this post on python HTTP module, we will try attempting making connections and making HTTP requests like GET, POST and PUT. Let’s get started.

Making HTTP Connections

We will start with the simplest thing HTTP module can do. We can easily make HTTP connections using this module. Here is a sample program:

import http.client

connection = http.client.HTTPConnection('www.python.org', 80, timeout=10)
print(connection)

Let’s see the output for this program: python http connection In this script, we connected to the URL on Port 80 with a specific timeout.

Python HTTP GET

Now, we will use HTTP client to get a response and a status from a URL. Let’s look at a code snippet:

import http.client

connection = http.client.HTTPSConnection("www.journaldev.com")
connection.request("GET", "/")
response = connection.getresponse()
print("Status: {} and reason: {}".format(response.status, response.reason))

connection.close()

In above script, we used a URL and checked the status with the connection object. Let’s see the output for this program: Remember to close a connection once you’re done with the connection object. Also, notice that we used a HTTPSConnection to establish the connection as the website is served over HTTPS protocol.

Getting SSL: CERTIFICATE_VERIFY_FAILED Error?

When I first executed above program, I got following error related to SSL certificates.

$ python3.6 http_client.py 
Traceback (most recent call last):
  File "http_client.py", line 4, in <module>
    connection.request("GET", "/")
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    context=self, session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in init
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)
$ 

From the output, it was clear that it has to do something with the SSL certificates. But website certificate is fine, so it has to be something with my setup. After some googling, I found that on MacOS, we need to run Install Certificates.command file present in the Python installation directory to fix this issue. Below image shows the output produced by this command execution, it looks like it’s installing latest certificates to be used when making SSL connections. python ssl CERTIFICATE_VERIFY_FAILED fix Note that I got this error on Mac OS. However, on my Ubuntu system, it worked perfectly fine.

Python HTTP Client Ubuntu
Python HTTP Client Ubuntu

Getting the Header list from Response

From the response we receive, the headers usually also contain important information about the type of data sent back from the server and the response status as well. We can get a list of headers from the response object itself. Let’s look at a code snippet which is a little-modified version of the last program:

import http.client
import pprint

connection = http.client.HTTPSConnection("www.journaldev.com")
connection.request("GET", "/")
response = connection.getresponse()
headers = response.getheaders()
pp = pprint.PrettyPrinter(indent=4)
pp.pprint("Headers: {}".format(headers))

Let’s see the output for this program: python http request headers

Python HTTP POST

We can POST data to a URL as well with the HTTP module and get a response back. Here is a sample program:

import http.client
import json

conn = http.client.HTTPSConnection('www.httpbin.org')

headers = {'Content-type': 'application/json'}

foo = {'text': 'Hello HTTP #1 **cool**, and #1!'}
json_data = json.dumps(foo)

conn.request('POST', '/post', json_data, headers)

response = conn.getresponse()
print(response.read().decode())

Let’s see the output for this program: python http post Feel free to use the HTTP Bin library to try more requests.

Python HTTP PUT Request

Of course, we can also perform a PUT request using the HTTP module itself. We will use the last program itself. Let’s look at a code snippet:

import http.client
import json

conn = http.client.HTTPSConnection('www.httpbin.org')

headers = {'Content-type': 'application/json'}

foo = {'text': 'Hello HTTP #1 **cool**, and #1!'}
json_data = json.dumps(foo)


conn.request("PUT", "/put", json_data)
response = conn.getresponse()
print(response.status, response.reason)

Let’s see the output for this program: python http request, python http put example

Conclusion

In this lesson, we studied simple HTTP operations which can be done using http.client. We can also create python http server using SimpleHTTPServer module. Reference: API Doc

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about us


About the authors
Default avatar
Shubham

author

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
JournalDev
DigitalOcean Employee
DigitalOcean Employee badge
September 27, 2021

Hi there, I’m using this library in my python codes, wants to send a POST request with username and password together with JSON payload, but that is not working. Seems like it’s not recognizing the credentials when passed. Let me know if any idea. Thanks

- JoshM

    JournalDev
    DigitalOcean Employee
    DigitalOcean Employee badge
    May 11, 2021

    pip install *** ???

    - ree

      JournalDev
      DigitalOcean Employee
      DigitalOcean Employee badge
      June 24, 2020

      connection = HTTPSConnection(forwarder_host, port=9030, context=context, timeout=30) connection.request(method=“POST”, url=event_request_url, headers=request_headers, body=json.dumps(request_body_dict_events)) response = connection.getresponse() print(response) connection.close() connection.request(method=“POST”, url=event_request_url, headers=request_headers, body=json.dumps(request_body_dict_events)) Even after closing the connection the next request is getting sent without any error.

      - Ameya Marathe

        JournalDev
        DigitalOcean Employee
        DigitalOcean Employee badge
        June 15, 2020

        thanks. this was very helpful

        - gaurav

          JournalDev
          DigitalOcean Employee
          DigitalOcean Employee badge
          December 18, 2019

          It work fine for me, Windows 10 Pro Thank you very much

          - CAGB

            JournalDev
            DigitalOcean Employee
            DigitalOcean Employee badge
            March 12, 2019

            hi, i want to send the data from python to php i am not getting any errors in my python code but results are going to php can please help me and give me a proper idea to solve this problem my python code: import requests data = {‘name’: jhon} r = requests.post(“url.php”, params=data) print(r.text) my php code: $name = htmlspecialchars($_GET[“name”]); echo “name: $name”;

            - kushal

              JournalDev
              DigitalOcean Employee
              DigitalOcean Employee badge
              January 24, 2019

              Can u write a python script for servlet asking user to enter username and password

              - Pallavinoolu

                JournalDev
                DigitalOcean Employee
                DigitalOcean Employee badge
                January 3, 2019

                You just explained that ‘SSL Certificate’ issue only on the Mac system… Could you please post for other platform also… better for ‘linux’ machine.

                - Jai K

                  Try DigitalOcean for free

                  Click below to sign up and get $200 of credit to try our products over 60 days!

                  Sign up

                  Join the Tech Talk
                  Success! Thank you! Please check your email for further details.

                  Please complete your information!

                  Get our biweekly newsletter

                  Sign up for Infrastructure as a Newsletter.

                  Hollie's Hub for Good

                  Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

                  Become a contributor

                  Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

                  Welcome to the developer cloud

                  DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

                  Learn more
                  DigitalOcean Cloud Control Panel