Tutorial

Command-line Basics: Network Requests

Updated on August 26, 2021
    author

    joshtronic

    Command-line Basics: Network Requests

    Making network requests from the command-line comes in a variety of shapes and sizes. Some folks swear by curl others love wget. For me, I love the lwp-request package which provides familiarly named GET, POST and HEAD commands. The third of the list, HEAD, is something I use quite regularly to troubleshoot the headers returned by servers.

    Getting started

    The commands we’re going to use are provided by a package called lwp-request.

    The package in question is written in Perl and tends to come standard with quite a few Linux distributions as well as other Unix-like operating systems like macOS.

    The lwp-request command could be invoked directly with a series of arguments, but we’ll be going over the helper scripts that it comes with for the sake of simplicity.

    If you are lacking GET, POST and HEAD on your system, you most likely will be able to install the lwp-request package from your favorite package manager.

    On Ubuntu and Debian derived systems, you can install it using the following commands:

    1. sudo apt update
    2. sudo apt install libwww-perl

    On CentOS, Rocky Linux, Fedora, and other RedHat derived systems, you can install it with support for HTTPS URLs using the following command:

    1. sudo dnf install perl-libwww-perl.noarch perl-LWP-Protocol-https.noarch

    GET

    Let’s say you loved the Alligator.io logo so much, you wanted to download it locally. It’s a pretty amazing logo, who wouldn’t want their own personal copy?

    To GET the file, you can simply run:

    1. GET https://alligator.io/images/logo-fancy.svg

    Not so fast! All that does is show a bunch of SVG markup.

    In true Unix Philosophy fashion, the GET command does one thing really well and that’s GETting a file.

    This is great when you want to check a URL to see what’s being returned by the web server, but if you want to truly download that lovely logo, you’ll want to send the output to a file:

    1. GET https://alligator.io/images/logo-fancy.svg > logo-fancy.svg

    Now we have that fantastic Alligator.io logo downloaded to a local file!

    POST

    The GET command allows us to consume files from remote servers while POST allows us to send data to a server for processing as well as return it’s output.

    At it’s very least, the syntax for POST is the same as GET:

    1. POST https://httpbin.org/post

    This will then prompt you for the content you’d like to POST. The string expected should be in a query string format that looks something like this:

    reptile=alligator&color=#008f68
    

    When you’re done entering in your content, simply hit CTRL-D and the content will be POSTed. The service we’re posting to will mirror back the request:

    {
      "args": {},
      "data": "",
      "files": {},
      "form": {
        "color": "#008f68\n",
        "reptile": "alligator"
      },
      "headers": {
        "Content-Length": "32",
        "Content-Type": "application/x-www-form-urlencoded",
        "Host": "httpbin.org",
        "User-Agent": "lwp-request/6.39 libwww-perl/6.39"
      },
      "json": null,
      "origin": "203.0.113.5",
      "url": "https://httpbin.org/post"
    }
    

    As mentioned, HEAD is not only extremely useful for debugging and troubleshooting, but I’m pretty sure it’s in my top 5 of all-time favorite command-line utilities.

    Similar to GET and POST, the syntax for HEAD is quite minimal:

    1. HEAD http://alligator.io/

    This will return a 200 OK and information about the headers returned by the web service.

    Unfortunately, this isn’t quite right, because we serve up Alligator.io over HTTPS like the good security-minded reptilian webizens that we are.

    The HEAD command by default only gives you information about the last stop in the request chain. To see all of the requests, including the automatic 301 Moved Permanently, pass in the -S argument:

    1. HEAD -S http://alligator.io/

    Which gives us a bit more insight:

    Output
    HEAD http://alligator.io/ 301 Moved Permanently HEAD https://alligator.io/ 200 OK Cache-Control: public, max-age=0, must-revalidate Connection: close Date: Sat, 29 Jun 2019 00:49:18 GMT Age: 1 ETag: "8b85849c835909679fc1ba80b307d144-ssl" Server: Netlify Content-Length: 0 Content-Type: text/html; charset=UTF-8 Client-Date: Sat, 29 Jun 2019 00:49:18 GMT Client-Peer: 203.0.113.1:443 Client-Response-Num: 1 Client-SSL-Cert-Issuer: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 Client-SSL-Cert-Subject: /CN=alligator.io Client-SSL-Cipher: TLS_AES_256_GCM_SHA384 Client-SSL-Socket-Class: IO::Socket::SSL Strict-Transport-Security: max-age=31536000 X-NF-Request-ID: 60babe56-c0ea-4658-aa5a-3e185f1e851f-10342

    Bonus

    Monochromatic output got you down? If so, you could alias HTTPie’s http command to GET, POST and HEAD.

    HTTPie can do everything the lwp-request library does, with similar syntax, with the added bonus of colorful output.

    On Ubuntu and Debian derived systems you can install HTTPie with the following commands:

    1. sudo apt update
    2. sudo apt install httpie

    On Centos, Rocky Linux, Fedora, and RedHat derived distributions you can install HTTPie using the following commands, as long as you have the EPEL :

    1. sudo dnf install epel-release
    2. sudo dnf install httpie

    My local aliases look like this:

    1. # HTTPie aliases
    2. alias GET='http'
    3. alias POST='http POST'
    4. alias HEAD='http HEAD'

    Conclusion

    Next time you need to make a network request to an API or are troubleshooting the headers returned by a server, you can leave Postman and similar tools at the door.

    You can omit your browser entirely, too!

    Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

    Learn more about our products

    About the author(s)

    Category:
    Tutorial
    Tags:

    While we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial.

    Still looking for an answer?

    Ask a questionSearch for more help

    Was this helpful?
     
    Leave a comment
    

    This textbox defaults to using Markdown to format your answer.

    You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

    Join the Tech Talk
    Success! Thank you! Please check your email for further details.

    Please complete your information!

    Become a contributor for community

    Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

    DigitalOcean Documentation

    Full documentation for every DigitalOcean product.

    Resources for startups and SMBs

    The Wave has everything you need to know about building a business, from raising funding to marketing your product.

    Get our newsletter

    Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

    New accounts only. By submitting your email you agree to our Privacy Policy

    The developer cloud

    Scale up as you grow — whether you're running one virtual machine or ten thousand.

    Get started for free

    Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

    *This promotional offer applies to new accounts only.