How To Use Makefiles to Automate Repetitive Tasks

Introduction

If you have experience installing software from source code on your Linux server, you have probably come across the make utility. This tool is primarily used to compile and build programs. It allows the source code’s author to lay out the steps required to build that specific project.

Although make was created to automate software compilation, the tool was engineered in a flexible enough fashion that it can be used to automate almost any task that you can accomplish from the command line. In this guide, we will discuss how you can repurpose make to automate repetitive tasks that happen in sequence.

Prerequisites

A Linux environment of any kind will work for this tutorial. Package install instructions are provided for both Ubuntu/Debian Linux and for Red Hat/Rocky Linux.

Installing Make

Most Linux distributions allow you to install a compiler with a single command, but do not provide one by default.

On Ubuntu, you can install a package called build-essential that will provide all the packages needed for a modern, well-supported compiler environment. Update your package sources and install the package with apt:

sudo apt update
sudo apt install build-essential

On Rocky Linux or other Red Hat derivatives, you can install a group of packages called Development Tools to provide the same compiler functionality. Install the packages with dnf:

dnf groups mark install "Development Tools"
dnf groupinstall "Development Tools"

You can verify that a compiler is available by checking for the existence of the make command on your system. In order to do that, use the which command:

which make

Output
/usr/bin/make

Now you have the tools that will allow you to take advantage of make in its usual capacity as well.

Understanding Makefiles

The primary way that the make command receives instructions is through the use of a Makefile.

Makefiles are directory specific, meaning that make will search in the directory where it was called to find these files. We should put the Makefile in the root of whatever the task that we are going to be performing, or where it makes most sense to call the scripts we will write.

Inside the Makefile, we follow a specific format. Make uses the concepts of targets, sources, and commands in this way:

target: source
    command

The alignment and format of this is very important. We will discuss the format and meaning of each of these components here:

Target

The target is a user-specified name to refer to a group of commands. Think of it as similar to a function in a programming language.

A target is aligned on the left-hand column, is a continuous word (no spaces), and ends with a colon (:).

When calling make, we can specify a target by typing:

make target_name

Make will then check the Makefile and execute the commands associated with that target.

Source

Sources are references to files or other targets. They represent prerequisites or dependencies for the target they are associated with.

For instance, you could have a section of your Makefile that looks like this:

target1: target2
    target1_command

target2:
    target2_command

In this example, we could call target1 like this:

make target1

Make would then go to the Makefile and search for the target1 target. It would then check to see if there are any sources specified.

It would find the target2 source dependency and jump temporarily to that target.

From there, it would check if target2 had any sources listed. It does not, so it would proceed to execute target2_command. At this point, make would reach the end of the target2 command list and pass control back to the target1 target. It would then execute target1_command and exit.

Sources can be either files or targets themselves. Make uses file timestamps to see if a file has been changed since its last invocation. If a change to a source file has been made, that target is re-run. Otherwise, it marks the dependency as fulfilled and continues to the next source, or the commands if that was the only source.

The general idea is that by adding sources, we can build a sequential set of dependencies that must be executed before the current target. You can specify more than one source, separated by spaces, after any target. You can begin to see how you can specify elaborate sequences of tasks.

Commands

What gives the make command such flexibility is that the command portion of the syntax is very open-ended. You can specify any command to run under the target. You can add as many commands as needed.

Commands are specified on the line after the target declaration. They are indented by one tab character. Some versions of make are flexible about the way that you indent the command section, but in general, you should stick with a single tab to ensure that make will recognize your intent.

Make considers every indented line under the target definition to be a separate command. You can add as many indented lines and commands as you would like. Make will go through them one at a time.

There are a few things that we can place before commands to tell make to handle them differently:

• -: A dash before a command tells make to not abort if an error is encountered. For instance, this could be useful if you want to execute a command on a file if it is present, and do nothing if it is not.

• @: If you lead a command with the @ symbol, the command call itself will not be printed to standard output. This is mainly used just to clean up the output that make produces.

Additional Features

Some additional features can help you create more complex rule chains in your Makefile.

Variables

Make recognizes variables (or macros), which behave as placeholders for substitution in your makefile. It is best to declare these at the top of your file.

The name of each variable is completely capitalized. Following the name, an equal sign assigns the name to the value on the right side. For instance, if we wanted to define an installation directory as /usr/bin, we could add INSTALLDIR=/usr/bin at the top of the file.

Later in the file, we can reference this location by using the $(INSTALLDIR) syntax.

Escape Newlines

Another useful thing that we can do is allow commands to span multiple lines.

We can use any command or shell functionality within the command section. This includes escaping newline characters by ending the line with \:

target: source
    command1 arg1 arg2 arg3 arg4 \
    arg5 arg6

This becomes more important if you take advantage of some of the shell’s more programmatic functionality, like if-then statements:

target: source
    if [ "condition_1" == "condition_2" ];\
    then\
        command to execute;\
        another command;\
    else\
        alternative command;\
    fi

This will execute this block as if it were a one line command. In fact, we could have written this as one line, but it improves readability considerably breaking it down like this.

If you are going to escape end of line characters, be certain to not have any extra spaces or tabs after the \, or else you will get an error.

File Suffix Rules

An additional feature that you can use for file processing is file suffixes. These are general rules that provide a way of processing files based on their extension.

For instance, if you want to process all .jpg files in a directory and convert them into .png files using the ImageMagick suite, we could have something like this in our Makefile:

.SUFFIXES: .jpg .png

.jpg.png:
    @echo converting $< to $@
    convert $< $@

There are a few things we need to look at here.

The first part is the .SUFFIXES: declaration. This tells make about all of the suffixes we will use in file suffixes. Some suffixes that are used often in compiling source code, like .c and .o files are included by default and do not need to be labeled in this declaration.

The next part is the declaration of the actual suffix rule. This takes the form of original_extension.target_extension:.

This is not an actual target, but it will match any call for files with the second extension and build them out of the file in the first extension.

In our case, we can call make like this to build a file called file.png if there is a file.jpg in our directory:

make file.png

make will find the png file in the .SUFFIXES declaration and see the rule for creating .png files. It will then look for the target file with the .png replaced by .jpg in the directory. It will then execute the commands that follow.

The suffix rules use some variables that we have not been introduced to yet. These help make substitute different information based on what part of the process it is currently in:

• $?: This variable contains the list of dependencies for the current target that are more recent than the target. These would be the targets that must be re-done before executing the commands under this target.

• $@: This variable is the name of the current target. This allows us to reference the file you are trying to make, even though this rule was matched through a pattern.

• $<: This is the name of the current dependency. In the case of suffix rules, this is the name of the file that is used to create the target. In our example, this would contain file.jpg

• $*: This file is the name of the current dependency with the matched extension stripped off. Consider this an intermediate stage between the target and source files.

Create A Conversion Makefile

We will create a Makefile that will do some image manipulations and then upload the files to our file server, so that our website can then display them.

If you would like to follow along, before you begin, ensure that you have the ImageMagick packages installed. These are command line tools for manipulating images and we will make use of them in our script.

On Ubuntu or Debian, update your package sources and install with apt:

sudo apt-get update
sudo apt-get install imagemagick

On Red Hat or Rocky, you’ll need to add the epel-release repo to get extra packages like this one, then install the package using dnf:

dnf install epel-release
dnf install ImageMagick

In your current directory, create a file called Makefile:

nano Makefile

Within this file, we will start implementing our conversion targets.

Convert all JPG Files to PNG

Our server has been set up to serve .png images exclusively. Because of this, we need to convert any .jpg files to .png before uploading.

As we learned above, a suffix rule is a great way of doing this. We will start out with the .SUFFIX declaration that will list the formats we are converting between: .SUFFIXES: .jpg .png.

Afterwards, we can make a rule that will change .jpg files into .png files. We can do this using the convert command from the ImageMagick suite. The convert command syntax is convert from_file to_file.

To implement this command, we need the suffix rule that specifies the format we’re starting with and the format we’re ending with:

.SUFFIXES: .jpg .png

.jpg.png:           ## This is the suffix rule declaration

Now that we have the rule that will match, we need to implement the actual convert step.

Because we don’t know exactly what filename will be matched here, we need to use the variables that we learned about. Specifically, we need to reference $< as the original file, and $@ as the file we are converting to. If we combine this with what we know about the convert command, we get this rule:

.SUFFIXES: .jpg .png

.jpg.png:
    convert $< $@

Let’s add some functionality so that we can be told explicitly what is happening with an echo statement. We will include the @ symbol before the new command and the command we already had in order to silence the actual command from being printed when it is executed:

.SUFFIXES: .jpg .png

.jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

At this point, we should save and close the file so that we can test it.

Get a jpg file into the current directory. If you don’t have a file on hand, you can download one from the DigitalOcean website by using wget:

wget https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/PoweredByDO/DO_Powered_by_Badge_blue.png
mv DO_Powered_by_Badge_blue.png badge.jpg

You can test whether your make file is working thus far by asking it to create a badge.png file:

make badge.png

Output
converting badge.jpg to badge.png using ImageMagick...
conversion to badge.png successful!

Make will go to the Makefile, see the .png in the .SUFFIXES declaration and then go to the suffix rule that matches. It then runs the commands listed.

Create a File List

At this point, make can create a .png file if we explicitly tell it that we want that file.

It would be better if it just created a list of .jpg files in the current directory and then converted those. We can do this by creating a variable that holds all of our files to be converted.

The best way to do this is with the wildcard directive like JPG_FILES=$(wildcard *.jpg).

We could just specify a target with a bash wildcard like JPG_FILES=*.jpg, but this has a shortcoming. If there are no .jpg files, this actually tries to run the conversion commands on a file called *.jpg, which will fail.

The wildcard syntax we mentioned above compiles a list of .jpg files in the current directory, and if none exist, it doesn’t set the variable to anything.

While we are doing this, we should try to handle a slight variation in .jpg files that is common. These image files are often seen with the .jpeg extension instead of .jpg. To handle these in an automated way, we can change their name in our program to .jpg files.

Instead of the above lines, we will use these two:

JPEG=$(wildcard *.jpg *.jpeg)     ## Has .jpeg and .jpg files
JPG=$(JPEG:.jpeg=.jpg)            ## Only has .jpg files

The first line compiles a list of .jpg and .jpeg files in the current directory and stores them in a variable called JPEG.

The second line references this variable and does a name translation to convert the names in the JPEG variable that end with .jpeg into names that end with .jpg. This is done with the syntax $(VARNAME:.convert_from=.convert_to).

At the end of these two lines, we will have a new variable called JPG which contains only .jpg filenames. Some of these files may not actually exist on the system, because they are actually .jpeg files (no actual renaming took place). This is okay because we are only using this list to make a new list of .png files we want to create:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)

Now, we have a list of files we want to request in the variable PNG. This list contains only .png filenames, because we did another name conversion. Now, every file that was a .jpg or .jpeg file in this directory has been used to compile a list of .png files we want to create.

We also need to update the .SUFFIXES declaration and the suffixes rule to reflect that we are now handling .jpeg files:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)
.SUFFIXES: .jpg .jpeg .png

.jpeg.png .jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

As you can see, we have added the .jpeg to the suffixes list and also included another suffix match for our rule.

Create Some Targets

We have quite a lot in our Makefile right now, but we don’t have any normal targets yet. Let’s fix that so that we can pass our PNG list to our suffix rule:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)
.SUFFIXES: .jpg .jpeg .png

convert: $(PNG)

.jpeg.png .jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

All this new target does is list the .png filenames that we gathered as a requirement. Make then sees if there’s a way that it can acquire the .png files and uses the suffix rule to do so.

Now, we can use this command to convert all of our .jpg and .jpeg files to .png files:

make convert

Let’s add another target. Another task that is generally done when uploading images to a server is to resize them. Having your images the correct size will save your users from having to resize images on the fly when they request them.

An ImageMagick command called mogrify can resize images in the way that we need. Let’s say the area where our images will be displayed on our site is 500px wide. We can convert for this area with the command:

mogrify -resize 500\> file.png

This will resize any images larger than 500px wide to fit this area, but will not touch smaller images. This is what we want. As a target, we can add this rule:

resize: $(PNG)
    @echo resizing file...
    @mogrify -resize 648\> $(PNG)
    @echo resizing is complete!

We can add this to our file like this:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)
.SUFFIXES: .jpg .jpeg .png

convert: $(PNG)

resize: $(PNG)
    @echo resizing file...
    @mogrify -resize 648\> $(PNG)
    @echo resizing is complete!

.jpeg.png .jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

Now, we can string these two targets together as dependencies of another target:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)
.SUFFIXES: .jpg .jpeg .png

webify: convert resize

convert: $(PNG)

resize: $(PNG)
    @echo resizing file...
    @mogrify -resize 648\> $(PNG)
    @echo resizing is complete!

.jpeg.png .jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

You may notice that resize implicitly will run the same commands as convert. We are going to specify them both though in case that is not always the case. The convert could in the future contain more elaborate processing.

The webify target now converts and resizes images.

Upload Files to Remote Server

Now that we have our images ready for the web, we can create a target to upload them to the static images directory on our server. We can do this by passing our list of converted files to scp:

Our target will look something like this:

upload: webify
    scp $(PNG) root@ip_address:/path/to/static/images

This will upload all of our files to the remote server. Our file now looks like this:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)
.SUFFIXES: .jpg .jpeg .png

upload: webify
    scp $(PNG) root@ip_address:/path/to/static/images

webify: convert resize

convert: $(PNG)

resize: $(PNG)
    @echo resizing file...
    @mogrify -resize 648\> $(PNG)
    @echo resizing is complete!

.jpeg.png .jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

Clean Up

Let’s add a cleaning option to get rid of all of the local .png files after they have been uploaded to the remote server:

clean:
    rm *.png

Now, we can add another target at the top, that calls this one after we upload our files to the remote server. This will be the most complete target, and the one that we want to be default.

To specify this, we will put it as the first target available. This will be used as the default. We will call it all by convention:

JPEG=$(wildcard *.jpg *.jpeg)
JPG=$(JPEG:.jpeg=.jpg)
PNG=$(JPG:.jpg=.png)
.SUFFIXES: .jpg .jpeg .png

all: upload clean

upload: webify
    scp $(PNG) root@ip_address:/path/to/static/images

webify: convert resize

convert: $(PNG)

resize: $(PNG)
    @echo resizing file...
    @mogrify -resize 648\> $(PNG)
    @echo resizing is complete!

clean:
    rm *.png

.jpeg.png .jpg.png:
    @echo converting $< to $@ using ImageMagick...
    @convert $< $@
    @echo conversion to $@ successful!

With these last touches, if you enter the directory with the Makefile and .jpg or .jpeg files, you can call make without any arguments to process your files, send them to your server, and then delete the .png files you uploaded.

make

As you can see, it is possible to string together tasks and also to cherry pick a process up to a certain point. For instance, if you only want to convert your files and need to host them on a different server, you can just use the webify target.

Conclusion

At this point, you should have a good idea of how to use Makefiles in general. More specifically, you should know how to use make as a tool for automating most kinds of procedures.

While in some cases it may work better to write a script, Makefiles are a way of setting up a structured, hierarchical relationship between processes. Learning how to leverage this tool can help make repetitive tasks more manageable.

.SUFFIXES: .jpg .png

.jpg.png:
	@echo converting $< to $@ using ImageMagick...
	@convert $< $@
	@echo conversion to $@ successful!