Non-recursive make (gmake) part 1: the basic GNUmakefile layouts

For a while I’ve been converting recursive-make projects to non-recursive (“NR”) makes. I’ve learned some necessary conventions, some tricks and some limitations. All in all, non-recursive make trees are a reliable and efficient way to work.

Why

The “killer” reason for NR make is speed. I have to work with black-box 3rd-party software, and real-world connections. This means that automated tests sometimes have to use “sleep n” while something external completes. If n is too short, the test is unreliable. If n is too long, the tests take forever. And that can be really long (hours or days) as your tests mount up.

“make -kj test” is the obvious answer: run tests in parallel, and all your tests together take no longer than the longest test. Unfortunately, “make -j” almost always falls apart in recursive make set-ups; make doesn’t really understand what the real dependencies are.

Here’s where NR make steps in: you can reliably express all dependencies in a project, and “make -C $ROOTDIR” is not detectably slower than just “make” in a sub-directory.

For large projects, with hundreds or thousands of sub-projects, NR make is faster simply for lack of shelling out so often. A typical Unix box can fstat hundreds of thousands of files per second; and a multi-core box can run a lot of compiles in parallel. Even on a small project of mine (5 subprojects, 200 files) there’s a 5:1 speed-up.

When?

There’s a limit to what you may do for speed. Writing a project’s makefiles from scratch is easy. Adapting an existing makefile system takes more work. Adapting a third-party project’s make is usually more pain than it’s worth. For that reason, I like to segregate third-party code, built from source, into its own project, and export all its headers, libraries, binaries, etc. separately — without even listing those exports as dependencies of the NR make targets.

What?

Side note: I use GMAKE. It’s the one flavour of MAKE I know that makes grafting Makefiles together easy. Sorry for not addressing any other interesting or wide-spread versions of MAKE. Perhaps I’ll give BSDMAKE a go, if only for Brian Somers’ interest.

I like to have my makes both ways: project-wide and subproject local. It helps me introduce people to the NR make scheme, one step at a time. So I build a ROOT GNUmakefile that includes sub-project makefiles directly.

I don’t care about building multiple outputs at the same time (release, debug, profiling, … or xplats).

I like makefiles that don’t use a lot of unreadable magic (e.g. IF blocks, or use of $(shell …)). In fact, I confess to being so old-school that I prefer “conditionals” like:

CFLAGS.debug = -O0
CFLAGS.profile = -pg
CFLAGS += $(CFLAGS.$(BUILDTYPE))

How?

We’ll start with the top-level makefile, and work down.

include rules.mk
src     ?= .
hx      := $(src)/hx
madns   := $(src)/madns
util    := $(src)/util

include $(hx)/GNUmakefile
include $(madns)/GNUmakefile
include $(util)/GNUmakefile

Immediately you think, “Oho! Much dark magic is probably buries in ‘rules.mk'”. We’ll look at that later, and be the judge then. For now I’l say that rules.mk sets or alters the standard variables used by GMAKE’s default rules (e.g. CFLAGS, CPPFLAGS, LDLIBS) and defines some pattern rules (e.g. %.so : %.a ; …) and some handy-to-haves I’ve mentioned in previous GMAKE-tricks posts.

Note that even this makefile is potentially includable inside another parent makefile, that would define $(src) differently.

And here is one of the sub-makefiles — $(madns)/GNUmakefile. Remember this makefile is set up to be USABLE directly, but primarily to be includable.

include ../rules.mk
export madns    ?= .
# madns imports libtap from util:
util            ?= ../util
PATH            := $(madns):$(PATH)
#---------------- PRIVATE vars:
madns.test      = $(madns)/madns_t
#---------------- PUBLIC TARGETS (see rules.mk):
all  .PHONY     : madns.all
test .PHONY     : madns.test
#---------------- inputs to "install":
madns.bin       = $(madns)/hostip
madns.lib       = $(madns)/madns.a
#---------------- PRIVATE TARGETS:
madns.all       : $(madns.bin) $(madns.lib) $(madns.test)
madns.test      : $(madns.test:%=%.pass)

$(madns.bin)    : $(madns.lib)
$(madns.lib)    : $(madns)/madns.o

$(madns.test)   : LDLIBS += -pthread
$(madns.test)   : CFLAGS += -I$(util)
$(madns.test)   : $(madns.lib) $(util)/libtap.a

.INTERMEDIATE   : $(madns)/madns.o $(madns)/madns_t.o
-include $(madns)/*.d

You’ll notice that there are only two types of targets: bog-standard GMAKE targets and variables (all, test, ..) and targets whose names begin with a unique identifier for this sub-project. This is the one unbreakable rule: MAKE has no concept of local scope, so EVERY variable and target must somehow be localized; or else it must be treated as an unordered global; e.g.:

ALL += $(madns)
all : madns.all 

This also applies to file references: every (!) file mentioned must have an explicit directory, taken from some variable that begins with “madns”. The root makefile cannot (!) chdir into each subdirectory because all references across all subprojects must refer to the same paths.

You may also notice a peculiar convention, that there are variables with the same names as targets. If this is confusing, my apologies. My imagination for variable names tends to run low; and it seems reasonable that variables related to targets can share names. For now, you’ll have to rest assured that I don’t use this as any particular kind of magic. In fact, here’s where I WISH there were GMAKE magic to help; say. something like $(^madns.test) to list the antecedents of target “madns.test”. Ah well, no such luck.

There’s a bit of boiler-plate in there that I have yet to fit into generic rules.mk: the .INTERMEDIATE rule and the “-include $(madns)/*.d”. Again, more on that later, but I’d welcome any elegant solution to that.

Advertisements

About mischasan

I've had the privilege to work in a field where abstract thinking has concrete value. That applies at the macro level --- optimizing actions on terabyte database --- or the micro level --- fast parallel string searches in memory. You can find my documents on production-system radix sort (NOT just for academics!) and some neat little tricks for developers, on my blog https://mischasan.wordpress.com My e-mail sig (since 1976): Engineers think equations approximate reality. Physicists think reality approximates the equations. Mathematicians never make the connection.
This entry was posted in make, non-recursive make and tagged , , . Bookmark the permalink.

4 Responses to Non-recursive make (gmake) part 1: the basic GNUmakefile layouts

  1. mischasan says:

    Ha! Finally found the answer to that last problem of intermediate files being cleaned up. The directive:

    .SECONDARY:

    ensures that intermediate files are left around, so that gmake no longer annoyingly rebuilds some targets TWICE before it decides there’s nothing left to do. No more .INTERMEDIATE: boilerplate.

  2. trevd says:

    No doubt you’ve seen this 1997 essay “Recursive Make Considered Harmful”
    hxxp://miller.emu.id.au/pmiller/books/rmch .

    Also the Android Build System is implemented in a non recursive style and currently operates on about ~400 projects. hxxps://android.googlesource.com/platform/build/+/master .
    [ TIP : core/main.mk is the actual “entry point” of the system not the more obviously named Makefile ]

    Great Blog BTW! Lots of interesting stuff especially the sse stuff

    • mischasan says:

      Comments appreciated. Hope you have some use for the SSE stuff.
      Yes, I’ve read “RMCH” … it’s what convinced me that the effort to derecurse (“discurse”? 🙂 ) makefiles would be worth it.
      Thanks for the exact link to the original document.

      As for the android build system … wow. That’s a *LOT* of makefile(s).
      Given its origin, I’m guessing some smart people contributed to its structure, so I’ll give it a thorough read. My initial reaction, though, is …

      I know two reasonable solutions to the make namespace problem:
      appending (+=) to (global) input variables of global non-generic rules; and generic rules for module-specific targets like:

      %.pass : % ; $* >& $*.fail && mv -f $*.fail $@

      … where the module makefile can define “$(mytestprog).pass”. The generic rules with actions can be in platform-specific files, and tend to be readably small. This model also makes it possible to run (e.g):

      $ make test01.pass

      As you can guess, I prefer generic rules, because it is easy to add specifics for a target in a sub-project, such as scoped variable settings and extra dependencies — though the global variables have their place.

      $(mytestprog).pass : mytestdata
      extra_clean += mytestdata

      … with, say, a generic rule-action:

      % : %.gz ; gunzip -c $^ >$@

      It’s harder to do that in the global-rules model, usually needing extra magic parameters to make a global rule do several things.
      I wrote the “rules” script to try and locate all the magic inputs in some existing systems; with limited success.

      A third solution is lots of user-defined gmake functions, but the only examples I’ve seen badly obscured the makefile’s intent and action; like C preprocessor abuse. The less magic in a build system, the better.

      • trevd says:

        Well the other half of the story is the Android.mk makefile fragment which accompanies each project. They basically use the same variables in each file and have a clear_var makefile which handles initialization for want of a better word. There’s also a full device specification structure which can be found in any of the device/* repositories

        I think they “cheat” a little by employing a couple of python scripts to do some of the heavy lifting.

        I’ll say It’s very much a “Build System” ( a’la build root I suppose ), they’re engineered hook points which makes it possible to extend without ever touching the main system. It does make building a full Operating System as easy as typing “make” which definitely lowers the barrier of entry for mere mortals like myself.

        It’s not all rainbows and unicorns however as there’s little documentation and what there is outdated and perhaps now even incorrect plus getting an handle on anything of scale obviously involves a time investment which some folks can’t/won’t make … that’s what consultants are for I suppose.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s