Accelerate Application Builds With Dmake

By Hunter Li, July 2008 (Updated by Nikolay Molchanov, June 2016)

Modern commercial applications are usually very complicated. They might consist of a large number of source files with thousands or even millions of lines of code. The source files are typically organized and managed by the so-called project files, or make files, or build files for some specific build utilities, such as the make utility or ant utility to build applications in a managed way. Most of these utilities work in the serial mode, that is, based on the dependencies of the build targets to dispatch build jobs one after the other. For the medium-sized or more complicated applications, it can take hours up to days to fully rebuild the entire application. Fast machines seem to be the only way to accelerate the build process. The question is: Besides a fast machine, is there any other way to accelerate this process in today's networked environment? The answer is yes, and Oracle Developer Studio software could be the solution.

Oracle Developer Studio software is an integrated development environment. It contains not only compilers but a lot of useful tools, such as debugging tools, performance analysis and optimization tools, and development tools. One of these tools is dmake, which can accelerate the application build by parallelizing build activities over multiple CPUs or even multiple servers. This article uses examples to illustrate the steps to set up a distributed build environment and describes some guidelines on the makefile modification for dmake.

Contents

Dmake Defined

The full name of dmake is Distributed Make. It comes with Oracle Developer Studio software and is a superset of the standard Oracle Solaris make utility. Oracle Developer Studio software has both Oracle Solaris and Linux versions, so dmake can run on both platforms. Dmake has a unique feature that the Oracle Solaris make utility does not have: it supports building applications concurrently. Since Sun Studio 12, it has four make modes:

  • Serial mode: In the serial mode, dmake behaves just like the standard serial Oracle Solaris make. It dispatches build jobs only to the local dmake host one by one.
  • Parallel mode: In this mode, dmake makes targets concurrently. It still dispatches the build jobs to the local dmake host, but it dispatches multiple jobs concurrently as defined. If the local machine has multiple CPUs, it will truly work in the concurrent mode, thus shortening the build duration.
  • Distributed mode: This is the fully distributed and concurrent mode. Dmake distributes build jobs concurrently to all preconfigured build servers over networks in a predefined way. And the built outcome will be put back to a common folder of the dmake host (that is, a shared directory of the dmake control server).
  • Grid mode: This mode works with Sun Grid Engine. Dmake distributes build jobs via Sun Grid Engine, but this function is beyond this article's scope.

By dispatching build jobs to multiple CPUs or even to multiple servers, dmake parallelizes the build processes and shortens the duration. This ability differentiates dmake from other standard build utilities.

When to Use Dmake

Dmake accelerates the build process by taking advantage of multiple CPUs or multiple servers to build application components concurrently. Spare CPU resources or servers are expected. To build the application properly in the concurrent mode, some makefiles might need to be slightly modified to define which parts of the application can be built simultaneously, while others can't. As such, Dmake can dispatch build jobs in the appropriate sequence. In distributed mode, source files and outcomes are transferred among the dmake host and build servers over the network. It might introduce some network load, though not much. Hence, before you decide to use dmake, you need to consider and compare the benefits of using the concurrent build versus some extra work or overhead mentioned here.

Here are some factors to consider when deciding whether to use dmake:

  • If you have a complicated application in terms of the number of components and lines of code and if the serial build will take more than half an hour, then using dmake might be a good idea.
  • If you have only one build server, make sure that it has multiple CPUs and has enough spare CPU resources. A uniprocessor system or a busy system won't demonstrate a good concurrent work process.
  • If you have multiple servers, make sure they have about the same architecture in terms of CPUs. You can't mix a SPARC server with a x86/x64 server to build one type of binary.
  • The network should not be overloaded or be involved in any situations that could cause bad network latency. Otherwise, the distributed mode could be even worse than the local serial mode.
  • The distributed build mode requires the remote shell (rsh) and NFS. If the systems are not placed in a secured network environment, the distributed mode might not be an option due to security concerns.

Configure a Distributed Build Environment

By default, dmake works in parallel mode. In other words, you don't have to do any extra work to enable the parallel mode. However, as noted previously, to dispatch build jobs in the appropriate sequence in terms of dependencies, you need to modify some makefiles first. Some guidelines are discussed in the next section.

To enable the distributed mode, you must configure the build environment beforehand. Here's the concept and configuration example to set up the dmake distributed mode:

  • The dmake control server is called dmake host. Usually it contains source code. The server that actually builds the application is called the build server. The dmake host can also be a build server.
  • It is highly recommended that the dmake host and all build servers use the same version of the operating system with the same patch level. Otherwise, binary difference might cause link problems.
  • The dmake host and all build servers should use the same version of Oracle Developer Studio software.
  • The dmake host shares the source code via NFS. The shared directory and all subdirectories and files under it must be accessible from all build servers with read, write, and execute permissions, as shown here. 
    

root@dmake_host# share -F nfs \
 -o rw=<build servers delimited by colon> \
 /fullpath/to_source_root
  • All build servers must be able to access the same physical directory of the dmake host via NFS using the same mount path, as shown here. 
    

root@build_server# mount -F nfs dmake_host:/fullpath/to_source_root \
 /fullpath/to_source_root
  • The dmake host name and all build server names must be resolvable by each other. Hence, /etc/hosts and /etc/inet/ipnodes should be configured properly in the dmake host and all build servers to resolve all host names.
  • The dmake host must be able to execute commands in the remote build server by rsh without being prompted for a password. Hence, all build servers and the dmake host should have one identical user login account in terms of the user login name and the user ID. In addition, the dmake host with that user login account should be trusted by all build servers, as shown here. 
    

appuser@build_server% id
uid=201(appuser) gid=1
appuser@build_server% cat $HOME/.rhosts
dmake_host appuser

    

appuser@dmake_host% id
uid=201(appuser) gid=1

    The appuser in the preceding code is an example user. Any subsequent occurrence of the user account described in the following sections of this article refers to this user account.

  • The bin directory in which the dmake utility and other Oracle Developer Studio binaries are installed must be accessible from all build servers, and by the dmake host via the rsh command. To achieve that, the user login shell of all build servers should be configured to use the C-shell (csh), because the environment variables only defined in the $HOME/.cshrc can be applied when executing rsh commands, as shown here. 
    

appuser@build_server% echo $SHELL
/bin/csh
appuser@build_server% cat $HOME/.cshrc
setenv PATH /opt/SUNWspro/bin:/usr/bin

    

appuser@dmake_host% rsh build_server which dmake
/opt/SUNWspro/bin/dmake
  • /etc/opt/SPROdmake/dmake.conf defines at least the job capacity that the build server can accept from the dmake host. It must exist on the build server and be accessible from the user account of the build server. Without this file, no dmake job will be allowed to run on that build server. See the following example. 
    

appuser@build_server% cat /etc/opt/SPROdmake/dmake.conf
max_jobs: 2
nice_prio: 5

    The preceding example sets the maximum job capacity to two jobs for the build server, and the optional nice priority to five.

  • The dmake host defines how the distributed build environment is finally configured. By default, dmake searches for the dmake runtime configuration file, .dmakerc, in the home directory of the build user on the dmake host. An example of .dmakerc is shown here. 
    

appuser@dmake_host% cat $HOME/.dmakerc
group "solaris10_x64" {
 host build_server_1 { jobs = 2, path="/opt/SUNWspro/bin" }
 host build_server_2 { jobs = 5, path="/export/home/SUNWspro/bin" }
 host build_server_3 { jobs = 8, path="/opt/SUNWspro/bin" }
}

    The example .dmakerc file defines a build server group named solaris10_x64 that consists of three build servers. The build_server_1 can accept a maximum of two build jobs, while build_server_2 and build_server_3 can accept more jobs--five and eight, respectively. The path to the dmake binary is different on the three build servers. They are explicitly specified.

    Multiple groups can exist in one .dmakerc file. Unless explicitly specifying the group by the dmake -g command-line option or by the DMAKE_GROUP environment variable or macro, the first group instance is used.

  • Two environment variables are recommended in the .cshrc or .profile file on the dmake host, DMAKE_MODE and DMAKE_ODIR. DMAKE_MODE overrides the default parallel build mode. DMAKE_ODIR specifies a common physical directory that dmake can write temporary output files to and read temporary output files from. By default, the directory used is $(HOME)/.dmake, and it must be visible to all build servers. If possible, use the same source root directory as the parent directory for .dmake because the source root directory must be shared via NFS to the network anyway, and to reduce the exposure of the irrelevant files in the user's home directory. Thus, DMAKE_ODIR should be set to .dmake under the source root directory on the dmake host. To further reduce the exposure of the source files and newly created files, set umask to 0077.

    An example of the .cshrc file is shown here.

    

appuser@dmake_host% cat $HOME/.cshrc
setenv PATH /opt/SUNWspro/bin:/usr/bin:/usr/sbin:/usr/ucb:.
setenv BASE_PATH /fullpath/to_source_root
setenv MAKE dmake
setenv DMAKE_MODE distributed
setenv DMAKE_ODIR /fullpath/to_source_root/.dmake
umask 0077

    Thus, you don't have to specify dmake command-line options each time you type dmake on the dmake host. And, of all build servers, only appuser can visit the shared directory of the dmake host.

Now the distributed build environment is set.

Guidelines on Makefile Customization for Dmake

To run dmake in the parallel mode or distributed mode, the makefile usually needs to be slightly changed to define which components can be built concurrently while others can't, in terms of dependency. Otherwise, dmake could erroneously launch both the dependency and the dependent components simultaneously or even ahead of the dependent ones. Thus, the build will fail.

Makefiles can be organized differently from application to application. It's beyond the scope of this article to give guidelines for all cases. The guidelines in the following sections might only work for the nested makes, that is, when there's one makefile per component, and a top-level makefile describes the build sequence by calling to every make for each component one after the other. The idea for the nested makes might be borrowed by other cases.

In the case of the nested makes, you might put the focus on the top-level makefile first.

  • For each called make, create a subtarget label and put the make string as the action.
  • Put all subtarget labels created previously as the dependencies to the implicit "all" target label.
  • Analyze whether there is any dependency relationship among the subtargets. If yes, properly set up the dependency of the related subtargets.
  • Set the dmake built-in target .PARALLEL to all subtarget labels to enable the full parallel build, and let dmake judge the runtime build sequence based on the dependencies.

Here's a simple example that illustrates the idea:

  • The main.mak builds the final executable. It relies on four libraries, libmylib1.so, libmylib2.so, libmylib3.so, and libmylib4.so. The libmylib1.so calls functions in the libmylib2.so. The top-level makefile calls to all makes. It can be changed from this code: 
    

appuser@dmake_host% cat Makefile
include $(BASE_PATH)/Makefile.in

all:
        $(MAKE) -f $(BASE_PATH)/src/mylib2/mylib2.mak
        $(MAKE) -f $(BASE_PATH)/src/mylib3/mylib3.mak
        $(MAKE) -f $(BASE_PATH)/src/mylib4/mylib4.mak
        $(MAKE) -f $(BASE_PATH)/src/mylib1/mylib1.mak
        $(MAKE) -f $(BASE_PATH)/src/main/main.mak

clean:
        $(MAKE) -f $(BASE_PATH)/src/main/main.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib1/mylib1.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib2/mylib2.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib3/mylib3.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib4/mylib4.mak clean

    To this code: 

    

appuser@dmake_host% cat Makefile
include $(BASE_PATH)/Makefile.in

TARGET= libmylib1.so \
        libmylib2.so \
        libmylib3.so \
        libmylib4.so \
        main

.PARALLEL: $(TARGET)

all: $(TARGET)

main: libmylib1.so libmylib2.so libmylib3.so libmylib4.so
        $(MAKE) -f $(BASE_PATH)/src/main/main.mak
libmylib1.so: libmylib2.so
        $(MAKE) -f $(BASE_PATH)/src/mylib1/mylib1.mak
libmylib2.so:
        $(MAKE) -f $(BASE_PATH)/src/mylib2/mylib2.mak
libmylib3.so:
        $(MAKE) -f $(BASE_PATH)/src/mylib3/mylib3.mak
libmylib4.so:
        $(MAKE) -f $(BASE_PATH)/src/mylib4/mylib4.mak

clean:
        $(MAKE) -f $(BASE_PATH)/src/main/main.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib1/mylib1.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib2/mylib2.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib3/mylib3.mak clean
        $(MAKE) -f $(BASE_PATH)/src/mylib4/mylib4.mak clean

The makefile of a component usually does not require change because in most cases, the component makefile contains only one target. If there are multiple targets in one component makefile, it can be treated as a sub top-level makefile. Refer to the top-level makefile guidelines for customization.

However, even if the component makefile contains one target, if one build rule would generate an interim type of object that is fed as a source type of object to another rule, adjusting the build sequence with the dmake built-in special targets, including .NO_PARALLEL, .PARALLEL, .LOCAL, and .WAIT, is necessary. The idea is to have all the interim objects generated by the first rule prior to the second rule.

Here's an example:

  • In the Oracle Solaris 10 environment, users can define their own DTrace probe in the C/C++ source code. In this example, main.c contains a user-defined DTrace probe that is described by main_dtrace.d. My1func.c contains a probe described by my1func_dtrace.d. My2func.c contains a probe described by my2func_dtrace.d. To build the final main executable, use cc to compile main.c to get main.o. Then, use dtrace to compile main_dtrace.d with the interim main.o to get main_dtrace.o. Similarly, you can get my1func.o and my1func_dtrace.o, my2func.o and my2func_dtrace.o. Finally, link main.o, main_dtrace.o, my1func.o, my1func_dtrace.o, my2func.o, and my2func_dtrace.o together to get the final main executable. The original makefile is shown here: 
    

appuser@dmake_host% cat main.mak
include $(BASE_PATH)/Makefile.in

DTRACE=dtrace

TARGET=$(BIN_PATH)/main

SRC_PATH=$(BASE_PATH)/src/main

OBJECT= $(SRC_PATH)/main.o \
        $(SRC_PATH)/main_dtrace.o \
        $(SRC_PATH)/my1func.o \
        $(SRC_PATH)/my1func_dtrace.o \
        $(SRC_PATH)/my2func.o \
        $(SRC_PATH)/my2func_dtrace.o

all: $(TARGET)

$(TARGET): $(OBJECT)
        $(CC) $(CFLAGS) -o $@ $? -ldtrace -L$(LIB_PATH) \
        -lmylib1 -lmylib2 -lmylib3 -lmylib4

%_dtrace.o: %_dtrace.d
        $(DTRACE) -o $@ -G -s $< $(<:%_dtrace.d=%.o)

.SUFFIXES: .c.o

.c.o:
        $(CC) -o $@ -c $<

clean:
        $(RM) $(TARGET) $(OBJECT)

    The output of running the standard serial Solaris make is shown here:

    

appuser@dmake_host% make -f main.mak
cc -o /export/home/appuser/src/main/main.o -c /export/home/appuser/
src/main/main.c
dtrace -o /export/home/appuser/src/main/main_dtrace.o -G -s /export
/home/appuser/src/main/main_dtrace.d /export/home/appuser/src/main/
main.o
cc -o /export/home/appuser/src/main/my1func.o -c /export/home/appus
er/src/main/my1func.c
dtrace -o /export/home/appuser/src/main/my1func_dtrace.o -G -s /exp
ort/home/appuser/src/main/my1func_dtrace.d /export/home/appuser/src
/main/my1func.o
cc -o /export/home/appuser/src/main/my2func.o -c /export/home/appus
er/src/main/my2func.c
dtrace -o /export/home/appuser/src/main/my2func_dtrace.o -G -s /exp
ort/home/appuser/src/main/my2func_dtrace.d /export/home/appuser/src
/main/my2func.o
cc -o /export/home/appuser/bin/main /export/home/appuser/src/main/m
ain.o /export/home/appuser/src/main/main_dtrace.o /export/home/appu
ser/src/main/my1func.o /export/home/appuser/src/main/my1func_dtrace
.o /export/home/appuser/src/main/my2func.o /export/home/appuser/src
/main/my2func_dtrace.o -ldtrace -L/export/home/appuser/lib -lmylib1
 -lmylib2 -lmylib3 -lmylib4

    But if you run dmake directly with the distributed mode without customizing the makefile beforehand, cc and dtrace could be launched simultaneously. There's a chance that the interim object file is not ready when dtrace starts to refer to it. Thus, dtrace fails to build the relevant *_dtrace.o. To overcome this problem, adjust the makefile to run cc to build all interim objects ahead of dtrace by reordering all interim objects ahead of all *_dtrace.o and using .WAIT to synchronize the process.

    The new makefile is shown below. See the section in bold, which fixes the problem.

    

appuser@dmake_host% cat main.mak
include $(BASE_PATH)/Makefile.in

DTRACE=dtrace

TARGET=$(BIN_PATH)/main

SRC_PATH=$(BASE_PATH)/src/main

OBJECT=$(SRC_PATH)/main.o \
$(SRC_PATH)/my1func.o \                                       
$(SRC_PATH)/my2func.o \ 
.WAIT \                                      
$(SRC_PATH)/main_dtrace.o \                                         
$(SRC_PATH)/my1func_dtrace.o \                                         
$(SRC_PATH)/my2func_dtrace.o

all: $(TARGET)

$(TARGET): $(OBJECT)
        $(CC) $(CFLAGS) -o $@ $? -ldtrace -L$(LIB_PATH) \
        -lmylib1 -lmylib2 -lmylib3 -lmylib4

%_dtrace.o: %_dtrace.d
        $(DTRACE) -o $@ -G -s $<$(<:%_dtrace.d=%.o)

.SUFFIXES: .c.o

.c.o:
        $(CC) -o $@ -c $<

clean:
        $(RM) $(TARGET) $(OBJECT)