mirror of
https://github.com/torvalds/linux.git
synced 2024-09-20 23:13:00 +00:00
2d3b1b8f0d
The incremental build of Linux kernel is pretty slow when lots of
objects are compiled. The rebuild of allmodconfig may take a few
minutes even when none of the objects needs to be rebuilt.
The time-consuming part in the incremental build is the evaluation of
if_changed* macros since they are used in the recipes to compile C and
assembly source files into objects.
I notice the following code in if_changed* is expensive:
$(filter-out $(PHONY) $(wildcard $^),$^)
In the incremental build, every object has its .*.cmd file, which
contains the auto-generated list of included headers. So, $^ are
expanded into the long list of the source file + included headers,
and $(wildcard $^) checks whether they exist.
It may not be clear why this check exists there.
Here is the record of my research.
[1] The first code addition into Kbuild
This code dates back to 2002. It is the pre-git era. So, I copy-pasted
it from the historical git tree.
| commit 4a6db0791528c220655b063cf13fefc8470dbfee (HEAD)
| Author: Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de>
| Date: Mon Jun 17 00:22:37 2002 -0500
|
| kbuild: Handle removed headers
|
| New and old way to handle dependencies would choke when a file
| #include'd by other files was removed, since the dependency on it was
| still recorded, but since it was gone, make has no idea what to do about
| it (and would complain with "No rule to make <file> ...")
|
| We now add targets for all the previously included files, so make will
| just ignore them if they disappear.
|
| diff --git a/Rules.make b/Rules.make
| index 6ef827d3df39..7db5301ea7db 100644
| --- a/Rules.make
| +++ b/Rules.make
| @@ -446,7 +446,7 @@ if_changed = $(if $(strip $? \
| # execute the command and also postprocess generated .d dependencies
| # file
|
| -if_changed_dep = $(if $(strip $? \
| +if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\
| $(filter-out $(cmd_$(1)),$(cmd_$@))\
| $(filter-out $(cmd_$@),$(cmd_$(1)))),\
| @set -e; \
| diff --git a/scripts/fixdep.c b/scripts/fixdep.c
| index b5d7bee8efc7..db45bd1888c0 100644
| --- a/scripts/fixdep.c
| +++ b/scripts/fixdep.c
| @@ -292,7 +292,7 @@ void parse_dep_file(void *map, size_t len)
| exit(1);
| }
| memcpy(s, m, p-m); s[p-m] = 0;
| - printf("%s: \\\n", target);
| + printf("deps_%s := \\\n", target);
| m = p+1;
|
| clear_config();
| @@ -314,7 +314,8 @@ void parse_dep_file(void *map, size_t len)
| }
| m = p + 1;
| }
| - printf("\n");
| + printf("\n%s: $(deps_%s)\n\n", target, target);
| + printf("$(deps_%s):\n", target);
| }
|
| void print_deps(void)
The "No rule to make <file> ..." error can be solved by passing -MP to
the compiler, but I think the detection of header removal is a good
feature. When a header is removed, all source files that previously
included it should be re-compiled. This makes sure we has correctly
got rid of #include directives of it.
This is also related with the behavior of $?. The GNU Make manual says:
$?
The names of all the prerequisites that are newer than the target,
with spaces between them.
This does not explain whether a non-existent prerequisite is considered
to be newer than the target.
At this point of time, GNU Make 3.7x was used, where the $? did not
include non-existent prerequisites. Therefore,
$(filter-out FORCE $(wildcard $^),$^)
was useful to detect the header removal, and to rebuild the related
objects if it is the case.
[2] Change of $? behavior
Later, the behavior of $? was changed (fixed) to include prerequisites
that did not exist.
First, GNU Make commit 64e16d6c00a5 ("Various changes getting ready for
the release of 3.81.") changed it, but in the release test of 3.81, it
turned out to break the kernel build.
See these:
- http://lists.gnu.org/archive/html/bug-make/2006-03/msg00003.html
- https://savannah.gnu.org/bugs/?16002
- https://savannah.gnu.org/bugs/?16051
Then, GNU Make commit 6d8d9b74d9c5 ("Numerous updates to tests for
issues found on Cygwin and Windows.") reverted it for the 3.81 release
to give Linux kernel time to adjust to the new behavior.
After the 3.81 release, GNU Make commit 7595f38f62af ("Fixed a number
of documentation bugs, plus some build/install issues:") re-added it.
[3] Adjustment to the new $? behavior on Kbuild side
Meanwhile, the kernel build was changed by commit 4f1933620f ("kbuild:
change kbuild to not rely on incorrect GNU make behavior") to adjust to
the new $? behavior.
[4] GNU Make 3.82 released in 2010
GNU Make 3.82 was the first release that integrated the correct $?
behavior. At this point, Kbuild dealt with GNU Make versions with
different $? behaviors.
3.81 or older:
$? does not contain any non-existent prerequisite.
$(filter-out $(PHONY) $(wildcard $^),$^) was useful to detect
removed include headers.
3.82 or newer:
$? contains non-existent prerequisites. When a header is removed,
it appears in $?. $(filter-out $(PHONY) $(wildcard $^),$^) became
a redundant check.
With the correct $? behavior, we could have dropped the expensive
check for 3.82 or later, but we did not. (Maybe nobody noticed this
optimization.)
[5] The .SECONDARY special target trips up $?
Some time later, I noticed $? did not work as expected under some
circumstances. As above, $? should contain non-existent prerequisites,
but the ones specified as SECONDARY do not appear in $?.
I asked this in GNU Make ML, and it seems a bug:
https://lists.gnu.org/archive/html/bug-make/2019-01/msg00001.html
Since commit 8e9b61b293 ("kbuild: move .SECONDARY special target to
Kbuild.include"), all files, including headers listed in .*.cmd files,
are treated as secondary.
So, we are back into the incorrect $? behavior.
If we Kbuild want to react to the header removal, we need to keep
$(filter-out $(PHONY) $(wildcard $^),$^) but this makes the rebuild
so slow.
[Summary]
- I believe noticing the header removal and recompiling related objects
is a nice feature for the build system.
- If $? worked correctly, $(filter-out $(PHONY),$?) would be enough
to detect the header removal.
- Currently, $? does not work correctly when used with .SECONDARY,
and Kbuild is hit by this bug.
- I filed a bug report for this, but not fixed yet as of writing.
- Currently, the header removal is detected by the following expensive
code:
$(filter-out $(PHONY) $(wildcard $^),$^)
- I do not want to revert commit 8e9b61b293 ("kbuild: move
.SECONDARY special target to Kbuild.include"). Specifying
.SECONDARY globally is clean, and it matches to the Kbuild policy.
This commit proactively removes the expensive check since it makes the
incremental build faster. A downside is Kbuild will no longer be able
to notice the header removal.
You can confirm it by the full-build followed by a header removal, and
then re-build.
$ make defconfig all
[ full build ]
$ rm include/linux/device.h
$ make
CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
DESCEND objtool
CHK include/generated/compile.h
Kernel: arch/x86/boot/bzImage is ready (#11)
Building modules, stage 2.
MODPOST 12 modules
Previously, Kbuild noticed a missing header and emits a build error.
Now, Kbuild is fine with it. This is an unusual corner-case, not a big
deal. Once the $? bug is fixed in GNU Make, everything will work fine.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
332 lines
12 KiB
Plaintext
332 lines
12 KiB
Plaintext
# SPDX-License-Identifier: GPL-2.0
|
|
####
|
|
# kbuild: Generic definitions
|
|
|
|
# Convenient variables
|
|
comma := ,
|
|
quote := "
|
|
squote := '
|
|
empty :=
|
|
space := $(empty) $(empty)
|
|
space_escape := _-_SPACE_-_
|
|
pound := \#
|
|
|
|
###
|
|
# Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
|
|
dot-target = $(dir $@).$(notdir $@)
|
|
|
|
###
|
|
# The temporary file to save gcc -MD generated dependencies must not
|
|
# contain a comma
|
|
depfile = $(subst $(comma),_,$(dot-target).d)
|
|
|
|
###
|
|
# filename of target with directory and extension stripped
|
|
basetarget = $(basename $(notdir $@))
|
|
|
|
###
|
|
# real prerequisites without phony targets
|
|
real-prereqs = $(filter-out $(PHONY), $^)
|
|
|
|
###
|
|
# Escape single quote for use in echo statements
|
|
escsq = $(subst $(squote),'\$(squote)',$1)
|
|
|
|
###
|
|
# Easy method for doing a status message
|
|
kecho := :
|
|
quiet_kecho := echo
|
|
silent_kecho := :
|
|
kecho := $($(quiet)kecho)
|
|
|
|
###
|
|
# filechk is used to check if the content of a generated file is updated.
|
|
# Sample usage:
|
|
#
|
|
# filechk_sample = echo $(KERNELRELEASE)
|
|
# version.h: FORCE
|
|
# $(call filechk,sample)
|
|
#
|
|
# The rule defined shall write to stdout the content of the new file.
|
|
# The existing file will be compared with the new one.
|
|
# - If no file exist it is created
|
|
# - If the content differ the new file is used
|
|
# - If they are equal no change, and no timestamp update
|
|
# - stdin is piped in from the first prerequisite ($<) so one has
|
|
# to specify a valid file as first prerequisite (often the kbuild file)
|
|
define filechk
|
|
$(Q)set -e; \
|
|
mkdir -p $(dir $@); \
|
|
{ $(filechk_$(1)); } > $@.tmp; \
|
|
if [ -r $@ ] && cmp -s $@ $@.tmp; then \
|
|
rm -f $@.tmp; \
|
|
else \
|
|
$(kecho) ' UPD $@'; \
|
|
mv -f $@.tmp $@; \
|
|
fi
|
|
endef
|
|
|
|
######
|
|
# gcc support functions
|
|
# See documentation in Documentation/kbuild/makefiles.rst
|
|
|
|
# cc-cross-prefix
|
|
# Usage: CROSS_COMPILE := $(call cc-cross-prefix, m68k-linux-gnu- m68k-linux-)
|
|
# Return first <prefix> where a <prefix>gcc is found in PATH.
|
|
# If no gcc found in PATH with listed prefixes return nothing
|
|
#
|
|
# Note: '2>/dev/null' is here to force Make to invoke a shell. Otherwise, it
|
|
# would try to directly execute the shell builtin 'command'. This workaround
|
|
# should be kept for a long time since this issue was fixed only after the
|
|
# GNU Make 4.2.1 release.
|
|
cc-cross-prefix = $(firstword $(foreach c, $(1), \
|
|
$(if $(shell command -v -- $(c)gcc 2>/dev/null), $(c))))
|
|
|
|
# output directory for tests below
|
|
TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
|
|
|
|
# try-run
|
|
# Usage: option = $(call try-run, $(CC)...-o "$$TMP",option-ok,otherwise)
|
|
# Exit code chooses option. "$$TMP" serves as a temporary file and is
|
|
# automatically cleaned up.
|
|
try-run = $(shell set -e; \
|
|
TMP="$(TMPOUT).$$$$.tmp"; \
|
|
TMPO="$(TMPOUT).$$$$.o"; \
|
|
if ($(1)) >/dev/null 2>&1; \
|
|
then echo "$(2)"; \
|
|
else echo "$(3)"; \
|
|
fi; \
|
|
rm -f "$$TMP" "$$TMPO")
|
|
|
|
# as-option
|
|
# Usage: cflags-y += $(call as-option,-Wa$(comma)-isa=foo,)
|
|
|
|
as-option = $(call try-run,\
|
|
$(CC) $(KBUILD_CFLAGS) $(1) -c -x assembler /dev/null -o "$$TMP",$(1),$(2))
|
|
|
|
# as-instr
|
|
# Usage: cflags-y += $(call as-instr,instr,option1,option2)
|
|
|
|
as-instr = $(call try-run,\
|
|
printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
|
|
|
|
# __cc-option
|
|
# Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
|
|
__cc-option = $(call try-run,\
|
|
$(1) -Werror $(2) $(3) -c -x c /dev/null -o "$$TMP",$(3),$(4))
|
|
|
|
# Do not attempt to build with gcc plugins during cc-option tests.
|
|
# (And this uses delayed resolution so the flags will be up to date.)
|
|
CC_OPTION_CFLAGS = $(filter-out $(GCC_PLUGINS_CFLAGS),$(KBUILD_CFLAGS))
|
|
|
|
# cc-option
|
|
# Usage: cflags-y += $(call cc-option,-march=winchip-c6,-march=i586)
|
|
|
|
cc-option = $(call __cc-option, $(CC),\
|
|
$(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS),$(1),$(2))
|
|
|
|
# cc-option-yn
|
|
# Usage: flag := $(call cc-option-yn,-march=winchip-c6)
|
|
cc-option-yn = $(call try-run,\
|
|
$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) $(1) -c -x c /dev/null -o "$$TMP",y,n)
|
|
|
|
# cc-disable-warning
|
|
# Usage: cflags-y += $(call cc-disable-warning,unused-but-set-variable)
|
|
cc-disable-warning = $(call try-run,\
|
|
$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
|
|
|
|
# cc-ifversion
|
|
# Usage: EXTRA_CFLAGS += $(call cc-ifversion, -lt, 0402, -O1)
|
|
cc-ifversion = $(shell [ $(CONFIG_GCC_VERSION)0 $(1) $(2)000 ] && echo $(3) || echo $(4))
|
|
|
|
# ld-option
|
|
# Usage: KBUILD_LDFLAGS += $(call ld-option, -X, -Y)
|
|
ld-option = $(call try-run, $(LD) $(KBUILD_LDFLAGS) $(1) -v,$(1),$(2),$(3))
|
|
|
|
# ld-version
|
|
# Note this is mainly for HJ Lu's 3 number binutil versions
|
|
ld-version = $(shell $(LD) --version | $(srctree)/scripts/ld-version.sh)
|
|
|
|
# ld-ifversion
|
|
# Usage: $(call ld-ifversion, -ge, 22252, y)
|
|
ld-ifversion = $(shell [ $(ld-version) $(1) $(2) ] && echo $(3) || echo $(4))
|
|
|
|
######
|
|
|
|
###
|
|
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.build obj=
|
|
# Usage:
|
|
# $(Q)$(MAKE) $(build)=dir
|
|
build := -f $(srctree)/scripts/Makefile.build obj
|
|
|
|
###
|
|
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.modbuiltin obj=
|
|
# Usage:
|
|
# $(Q)$(MAKE) $(modbuiltin)=dir
|
|
modbuiltin := -f $(srctree)/scripts/Makefile.modbuiltin obj
|
|
|
|
###
|
|
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.dtbinst obj=
|
|
# Usage:
|
|
# $(Q)$(MAKE) $(dtbinst)=dir
|
|
dtbinst := -f $(srctree)/scripts/Makefile.dtbinst obj
|
|
|
|
###
|
|
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.clean obj=
|
|
# Usage:
|
|
# $(Q)$(MAKE) $(clean)=dir
|
|
clean := -f $(srctree)/scripts/Makefile.clean obj
|
|
|
|
# echo command.
|
|
# Short version is used, if $(quiet) equals `quiet_', otherwise full one.
|
|
echo-cmd = $(if $($(quiet)cmd_$(1)),\
|
|
echo ' $(call escsq,$($(quiet)cmd_$(1)))$(echo-why)';)
|
|
|
|
# printing commands
|
|
cmd = @set -e; $(echo-cmd) $(cmd_$(1))
|
|
|
|
###
|
|
# if_changed - execute command if any prerequisite is newer than
|
|
# target, or command line has changed
|
|
# if_changed_dep - as if_changed, but uses fixdep to reveal dependencies
|
|
# including used config symbols
|
|
# if_changed_rule - as if_changed but execute rule instead
|
|
# See Documentation/kbuild/makefiles.rst for more info
|
|
|
|
ifneq ($(KBUILD_NOCMDDEP),1)
|
|
# Check if both commands are the same including their order. Result is empty
|
|
# string if equal. User may override this check using make KBUILD_NOCMDDEP=1
|
|
cmd-check = $(filter-out $(subst $(space),$(space_escape),$(strip $(cmd_$@))), \
|
|
$(subst $(space),$(space_escape),$(strip $(cmd_$1))))
|
|
else
|
|
cmd-check = $(if $(strip $(cmd_$@)),,1)
|
|
endif
|
|
|
|
# Replace >$< with >$$< to preserve $ when reloading the .cmd file
|
|
# (needed for make)
|
|
# Replace >#< with >$(pound)< to avoid starting a comment in the .cmd file
|
|
# (needed for make)
|
|
# Replace >'< with >'\''< to be able to enclose the whole string in '...'
|
|
# (needed for the shell)
|
|
make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
|
|
|
|
# Find any prerequisites that are newer than target or that do not exist.
|
|
# (This is not true for now; $? should contain any non-existent prerequisites,
|
|
# but it does not work as expected when .SECONDARY is present. This seems a bug
|
|
# of GNU Make.)
|
|
# PHONY targets skipped in both cases.
|
|
any-prereq = $(filter-out $(PHONY),$?)
|
|
|
|
# Execute command if command has changed or prerequisite(s) are updated.
|
|
if_changed = $(if $(any-prereq)$(cmd-check), \
|
|
$(cmd); \
|
|
printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
|
|
|
|
# Execute the command and also postprocess generated .d dependencies file.
|
|
if_changed_dep = $(if $(any-prereq)$(cmd-check),$(cmd_and_fixdep),@:)
|
|
|
|
cmd_and_fixdep = \
|
|
$(cmd); \
|
|
scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).cmd;\
|
|
rm -f $(depfile)
|
|
|
|
# Usage: $(call if_changed_rule,foo)
|
|
# Will check if $(cmd_foo) or any of the prerequisites changed,
|
|
# and if so will execute $(rule_foo).
|
|
if_changed_rule = $(if $(any-prereq)$(cmd-check),$(rule_$(1)),@:)
|
|
|
|
###
|
|
# why - tell why a target got built
|
|
# enabled by make V=2
|
|
# Output (listed in the order they are checked):
|
|
# (1) - due to target is PHONY
|
|
# (2) - due to target missing
|
|
# (3) - due to: file1.h file2.h
|
|
# (4) - due to command line change
|
|
# (5) - due to missing .cmd file
|
|
# (6) - due to target not in $(targets)
|
|
# (1) PHONY targets are always build
|
|
# (2) No target, so we better build it
|
|
# (3) Prerequisite is newer than target
|
|
# (4) The command line stored in the file named dir/.target.cmd
|
|
# differed from actual command line. This happens when compiler
|
|
# options changes
|
|
# (5) No dir/.target.cmd file (used to store command line)
|
|
# (6) No dir/.target.cmd file and target not listed in $(targets)
|
|
# This is a good hint that there is a bug in the kbuild file
|
|
ifeq ($(KBUILD_VERBOSE),2)
|
|
why = \
|
|
$(if $(filter $@, $(PHONY)),- due to target is PHONY, \
|
|
$(if $(wildcard $@), \
|
|
$(if $(any-prereq),- due to: $(any-prereq), \
|
|
$(if $(cmd-check), \
|
|
$(if $(cmd_$@),- due to command line change, \
|
|
$(if $(filter $@, $(targets)), \
|
|
- due to missing .cmd file, \
|
|
- due to $(notdir $@) not in $$(targets) \
|
|
) \
|
|
) \
|
|
) \
|
|
), \
|
|
- due to target missing \
|
|
) \
|
|
)
|
|
|
|
echo-why = $(call escsq, $(strip $(why)))
|
|
endif
|
|
|
|
###############################################################################
|
|
#
|
|
# When a Kconfig string contains a filename, it is suitable for
|
|
# passing to shell commands. It is surrounded by double-quotes, and
|
|
# any double-quotes or backslashes within it are escaped by
|
|
# backslashes.
|
|
#
|
|
# This is no use for dependencies or $(wildcard). We need to strip the
|
|
# surrounding quotes and the escaping from quotes and backslashes, and
|
|
# we *do* need to escape any spaces in the string. So, for example:
|
|
#
|
|
# Usage: $(eval $(call config_filename,FOO))
|
|
#
|
|
# Defines FOO_FILENAME based on the contents of the CONFIG_FOO option,
|
|
# transformed as described above to be suitable for use within the
|
|
# makefile.
|
|
#
|
|
# Also, if the filename is a relative filename and exists in the source
|
|
# tree but not the build tree, define FOO_SRCPREFIX as $(srctree)/ to
|
|
# be prefixed to *both* command invocation and dependencies.
|
|
#
|
|
# Note: We also print the filenames in the quiet_cmd_foo text, and
|
|
# perhaps ought to have a version specially escaped for that purpose.
|
|
# But it's only cosmetic, and $(patsubst "%",%,$(CONFIG_FOO)) is good
|
|
# enough. It'll strip the quotes in the common case where there's no
|
|
# space and it's a simple filename, and it'll retain the quotes when
|
|
# there's a space. There are some esoteric cases in which it'll print
|
|
# the wrong thing, but we don't really care. The actual dependencies
|
|
# and commands *do* get it right, with various combinations of single
|
|
# and double quotes, backslashes and spaces in the filenames.
|
|
#
|
|
###############################################################################
|
|
#
|
|
define config_filename
|
|
ifneq ($$(CONFIG_$(1)),"")
|
|
$(1)_FILENAME := $$(subst \\,\,$$(subst \$$(quote),$$(quote),$$(subst $$(space_escape),\$$(space),$$(patsubst "%",%,$$(subst $$(space),$$(space_escape),$$(CONFIG_$(1)))))))
|
|
ifneq ($$(patsubst /%,%,$$(firstword $$($(1)_FILENAME))),$$(firstword $$($(1)_FILENAME)))
|
|
else
|
|
ifeq ($$(wildcard $$($(1)_FILENAME)),)
|
|
ifneq ($$(wildcard $$(srctree)/$$($(1)_FILENAME)),)
|
|
$(1)_SRCPREFIX := $(srctree)/
|
|
endif
|
|
endif
|
|
endif
|
|
endif
|
|
endef
|
|
#
|
|
###############################################################################
|
|
|
|
# delete partially updated (i.e. corrupted) files on error
|
|
.DELETE_ON_ERROR:
|
|
|
|
# do not delete intermediate files automatically
|
|
.SECONDARY:
|