linux/scripts/Kbuild.include

320 lines
12 KiB
Plaintext
Raw Normal View History

# SPDX-License-Identifier: GPL-2.0
####
# kbuild: Generic definitions
# Convenient variables
comma := ,
quote := "
squote := '
empty :=
space := $(empty) $(empty)
kbuild: fix if_change and friends to consider argument order Currently, arg-check is implemented as follows: arg-check = $(strip $(filter-out $(cmd_$(1)), $(cmd_$@)) \ $(filter-out $(cmd_$@), $(cmd_$(1))) ) This does not care about the order of arguments that appear in $(cmd_$(1)) and $(cmd_$@). So, if_changed and friends never rebuild the target if only the argument order is changed. This is a problem when the link order is changed. Apparently, obj-y += foo.o obj-y += bar.o and obj-y += bar.o obj-y += foo.o should be distinguished because the link order determines the probe order of drivers. So, built-in.o should be rebuilt when the order of objects is changed. This commit fixes arg-check to compare the old/current commands including the argument order. Of course, this change has a side effect; Kbuild will react to the change of compile option order. For example, "-DFOO -DBAR" and "-DBAR -DFOO" should give no difference to the build result, but false positive should be better than false negative. I am moving space_escape to the top of Kbuild.include just for a matter of preference. In practical terms, space_escape can be defined after arg-check because arg-check uses "=" flavor, not ":=". Having said that, collecting convenient variables in one place makes sense from the point of readability. Chaining "%%%SPACE%%%" to "_-_SPACE_-_" is also a matter of taste at this point. Actually, it can be arbitrary as long as it is an unlikely used string. The only problem I see in "%%%SPACE%%%" is that "%" is a special character in "$(patsubst ...)" context. This commit just uses "$(subst ...)" for arg-check, but I am fixing it now in case we might want to use it in $(patsubst ...) context in the future. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-05-07 06:48:26 +00:00
space_escape := _-_SPACE_-_
Kbuild: fix # escaping in .cmd files for future Make I tried building using a freshly built Make (4.2.1-69-g8a731d1), but already the objtool build broke with orc_dump.c: In function ‘orc_dump’: orc_dump.c:106:2: error: ‘elf_getshnum’ is deprecated [-Werror=deprecated-declarations] if (elf_getshdrnum(elf, &nr_sections)) { Turns out that with that new Make, the backslash was not removed, so cpp didn't see a #include directive, grep found nothing, and -DLIBELF_USE_DEPRECATED was wrongly put in CFLAGS. Now, that new Make behaviour is documented in their NEWS file: * WARNING: Backward-incompatibility! Number signs (#) appearing inside a macro reference or function invocation no longer introduce comments and should not be escaped with backslashes: thus a call such as: foo := $(shell echo '#') is legal. Previously the number sign needed to be escaped, for example: foo := $(shell echo '\#') Now this latter will resolve to "\#". If you want to write makefiles portable to both versions, assign the number sign to a variable: C := \# foo := $(shell echo '$C') This was claimed to be fixed in 3.81, but wasn't, for some reason. To detect this change search for 'nocomment' in the .FEATURES variable. This also fixes up the two make-cmd instances to replace # with $(pound) rather than with \#. There might very well be other places that need similar fixup in preparation for whatever future Make release contains the above change, but at least this builds an x86_64 defconfig with the new make. Link: https://bugzilla.kernel.org/show_bug.cgi?id=197847 Cc: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-04-08 21:35:28 +00:00
pound := \#
###
# Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
dot-target = $(dir $@).$(notdir $@)
###
# The temporary file to save gcc -MMD generated dependencies must not
# contain a comma
depfile = $(subst $(comma),_,$(dot-target).d)
###
# filename of target with directory and extension stripped
basetarget = $(basename $(notdir $@))
###
# real prerequisites without phony targets
real-prereqs = $(filter-out $(PHONY), $^)
###
# Escape single quote for use in echo statements
escsq = $(subst $(squote),'\$(squote)',$1)
###
# Quote a string to pass it to C files. foo => '"foo"'
stringify = $(squote)$(quote)$1$(quote)$(squote)
###
# Easy method for doing a status message
kecho := :
quiet_kecho := echo
silent_kecho := :
kecho := $($(quiet)kecho)
###
# filechk is used to check if the content of a generated file is updated.
# Sample usage:
#
# filechk_sample = echo $(KERNELRELEASE)
# version.h: FORCE
# $(call filechk,sample)
#
# The rule defined shall write to stdout the content of the new file.
# The existing file will be compared with the new one.
# - If no file exist it is created
# - If the content differ the new file is used
# - If they are equal no change, and no timestamp update
define filechk
$(Q)set -e; \
mkdir -p $(dir $@); \
trap "rm -f $(dot-target).tmp" EXIT; \
{ $(filechk_$(1)); } > $(dot-target).tmp; \
if [ ! -r $@ ] || ! cmp -s $@ $(dot-target).tmp; then \
$(kecho) ' UPD $@'; \
mv -f $(dot-target).tmp $@; \
fi
endef
######
# gcc support functions
# See documentation in Documentation/kbuild/makefiles.rst
# cc-cross-prefix
# Usage: CROSS_COMPILE := $(call cc-cross-prefix, m68k-linux-gnu- m68k-linux-)
# Return first <prefix> where a <prefix>gcc is found in PATH.
# If no gcc found in PATH with listed prefixes return nothing
kbuild: use more portable 'command -v' for cc-cross-prefix To print the pathname that will be used by shell in the current environment, 'command -v' is a standardized way. [1] 'which' is also often used in scripts, but it is less portable. When I worked on commit bd55f96fa9fc ("kbuild: refactor cc-cross-prefix implementation"), I was eager to use 'command -v' but it did not work. (The reason is explained below.) I kept 'which' as before but got rid of '> /dev/null 2>&1' as I thought it was no longer needed. Sorry, I was wrong. It works well on my Ubuntu machine, but Alexey Brodkin reports noisy warnings on CentOS7 when 'which' fails to find the given command in the PATH environment. $ which foo which: no foo in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) Given that behavior of 'which' depends on system (and it may not be installed by default), I want to try 'command -v' once again. The specification [1] clearly describes the behavior of 'command -v' when the given command is not found: Otherwise, no output shall be written and the exit status shall reflect that the name was not found. However, we need a little magic to use 'command -v' from Make. $(shell ...) passes the argument to a subshell for execution, and returns the standard output of the command. Here is a trick. GNU Make may optimize this by executing the command directly instead of forking a subshell, if no shell special characters are found in the command and omitting the subshell will not change the behavior. In this case, no shell special character is used. So, Make will try to run it directly. However, 'command' is a shell-builtin command, then Make would fail to find it in the PATH environment: $ make ARCH=m68k defconfig make: command: Command not found make: command: Command not found make: command: Command not found In fact, Make has a table of shell-builtin commands because it must ask the shell to execute them. Until recently, 'command' was missing in the table. This issue was fixed by the following commit: | commit 1af314465e5dfe3e8baa839a32a72e83c04f26ef | Author: Paul Smith <psmith@gnu.org> | Date: Sun Nov 12 18:10:28 2017 -0500 | | * job.c: Add "command" as a known shell built-in. | | This is not a POSIX shell built-in but it's common in UNIX shells. | Reported by Nick Bowler <nbowler@draconx.ca>. Because the latest release is GNU Make 4.2.1 in 2016, this commit is not included in any released versions. (But some distributions may have back-ported it.) We need to trick Make to spawn a subshell. There are various ways to do so: 1) Use a shell special character '~' as dummy $(shell : ~; command -v $(c)gcc) 2) Use a variable reference that always expands to the empty string (suggested by David Laight) $(shell command$${x:+} -v $(c)gcc) 3) Use redirect $(shell command -v $(c)gcc 2>/dev/null) I chose 3) to not confuse people. The stderr would not be polluted anyway, but it will provide extra safety, and is easy to understand. Tested on Make 3.81, 3.82, 4.0, 4.1, 4.2, 4.2.1 [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html Fixes: bd55f96fa9fc ("kbuild: refactor cc-cross-prefix implementation") Cc: linux-stable <stable@vger.kernel.org> # 5.1 Reported-by: Alexey Brodkin <abrodkin@synopsys.com> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Tested-by: Alexey Brodkin <abrodkin@synopsys.com>
2019-06-06 04:13:58 +00:00
#
# Note: '2>/dev/null' is here to force Make to invoke a shell. Otherwise, it
# would try to directly execute the shell builtin 'command'. This workaround
# should be kept for a long time since this issue was fixed only after the
# GNU Make 4.2.1 release.
cc-cross-prefix = $(firstword $(foreach c, $(1), \
$(if $(shell command -v -- $(c)gcc 2>/dev/null), $(c))))
# output directory for tests below
TMPOUT = $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/).tmp_$$$$
# try-run
# Usage: option = $(call try-run, $(CC)...-o "$$TMP",option-ok,otherwise)
# Exit code chooses option. "$$TMP" serves as a temporary file and is
# automatically cleaned up.
try-run = $(shell set -e; \
TMP=$(TMPOUT)/tmp; \
TMPO=$(TMPOUT)/tmp.o; \
mkdir -p $(TMPOUT); \
trap "rm -rf $(TMPOUT)" EXIT; \
if ($(1)) >/dev/null 2>&1; \
then echo "$(2)"; \
else echo "$(3)"; \
fi)
# as-option
# Usage: cflags-y += $(call as-option,-Wa$(comma)-isa=foo,)
as-option = $(call try-run,\
$(CC) $(KBUILD_CFLAGS) $(1) -c -x assembler /dev/null -o "$$TMP",$(1),$(2))
# as-instr
# Usage: cflags-y += $(call as-instr,instr,option1,option2)
as-instr = $(call try-run,\
printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
# __cc-option
# Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
__cc-option = $(call try-run,\
$(1) -Werror $(2) $(3) -c -x c /dev/null -o "$$TMP",$(3),$(4))
# cc-option
# Usage: cflags-y += $(call cc-option,-march=winchip-c6,-march=i586)
cc-option = $(call __cc-option, $(CC),\
$(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS),$(1),$(2))
# cc-option-yn
# Usage: flag := $(call cc-option-yn,-march=winchip-c6)
cc-option-yn = $(call try-run,\
$(CC) -Werror $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(1) -c -x c /dev/null -o "$$TMP",y,n)
# cc-disable-warning
# Usage: cflags-y += $(call cc-disable-warning,unused-but-set-variable)
cc-disable-warning = $(call try-run,\
$(CC) -Werror $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
# cc-ifversion
# Usage: EXTRA_CFLAGS += $(call cc-ifversion, -lt, 0402, -O1)
cc-ifversion = $(shell [ $(CONFIG_GCC_VERSION)0 $(1) $(2)000 ] && echo $(3) || echo $(4))
# ld-option
# Usage: KBUILD_LDFLAGS += $(call ld-option, -X, -Y)
ld-option = $(call try-run, $(LD) $(KBUILD_LDFLAGS) $(1) -v,$(1),$(2),$(3))
# ld-ifversion
# Usage: $(call ld-ifversion, -ge, 22252, y)
ld-ifversion = $(shell [ $(CONFIG_LD_VERSION)0 $(1) $(2)0 ] && echo $(3) || echo $(4))
######
[PATCH] vDSO hash-style fix The latest toolchains can produce a new ELF section in DSOs and dynamically-linked executables. The new section ".gnu.hash" replaces ".hash", and allows for more efficient runtime symbol lookups by the dynamic linker. The new ld option --hash-style={sysv|gnu|both} controls whether to produce the old ".hash", the new ".gnu.hash", or both. In some new systems such as Fedora Core 6, gcc by default passes --hash-style=gnu to the linker, so that a standard invocation of "gcc -shared" results in producing a DSO with only ".gnu.hash". The new ".gnu.hash" sections need to be dealt with the same way as ".hash" sections in all respects; only the dynamic linker cares about their contents. To work with older dynamic linkers (i.e. preexisting releases of glibc), a binary must have the old ".hash" section. The --hash-style=both option produces binaries that a new dynamic linker can use more efficiently, but an old dynamic linker can still handle. The new section runs afoul of the custom linker scripts used to build vDSO images for the kernel. On ia64, the failure mode for this is a boot-time panic because the vDSO's PT_IA_64_UNWIND segment winds up ill-formed. This patch addresses the problem in two ways. First, it mentions ".gnu.hash" in all the linker scripts alongside ".hash". This produces correct vDSO images with --hash-style=sysv (or old tools), with --hash-style=gnu, or with --hash-style=both. Second, it passes the --hash-style=sysv option when building the vDSO images, so that ".gnu.hash" is not actually produced. This is the most conservative choice for compatibility with any old userland. There is some concern that some ancient glibc builds (though not any known old production system) might choke on --hash-style=both binaries. The optimizations provided by the new style of hash section do not really matter for a DSO with a tiny number of symbols, as the vDSO has. If someone wants to use =gnu or =both for their vDSO builds and worry less about that compatibility, just change the option and the linker script changes will make any choice work fine. Signed-off-by: Roland McGrath <roland@redhat.com> Cc: "Luck, Tony" <tony.luck@intel.com> Cc: Kyle McMartin <kyle@mcmartin.ca> Cc: Paul Mackerras <paulus@samba.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Jeff Dike <jdike@addtoit.com> Cc: Andi Kleen <ak@muc.de> Cc: Sam Ravnborg <sam@ravnborg.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-07-30 10:04:06 +00:00
###
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.build obj=
# Usage:
# $(Q)$(MAKE) $(build)=dir
build := -f $(srctree)/scripts/Makefile.build obj
###
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.dtbinst obj=
# Usage:
# $(Q)$(MAKE) $(dtbinst)=dir
dtbinst := -f $(srctree)/scripts/Makefile.dtbinst obj
###
# Shorthand for $(Q)$(MAKE) -f scripts/Makefile.clean obj=
# Usage:
# $(Q)$(MAKE) $(clean)=dir
clean := -f $(srctree)/scripts/Makefile.clean obj
# echo command.
# Short version is used, if $(quiet) equals `quiet_', otherwise full one.
echo-cmd = $(if $($(quiet)cmd_$(1)),\
echo ' $(call escsq,$($(quiet)cmd_$(1)))$(echo-why)';)
# printing commands
kbuild: change if_changed_rule for multi-line recipe The 'define' ... 'endef' directive is useful to confine a series of shell commands into a single macro: define foo [action1] [action2] [action3] endif Each action is executed in a separate subshell. However, rule_cc_o_c and rule_as_o_S in scripts/Makefile.build are written as follows (with a trailing semicolon in each cmd_*): define rule_cc_o_c [action1] ; \ [action2] ; \ [action3] ; endef All shell commands are concatenated with '; \' so that it looks like a single command from the Makefile point of view. This does not exploit the benefits of 'define' ... 'endef' form because a single shell command can be more simply written, like this: rule_cc_o_c = \ [action1] ; \ [action2] ; \ [action3] ; I guess the intention for the command concatenation was to let the '@set -e' in if_changed_rule cover all the commands. We can improve the readability by moving '@set -e' to the 'cmd' macro. The combo of $(call echo-cmd,*) $(cmd_*) in rule_cc_o_c and rule_as_o_S have been replaced with $(call cmd,*). The trailing back-slashes have been removed. Here is a note about the performance: the commands in rule_cc_o_c and rule_as_o_S were previously executed all together in a single subshell, but now each line in a separate subshell. This means Make will spawn extra subshells [1]. I measured the build performance for x86_64_defconfig + CONFIG_MODVERSIONS + CONFIG_TRIM_UNUSED_KSYMS and I saw slight performance regression, but I believe code readability and maintainability wins. [1] Precisely, GNU Make may optimize this by executing the command directly instead of forking a subshell, if no shell special characters are found in the command line and omitting the subshell will not change the behavior. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-11-30 01:05:27 +00:00
cmd = @set -e; $(echo-cmd) $(cmd_$(1))
###
# if_changed - execute command if any prerequisite is newer than
# target, or command line has changed
# if_changed_dep - as if_changed, but uses fixdep to reveal dependencies
# including used config symbols
# if_changed_rule - as if_changed but execute rule instead
# See Documentation/kbuild/makefiles.rst for more info
ifneq ($(KBUILD_NOCMDDEP),1)
# Check if both commands are the same including their order. Result is empty
kbuild: fix if_change and friends to consider argument order Currently, arg-check is implemented as follows: arg-check = $(strip $(filter-out $(cmd_$(1)), $(cmd_$@)) \ $(filter-out $(cmd_$@), $(cmd_$(1))) ) This does not care about the order of arguments that appear in $(cmd_$(1)) and $(cmd_$@). So, if_changed and friends never rebuild the target if only the argument order is changed. This is a problem when the link order is changed. Apparently, obj-y += foo.o obj-y += bar.o and obj-y += bar.o obj-y += foo.o should be distinguished because the link order determines the probe order of drivers. So, built-in.o should be rebuilt when the order of objects is changed. This commit fixes arg-check to compare the old/current commands including the argument order. Of course, this change has a side effect; Kbuild will react to the change of compile option order. For example, "-DFOO -DBAR" and "-DBAR -DFOO" should give no difference to the build result, but false positive should be better than false negative. I am moving space_escape to the top of Kbuild.include just for a matter of preference. In practical terms, space_escape can be defined after arg-check because arg-check uses "=" flavor, not ":=". Having said that, collecting convenient variables in one place makes sense from the point of readability. Chaining "%%%SPACE%%%" to "_-_SPACE_-_" is also a matter of taste at this point. Actually, it can be arbitrary as long as it is an unlikely used string. The only problem I see in "%%%SPACE%%%" is that "%" is a special character in "$(patsubst ...)" context. This commit just uses "$(subst ...)" for arg-check, but I am fixing it now in case we might want to use it in $(patsubst ...) context in the future. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-05-07 06:48:26 +00:00
# string if equal. User may override this check using make KBUILD_NOCMDDEP=1
cmd-check = $(filter-out $(subst $(space),$(space_escape),$(strip $(cmd_$@))), \
kbuild: fix if_change and friends to consider argument order Currently, arg-check is implemented as follows: arg-check = $(strip $(filter-out $(cmd_$(1)), $(cmd_$@)) \ $(filter-out $(cmd_$@), $(cmd_$(1))) ) This does not care about the order of arguments that appear in $(cmd_$(1)) and $(cmd_$@). So, if_changed and friends never rebuild the target if only the argument order is changed. This is a problem when the link order is changed. Apparently, obj-y += foo.o obj-y += bar.o and obj-y += bar.o obj-y += foo.o should be distinguished because the link order determines the probe order of drivers. So, built-in.o should be rebuilt when the order of objects is changed. This commit fixes arg-check to compare the old/current commands including the argument order. Of course, this change has a side effect; Kbuild will react to the change of compile option order. For example, "-DFOO -DBAR" and "-DBAR -DFOO" should give no difference to the build result, but false positive should be better than false negative. I am moving space_escape to the top of Kbuild.include just for a matter of preference. In practical terms, space_escape can be defined after arg-check because arg-check uses "=" flavor, not ":=". Having said that, collecting convenient variables in one place makes sense from the point of readability. Chaining "%%%SPACE%%%" to "_-_SPACE_-_" is also a matter of taste at this point. Actually, it can be arbitrary as long as it is an unlikely used string. The only problem I see in "%%%SPACE%%%" is that "%" is a special character in "$(patsubst ...)" context. This commit just uses "$(subst ...)" for arg-check, but I am fixing it now in case we might want to use it in $(patsubst ...) context in the future. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-05-07 06:48:26 +00:00
$(subst $(space),$(space_escape),$(strip $(cmd_$1))))
else
cmd-check = $(if $(strip $(cmd_$@)),,1)
endif
# Replace >$< with >$$< to preserve $ when reloading the .cmd file
# (needed for make)
Kbuild: fix # escaping in .cmd files for future Make I tried building using a freshly built Make (4.2.1-69-g8a731d1), but already the objtool build broke with orc_dump.c: In function ‘orc_dump’: orc_dump.c:106:2: error: ‘elf_getshnum’ is deprecated [-Werror=deprecated-declarations] if (elf_getshdrnum(elf, &nr_sections)) { Turns out that with that new Make, the backslash was not removed, so cpp didn't see a #include directive, grep found nothing, and -DLIBELF_USE_DEPRECATED was wrongly put in CFLAGS. Now, that new Make behaviour is documented in their NEWS file: * WARNING: Backward-incompatibility! Number signs (#) appearing inside a macro reference or function invocation no longer introduce comments and should not be escaped with backslashes: thus a call such as: foo := $(shell echo '#') is legal. Previously the number sign needed to be escaped, for example: foo := $(shell echo '\#') Now this latter will resolve to "\#". If you want to write makefiles portable to both versions, assign the number sign to a variable: C := \# foo := $(shell echo '$C') This was claimed to be fixed in 3.81, but wasn't, for some reason. To detect this change search for 'nocomment' in the .FEATURES variable. This also fixes up the two make-cmd instances to replace # with $(pound) rather than with \#. There might very well be other places that need similar fixup in preparation for whatever future Make release contains the above change, but at least this builds an x86_64 defconfig with the new make. Link: https://bugzilla.kernel.org/show_bug.cgi?id=197847 Cc: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-04-08 21:35:28 +00:00
# Replace >#< with >$(pound)< to avoid starting a comment in the .cmd file
# (needed for make)
# Replace >'< with >'\''< to be able to enclose the whole string in '...'
# (needed for the shell)
Kbuild: fix # escaping in .cmd files for future Make I tried building using a freshly built Make (4.2.1-69-g8a731d1), but already the objtool build broke with orc_dump.c: In function ‘orc_dump’: orc_dump.c:106:2: error: ‘elf_getshnum’ is deprecated [-Werror=deprecated-declarations] if (elf_getshdrnum(elf, &nr_sections)) { Turns out that with that new Make, the backslash was not removed, so cpp didn't see a #include directive, grep found nothing, and -DLIBELF_USE_DEPRECATED was wrongly put in CFLAGS. Now, that new Make behaviour is documented in their NEWS file: * WARNING: Backward-incompatibility! Number signs (#) appearing inside a macro reference or function invocation no longer introduce comments and should not be escaped with backslashes: thus a call such as: foo := $(shell echo '#') is legal. Previously the number sign needed to be escaped, for example: foo := $(shell echo '\#') Now this latter will resolve to "\#". If you want to write makefiles portable to both versions, assign the number sign to a variable: C := \# foo := $(shell echo '$C') This was claimed to be fixed in 3.81, but wasn't, for some reason. To detect this change search for 'nocomment' in the .FEATURES variable. This also fixes up the two make-cmd instances to replace # with $(pound) rather than with \#. There might very well be other places that need similar fixup in preparation for whatever future Make release contains the above change, but at least this builds an x86_64 defconfig with the new make. Link: https://bugzilla.kernel.org/show_bug.cgi?id=197847 Cc: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-04-08 21:35:28 +00:00
make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
kbuild: drop $(wildcard $^) check in if_changed* for faster rebuild The incremental build of Linux kernel is pretty slow when lots of objects are compiled. The rebuild of allmodconfig may take a few minutes even when none of the objects needs to be rebuilt. The time-consuming part in the incremental build is the evaluation of if_changed* macros since they are used in the recipes to compile C and assembly source files into objects. I notice the following code in if_changed* is expensive: $(filter-out $(PHONY) $(wildcard $^),$^) In the incremental build, every object has its .*.cmd file, which contains the auto-generated list of included headers. So, $^ are expanded into the long list of the source file + included headers, and $(wildcard $^) checks whether they exist. It may not be clear why this check exists there. Here is the record of my research. [1] The first code addition into Kbuild This code dates back to 2002. It is the pre-git era. So, I copy-pasted it from the historical git tree. | commit 4a6db0791528c220655b063cf13fefc8470dbfee (HEAD) | Author: Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de> | Date: Mon Jun 17 00:22:37 2002 -0500 | | kbuild: Handle removed headers | | New and old way to handle dependencies would choke when a file | #include'd by other files was removed, since the dependency on it was | still recorded, but since it was gone, make has no idea what to do about | it (and would complain with "No rule to make <file> ...") | | We now add targets for all the previously included files, so make will | just ignore them if they disappear. | | diff --git a/Rules.make b/Rules.make | index 6ef827d3df39..7db5301ea7db 100644 | --- a/Rules.make | +++ b/Rules.make | @@ -446,7 +446,7 @@ if_changed = $(if $(strip $? \ | # execute the command and also postprocess generated .d dependencies | # file | | -if_changed_dep = $(if $(strip $? \ | +if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\ | $(filter-out $(cmd_$(1)),$(cmd_$@))\ | $(filter-out $(cmd_$@),$(cmd_$(1)))),\ | @set -e; \ | diff --git a/scripts/fixdep.c b/scripts/fixdep.c | index b5d7bee8efc7..db45bd1888c0 100644 | --- a/scripts/fixdep.c | +++ b/scripts/fixdep.c | @@ -292,7 +292,7 @@ void parse_dep_file(void *map, size_t len) | exit(1); | } | memcpy(s, m, p-m); s[p-m] = 0; | - printf("%s: \\\n", target); | + printf("deps_%s := \\\n", target); | m = p+1; | | clear_config(); | @@ -314,7 +314,8 @@ void parse_dep_file(void *map, size_t len) | } | m = p + 1; | } | - printf("\n"); | + printf("\n%s: $(deps_%s)\n\n", target, target); | + printf("$(deps_%s):\n", target); | } | | void print_deps(void) The "No rule to make <file> ..." error can be solved by passing -MP to the compiler, but I think the detection of header removal is a good feature. When a header is removed, all source files that previously included it should be re-compiled. This makes sure we has correctly got rid of #include directives of it. This is also related with the behavior of $?. The GNU Make manual says: $? The names of all the prerequisites that are newer than the target, with spaces between them. This does not explain whether a non-existent prerequisite is considered to be newer than the target. At this point of time, GNU Make 3.7x was used, where the $? did not include non-existent prerequisites. Therefore, $(filter-out FORCE $(wildcard $^),$^) was useful to detect the header removal, and to rebuild the related objects if it is the case. [2] Change of $? behavior Later, the behavior of $? was changed (fixed) to include prerequisites that did not exist. First, GNU Make commit 64e16d6c00a5 ("Various changes getting ready for the release of 3.81.") changed it, but in the release test of 3.81, it turned out to break the kernel build. See these: - http://lists.gnu.org/archive/html/bug-make/2006-03/msg00003.html - https://savannah.gnu.org/bugs/?16002 - https://savannah.gnu.org/bugs/?16051 Then, GNU Make commit 6d8d9b74d9c5 ("Numerous updates to tests for issues found on Cygwin and Windows.") reverted it for the 3.81 release to give Linux kernel time to adjust to the new behavior. After the 3.81 release, GNU Make commit 7595f38f62af ("Fixed a number of documentation bugs, plus some build/install issues:") re-added it. [3] Adjustment to the new $? behavior on Kbuild side Meanwhile, the kernel build was changed by commit 4f1933620f57 ("kbuild: change kbuild to not rely on incorrect GNU make behavior") to adjust to the new $? behavior. [4] GNU Make 3.82 released in 2010 GNU Make 3.82 was the first release that integrated the correct $? behavior. At this point, Kbuild dealt with GNU Make versions with different $? behaviors. 3.81 or older: $? does not contain any non-existent prerequisite. $(filter-out $(PHONY) $(wildcard $^),$^) was useful to detect removed include headers. 3.82 or newer: $? contains non-existent prerequisites. When a header is removed, it appears in $?. $(filter-out $(PHONY) $(wildcard $^),$^) became a redundant check. With the correct $? behavior, we could have dropped the expensive check for 3.82 or later, but we did not. (Maybe nobody noticed this optimization.) [5] The .SECONDARY special target trips up $? Some time later, I noticed $? did not work as expected under some circumstances. As above, $? should contain non-existent prerequisites, but the ones specified as SECONDARY do not appear in $?. I asked this in GNU Make ML, and it seems a bug: https://lists.gnu.org/archive/html/bug-make/2019-01/msg00001.html Since commit 8e9b61b293d9 ("kbuild: move .SECONDARY special target to Kbuild.include"), all files, including headers listed in .*.cmd files, are treated as secondary. So, we are back into the incorrect $? behavior. If we Kbuild want to react to the header removal, we need to keep $(filter-out $(PHONY) $(wildcard $^),$^) but this makes the rebuild so slow. [Summary] - I believe noticing the header removal and recompiling related objects is a nice feature for the build system. - If $? worked correctly, $(filter-out $(PHONY),$?) would be enough to detect the header removal. - Currently, $? does not work correctly when used with .SECONDARY, and Kbuild is hit by this bug. - I filed a bug report for this, but not fixed yet as of writing. - Currently, the header removal is detected by the following expensive code: $(filter-out $(PHONY) $(wildcard $^),$^) - I do not want to revert commit 8e9b61b293d9 ("kbuild: move .SECONDARY special target to Kbuild.include"). Specifying .SECONDARY globally is clean, and it matches to the Kbuild policy. This commit proactively removes the expensive check since it makes the incremental build faster. A downside is Kbuild will no longer be able to notice the header removal. You can confirm it by the full-build followed by a header removal, and then re-build. $ make defconfig all [ full build ] $ rm include/linux/device.h $ make CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh DESCEND objtool CHK include/generated/compile.h Kernel: arch/x86/boot/bzImage is ready (#11) Building modules, stage 2. MODPOST 12 modules Previously, Kbuild noticed a missing header and emits a build error. Now, Kbuild is fine with it. This is an unusual corner-case, not a big deal. Once the $? bug is fixed in GNU Make, everything will work fine. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2019-11-07 15:09:44 +00:00
# Find any prerequisites that are newer than target or that do not exist.
# (This is not true for now; $? should contain any non-existent prerequisites,
# but it does not work as expected when .SECONDARY is present. This seems a bug
# of GNU Make.)
# PHONY targets skipped in both cases.
newer-prereqs = $(filter-out $(PHONY),$?)
# Execute command if command has changed or prerequisite(s) are updated.
if_changed = $(if $(newer-prereqs)$(cmd-check), \
$(cmd); \
printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
# Execute the command and also postprocess generated .d dependencies file.
if_changed_dep = $(if $(newer-prereqs)$(cmd-check),$(cmd_and_fixdep),@:)
kbuild: add fine grained build dependencies for exported symbols Like with kconfig options, we now have the ability to compile in and out individual EXPORT_SYMBOL() declarations based on the content of include/generated/autoksyms.h. However we don't want the entire world to be rebuilt whenever that file is touched. Let's apply the same build dependency trick used for CONFIG_* symbols where the time stamp of empty files whose paths matching those symbols is used to trigger fine grained rebuilds. In our case the key is the symbol name passed to EXPORT_SYMBOL(). However, unlike config options, we cannot just use fixdep to parse the source code for EXPORT_SYMBOL(ksym) because several variants exist and parsing them all in a separate tool, and keeping it in synch, is not trivially maintainable. Furthermore, there are variants such as EXPORT_SYMBOL_GPL(pci_user_read_config_##size); that are instanciated via a macro for which we can't easily determine the actual exported symbol name(s) short of actually running the preprocessor on them. Storing the symbol name string in a special ELF section doesn't work for targets that output assembly or preprocessed source. So the best way is really to leverage the preprocessor by having it output actual symbol names anchored by a special sequence that can be easily filtered out. Then the list of symbols is simply fed to fixdep to be merged with the other dependencies. That implies the preprocessor is executed twice for each source file. A previous attempt relied on a warning pragma for each EXPORT_SYMBOL() instance that was filtered apart from stderr by the build system with a sed script during the actual compilation pass. Unfortunately the preprocessor/compiler diagnostic output isn't stable between versions and this solution, although more efficient, was deemed too fragile. Because of the lowercasing performed by fixdep, there might be name collisions triggering spurious rebuilds for similar symbols. But this shouldn't be a big issue in practice. (This is the case for CONFIG_* symbols and I didn't want to be different here, whatever the original reason for doing so.) To avoid needless build overhead, the exported symbol name gathering is performed only when CONFIG_TRIM_UNUSED_KSYMS is selected. Signed-off-by: Nicolas Pitre <nico@linaro.org> Acked-by: Rusty Russell <rusty@rustcorp.com.au>
2016-01-22 18:41:57 +00:00
cmd_and_fixdep = \
kbuild: change if_changed_rule for multi-line recipe The 'define' ... 'endef' directive is useful to confine a series of shell commands into a single macro: define foo [action1] [action2] [action3] endif Each action is executed in a separate subshell. However, rule_cc_o_c and rule_as_o_S in scripts/Makefile.build are written as follows (with a trailing semicolon in each cmd_*): define rule_cc_o_c [action1] ; \ [action2] ; \ [action3] ; endef All shell commands are concatenated with '; \' so that it looks like a single command from the Makefile point of view. This does not exploit the benefits of 'define' ... 'endef' form because a single shell command can be more simply written, like this: rule_cc_o_c = \ [action1] ; \ [action2] ; \ [action3] ; I guess the intention for the command concatenation was to let the '@set -e' in if_changed_rule cover all the commands. We can improve the readability by moving '@set -e' to the 'cmd' macro. The combo of $(call echo-cmd,*) $(cmd_*) in rule_cc_o_c and rule_as_o_S have been replaced with $(call cmd,*). The trailing back-slashes have been removed. Here is a note about the performance: the commands in rule_cc_o_c and rule_as_o_S were previously executed all together in a single subshell, but now each line in a separate subshell. This means Make will spawn extra subshells [1]. I measured the build performance for x86_64_defconfig + CONFIG_MODVERSIONS + CONFIG_TRIM_UNUSED_KSYMS and I saw slight performance regression, but I believe code readability and maintainability wins. [1] Precisely, GNU Make may optimize this by executing the command directly instead of forking a subshell, if no shell special characters are found in the command line and omitting the subshell will not change the behavior. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-11-30 01:05:27 +00:00
$(cmd); \
kbuild: let fixdep directly write to .*.cmd files Currently, fixdep writes dependencies to .*.tmp, which is renamed to .*.cmd after everything succeeds. This is a very safe way to avoid corrupted .*.cmd files. The if_changed_dep has carried this safety mechanism since it was added in 2002. If fixdep fails for some reasons or a user terminates the build while fixdep is running, the incomplete output from the fixdep could be troublesome. This is my insight about some bad scenarios: [1] If the compiler succeeds to generate *.o file, but fixdep fails to write necessary dependencies to .*.cmd file, Make will miss to rebuild the object when headers or CONFIG options are changed. In this case, fixdep should not generate .*.cmd file at all so that 'arg-check' will surely trigger the rebuild of the object. [2] A partially constructed .*.cmd file may not be a syntactically correct makefile. The next time Make runs, it would include it, then fail to parse it. Once this happens, 'make clean' is be the only way to fix it. In fact, [1] is no longer a problem since commit 9c2af1c7377a ("kbuild: add .DELETE_ON_ERROR special target"). Make deletes a target file on any failure in its recipe. Because fixdep is a part of the recipe of *.o target, if it fails, the *.o is deleted anyway. However, I am a bit worried about the slight possibility of [2]. So, here is a solution. Let fixdep directly write to a .*.cmd file, but allow makefiles to include it only when its corresponding target exists. This effectively reverts commit 2982c953570b ("kbuild: remove redundant $(wildcard ...) for cmd_files calculation"), and commit 00d78ab2ba75 ("kbuild: remove dead code in cmd_files calculation in top Makefile") because now we must check the presence of targets. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-11-30 01:05:22 +00:00
scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).cmd;\
rm -f $(depfile)
kbuild: add fine grained build dependencies for exported symbols Like with kconfig options, we now have the ability to compile in and out individual EXPORT_SYMBOL() declarations based on the content of include/generated/autoksyms.h. However we don't want the entire world to be rebuilt whenever that file is touched. Let's apply the same build dependency trick used for CONFIG_* symbols where the time stamp of empty files whose paths matching those symbols is used to trigger fine grained rebuilds. In our case the key is the symbol name passed to EXPORT_SYMBOL(). However, unlike config options, we cannot just use fixdep to parse the source code for EXPORT_SYMBOL(ksym) because several variants exist and parsing them all in a separate tool, and keeping it in synch, is not trivially maintainable. Furthermore, there are variants such as EXPORT_SYMBOL_GPL(pci_user_read_config_##size); that are instanciated via a macro for which we can't easily determine the actual exported symbol name(s) short of actually running the preprocessor on them. Storing the symbol name string in a special ELF section doesn't work for targets that output assembly or preprocessed source. So the best way is really to leverage the preprocessor by having it output actual symbol names anchored by a special sequence that can be easily filtered out. Then the list of symbols is simply fed to fixdep to be merged with the other dependencies. That implies the preprocessor is executed twice for each source file. A previous attempt relied on a warning pragma for each EXPORT_SYMBOL() instance that was filtered apart from stderr by the build system with a sed script during the actual compilation pass. Unfortunately the preprocessor/compiler diagnostic output isn't stable between versions and this solution, although more efficient, was deemed too fragile. Because of the lowercasing performed by fixdep, there might be name collisions triggering spurious rebuilds for similar symbols. But this shouldn't be a big issue in practice. (This is the case for CONFIG_* symbols and I didn't want to be different here, whatever the original reason for doing so.) To avoid needless build overhead, the exported symbol name gathering is performed only when CONFIG_TRIM_UNUSED_KSYMS is selected. Signed-off-by: Nicolas Pitre <nico@linaro.org> Acked-by: Rusty Russell <rusty@rustcorp.com.au>
2016-01-22 18:41:57 +00:00
# Usage: $(call if_changed_rule,foo)
# Will check if $(cmd_foo) or any of the prerequisites changed,
# and if so will execute $(rule_foo).
if_changed_rule = $(if $(newer-prereqs)$(cmd-check),$(rule_$(1)),@:)
###
# why - tell why a target got built
# enabled by make V=2
# Output (listed in the order they are checked):
# (1) - due to target is PHONY
# (2) - due to target missing
# (3) - due to: file1.h file2.h
# (4) - due to command line change
# (5) - due to missing .cmd file
# (6) - due to target not in $(targets)
# (1) PHONY targets are always build
# (2) No target, so we better build it
# (3) Prerequisite is newer than target
# (4) The command line stored in the file named dir/.target.cmd
# differed from actual command line. This happens when compiler
# options changes
# (5) No dir/.target.cmd file (used to store command line)
# (6) No dir/.target.cmd file and target not listed in $(targets)
# This is a good hint that there is a bug in the kbuild file
ifeq ($(KBUILD_VERBOSE),2)
why = \
$(if $(filter $@, $(PHONY)),- due to target is PHONY, \
$(if $(wildcard $@), \
$(if $(newer-prereqs),- due to: $(newer-prereqs), \
$(if $(cmd-check), \
$(if $(cmd_$@),- due to command line change, \
$(if $(filter $@, $(targets)), \
- due to missing .cmd file, \
- due to $(notdir $@) not in $$(targets) \
) \
) \
) \
), \
- due to target missing \
) \
)
echo-why = $(call escsq, $(strip $(why)))
endif
###############################################################################
#
# When a Kconfig string contains a filename, it is suitable for
# passing to shell commands. It is surrounded by double-quotes, and
# any double-quotes or backslashes within it are escaped by
# backslashes.
#
# This is no use for dependencies or $(wildcard). We need to strip the
# surrounding quotes and the escaping from quotes and backslashes, and
# we *do* need to escape any spaces in the string. So, for example:
#
# Usage: $(eval $(call config_filename,FOO))
#
# Defines FOO_FILENAME based on the contents of the CONFIG_FOO option,
# transformed as described above to be suitable for use within the
# makefile.
#
# Also, if the filename is a relative filename and exists in the source
# tree but not the build tree, define FOO_SRCPREFIX as $(srctree)/ to
# be prefixed to *both* command invocation and dependencies.
#
# Note: We also print the filenames in the quiet_cmd_foo text, and
# perhaps ought to have a version specially escaped for that purpose.
# But it's only cosmetic, and $(patsubst "%",%,$(CONFIG_FOO)) is good
# enough. It'll strip the quotes in the common case where there's no
# space and it's a simple filename, and it'll retain the quotes when
# there's a space. There are some esoteric cases in which it'll print
# the wrong thing, but we don't really care. The actual dependencies
# and commands *do* get it right, with various combinations of single
# and double quotes, backslashes and spaces in the filenames.
#
###############################################################################
#
define config_filename
ifneq ($$(CONFIG_$(1)),"")
$(1)_FILENAME := $$(subst \\,\,$$(subst \$$(quote),$$(quote),$$(subst $$(space_escape),\$$(space),$$(patsubst "%",%,$$(subst $$(space),$$(space_escape),$$(CONFIG_$(1)))))))
ifneq ($$(patsubst /%,%,$$(firstword $$($(1)_FILENAME))),$$(firstword $$($(1)_FILENAME)))
else
ifeq ($$(wildcard $$($(1)_FILENAME)),)
ifneq ($$(wildcard $$(srctree)/$$($(1)_FILENAME)),)
$(1)_SRCPREFIX := $(srctree)/
endif
endif
endif
endif
endef
#
###############################################################################
kbuild: add .DELETE_ON_ERROR special target If Make gets a fatal signal while a shell is executing, it may delete the target file that the recipe was supposed to update. This is needed to make sure that it is remade from scratch when Make is next run; if Make is interrupted after the recipe has begun to write the target file, it results in an incomplete file whose time stamp is newer than that of the prerequisites files. Make automatically deletes the incomplete file on interrupt unless the target is marked .PRECIOUS. The situation is just the same as when the shell fails for some reasons. Usually when a recipe line fails, if it has changed the target file at all, the file is corrupted, or at least it is not completely updated. Yet the file’s time stamp says that it is now up to date, so the next time Make runs, it will not try to update that file. However, Make does not cater to delete the incomplete target file in this case. We need to add .DELETE_ON_ERROR somewhere in the Makefile to request it. scripts/Kbuild.include seems a suitable place to add it because it is included from almost all sub-makes. Please note .DELETE_ON_ERROR is not effective for phony targets. The external module building should never ever touch the kernel tree. The following recipe fails if include/generated/autoconf.h is missing. However, include/config/auto.conf is not deleted since it is a phony target. PHONY += include/config/auto.conf include/config/auto.conf: $(Q)test -e include/generated/autoconf.h -a -e $@ || ( \ echo >&2; \ echo >&2 " ERROR: Kernel configuration is invalid."; \ echo >&2 " include/generated/autoconf.h or $@ are missing.";\ echo >&2 " Run 'make oldconfig && make prepare' on kernel src to fix it."; \ echo >&2 ; \ /bin/false) Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-07-20 07:46:33 +00:00
# delete partially updated (i.e. corrupted) files on error
.DELETE_ON_ERROR:
# do not delete intermediate files automatically
.SECONDARY: