Build dependencies - processing order issues

[cladmi]

There currently are concurrency issues in the order dependencies are defined and used, and cpu dependencies cannot be easily parsed.

I plan to address this by moving the processing of Makefile.dep before including $(BOARD)/Makefile.include

Dependencies resolution uses the following variables

• USEMODULE
• USEPKG
• FEATURES_
• BOARD
• CPU
• CPU_FAM (which depend on files in CPU)
• CPU_ARCH (which depend on files in CPU)
• CPU_MODEL

Steps

• Fix CPU dependencies not being parsed easily as reported first in Makefile.dep: process cpu dependencies and fix stm32_common #9892
  • Add a new file in board that include the common files too and currently only defines CPU, Makefile.board. Reuse Makefile.features to define CPU
  • Migration of CPU/CPU_MODEL Tracking: move CPU/CPU_MODEL to Makefile.features #11477
    • Makefile.features: assert CPU is defined by BOARD/Makefile.features Makefile.features: assert CPU is defined by BOARD/Makefile.features #12417
  • Move CPU dependencies to $(CPU)/Makefile.dep
  • Fix the stm32_common issue described in Makefile.dep: process cpu dependencies and fix stm32_common #9892
    • Remove them being defined as dependencies cpu/stm32_common: fix source selection declared as module dependencies (broken) #10153
• At that point, Makefile.features can be split to directly do include $(RIOTCPU)/$(CPU)/Makefile.features instead of hardwriting it.
• Fix CPU_FAM and CPU_ARCH defined in Makefile.include when they should be already available for Makefile.dep. Define them in Makefile.cpu Makefile.features files.
• doc: document how to handle multiple cpu_models in a directory doc: document how to handle multiple cpu_models in a directory #12407
• Move all BOARD/CPU dependencies to $(BOARD)/Makefile.dep and $(CPU)/Makefile.dep
• Fix other USEMODULE defined in Makefile.include issues (can be done in parallel)
  • USEMODULE += $(SKETCH_MODULE) /sys/arduino/Makefile.include
• Move Makefile.dep before doing Makefile.include. Makefile.include: resolve dependencies before Makefile.include #14351
• Reorganize files inclusion to be done only when needed

Prerequisites cleanup

• mips-malta including common cpu features board/mips-malta: include the cpu features instead of common cpu #10064
• boards: always include cpu features boards: always include cpu features #10078
  • cpu/esp8266 re-defining CPU cpu/esp8266: remove duplicate 'CPU' variable #10061
  • hifive1 not processing CPU features boards/hifive1: include cpu/fe310 features #10063
• cpu/fe310: rtc depend on the rtt feature and hifive1 update cpu/fe310: rtc depend on the rtt feature and hifive1 update #10062
• static_tests: add build system checks static_tests: add build system checks #10060

And even future improvement to remove multiple duplication between board/board_common cpu/cpu_common path:

• With BOARD and CPU recursively including their Makefile.board and Makefile.cpu, define some BOARD_IMPLEMENTATION_DIRS like variables that can be re-used to remove duplication in Makefile, Makefile.deps, Makefile.include, Makefile.features.

Explanation

To summarize, the main Makefile.include does regarding modules:
(a more detailed version is available here https://gist.github.com/cladmi/1c5af63c753635b7cb173cf8311d0b5b)

include $(RIOTBOARD)/$(BOARD)/Makefile.features
    # Define FEATURES_PROVIDED
    # Include with hard written path CPU/Makefile.features

include $(RIOTBOARD)/$(BOARD)/Makefile.include
    # Defines CPU
    # Defines USEMODULE
    # Have specific behavior depending on USEMODULE/USEPKG
include $(RIOTBOARD)/$(CPU)/Makefile.include
    # Define USEMODULE
    # Have specific behavior depending on USEMODULE/USEPKG
    # Define module dependencies

include Makefile.dep
    # Resolve USEMODULE/USEPKG/FEATURES_USED using:
    # * USEMODULE
    # * USEPKG
    # * FEATURES_
    # * BOARD
    # * CPU
    # * CPU_FAM
    # * CPU_ARCH 

include $(RIOTBASE)/sys/Makefile.include
include $(RIOTBASE)/drivers/Makefile.include
-include $(USEPKG:%=$(RIOTPKG)/%/Makefile.include)
include $(RIOTMAKE)/bindist.inc.mk
    # These also add things to USEMODULE

The main Makefile.dep only includes $(BOARD)/Makefile.dep which may sometime also end up including hardwritten_cpu_name/Makefile.dep.

In the same time makefiles/info-global.inc.mk does:

include $(RIOTBOARD)/$(BOARD)/Makefile.features
include $(RIOTBASE)/Makefile.dep

And so never takes into account any of the dependencies defined in $(BOARD)/Makefile.include or any other Makeflie.include files.

[cladmi]

To not re-add another file for nothing, I thought about putting CPU in Makefile.features instead.

It would contain "everything that needs to be defined before being able to process dependencies".

[miri64]

I'm unclear on what the problem is with CPU defined in Makefile.include...

[cladmi]

Changing it is a mix of somehow required, it makes sense, do not repeat the same information 3 times, and many things in our makefiles have hard ordering requirements so leads to the proposed solution.

One hint that moving CPU earlier than Makefile.include is a good idea, is that right now the cpu name is already hardcoded in $(BOARD)/Makefile.features (or a common/Makefile.features):

RIOT/boards/samr21-xpro/Makefile.features

Line 15 in 1e6009c

-include $(RIOTCPU)/samd21/Makefile.features

Main issue

The first main issue that is causing me problems, is that board and CPUs need to declare dependencies, and some of their internal modules too. However, this is right now often done in $(BOARD)/Makefile.include and $(CPU)/Makefile.include but the dependencies are resolved later in Makefile.dep. So they use USEMODULE before it is defined.

One example is here:

RIOT/boards/common/nrf52xxxdk/Makefile.include

Lines 21 to 27 in 1e6009c

	# The following configuration is dependencies specific
	# but they are resolved later
	# Hack to know now if 'nordic_softdevice_ble' is used
	include $(RIOTBOARD)/$(BOARD)/Makefile.dep
	# special options when using SoftDevice
	ifneq (,$(filter nordic_softdevice_ble,$(USEPKG)))

Another one here:

RIOT/cpu/stm32_common/Makefile.include

Lines 13 to 15 in 1e6009c

	ifneq (,$(filter periph_flashpage periph_eeprom,$(FEATURES_REQUIRED)))
	FEATURES_REQUIRED += periph_flash_common
	endif

This one is even using FEATURES_REQUIRED to work because of course USEMODULE is not defined yet. But it fails if the main application only declares a dependency on a module that will add the FEATURES_REQUIRED as does the test in #9892 and the OTA task force branch.

Proposed solution

So for this I want to move the processing of dependencies, before parsing all modules Makefile.include files (boards, cpu, sys, drivers, pkgs).

Constraint

Moving the dependency handling before Makefile.include means to somehow do:

include $(RIOTBOARD)/$(BOARD)/Makefile.dep
include $(RIOTBOARD)/$(CPU)/Makefile.dep

...

include $(RIOTBOARD)/$(BOARD)/Makefile.include

Which would require having CPU defined.

I say somehow, because it could be $(BOARD)/Makefile.dep or one of its $(RIOTBOARD)/common/one_on_the_common/Makefile.dep that does it like we are doing for Makefile.features and directly includes the cpu_name/Makefile.dep

But I am not going to push toward having 3 times the cpu name defined between Makefile.features, Makefile.dep, Makefile.include.

---

Then on the somehow required:

The CPU variable is used in Makefile.dep

• CPU is used in https://github.com/RIOT-OS/RIOT/blob/master/cpu/stm32_common/Makefile.dep
  It is not entirely required, because this file is only included in https://github.com/RIOT-OS/RIOT/blob/master/cpu/stm32_common/periph/Makefile and could be replaced by SRC += instead. Here USEMODULE is currently private to this Makefile. So could be moved out of Makefile.dep.

However, it is still concept a dependency that the board depends/uses on this cpu so makes sense to be explicit when resolving Makefile.dep.

[cladmi]

We can also see this problem with the esp32 board that need to define CFLAGS depending on included modules. #9426

They could in theory be defined in inline if statement, but it is not a pattern used in other boards, and would be solved by re-ordering the processing.