cpu/sam0: flashpage write / read cycle produces different results depending on code layout in flash

[benpicco]

Description

When using mtd_write() / mtd_read(), 0xFF is read instead of the written buffer.
Adding a hexdump after the read / write operation makes the issue go away

--- a/tests/mtd_flashpage/main.c
+++ b/tests/mtd_flashpage/main.c
@@ -132,6 +133,11 @@ static void test_mtd_write_read(void)
     TEST_ASSERT_EQUAL_INT(0, ret);
 
     ret = mtd_read(dev, buf_read, TEST_ADDRESS1, sizeof(buf_read));
+
+    od_hex_dump(buf, sizeof(buf), 0);
+    puts("-----");
+    od_hex_dump(buf_read, sizeof(buf), 0);
+
     TEST_ASSERT_EQUAL_INT(0, ret);
     TEST_ASSERT_EQUAL_INT(0, memcmp(buf, buf_read, sizeof(buf)));
     TEST_ASSERT_EQUAL_INT(0, memcmp(buf_empty, buf_read + sizeof(buf), sizeof(buf_empty)));

Also compiling with PICOLIBC=1 makes the issue go away. Compiling with TOOLCHAIN=llvm did make the issue appear on CI for a while.

Steps to reproduce the issue

Run tests/periph_flashpage on same54-xpro

Expected results

The test success

Actual results

2020-09-02 12:29:44,130 # mtd_flashpage_tests.test_mtd_write_read (tests/mtd_flashpage/main.c 138) exp 0 was -190
2020-09-02 12:29:44,207 # 
2020-09-02 12:29:44,209 # run 4 failures 1

Versions

RIOT master (f68f19a)

[dylad]

I was able to reproduce this issue on SAME54. Good news is SAMLXX are not affected by this bug. Did not test on SAMD21 yet.
Also, slowing down the main clock of SAME54 doesn't change anything.
I'll investigate further later.

[benpicco]

What I tried so far:

• wrapping flashpage functions in _NVMCTRL->CTRLA.bit.CACHEDIS0 = 1 to work around errata 2.14.1 NVM Read Corruption
• waiting for NVMCTRL_INTFLAG_DONE after each command

So far to no avail.
(Both of this 'fixed' the issue for me, only to re-appear when I tried a different configuration like picolibc instead of newlib or using clang instead of gcc)