Restructure compatibility testing (#3779)

* Restructure compatibility testing

* Add PR number into Changelog

* Keep the original name for compatibility matrix

* Remove changelog entry

* Apply suggestions

---------

Co-authored-by: Chester Cheung <[email protected]>
Co-authored-by: Tyler Yahn <[email protected]>

Author: 3 people

.github/workflows/ci.yml

@@ -150,3 +150,12 @@ jobs:
         env:
           GOARCH: ${{ matrix.arch }}
         run: make test-short
+
+  test-compatibility:
+    runs-on: ubuntu-latest
+    needs: [compatibility-test]
+    steps:
+      - name: Test if compatibility-test passed
+        run: |
+          echo ${{ needs.compatibility-test.result }}
+          test ${{ needs.compatibility-test.result }} == "success"