Skip to content

Commit 8ab0643

Browse files
shaneuttshaneutt
committed
docs: add implementation ideas for gep 1709
1 parent e733481 commit 8ab0643

File tree

2 files changed

+308
-3
lines changed

2 files changed

+308
-3
lines changed

‎geps/gep-1709.md‎

Lines changed: 307 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77

88
Add selectable profiles for conformance tests which implementations can
99
subscribe to. Also add the ability to report conformance results back to the
10-
Gateway API project and receive recognition.
10+
Gateway API project and receive recognition. Conformance test reports will
11+
add conformance data to the implementations page for the reporter which can
12+
be linked to with badges from projects and repositories.
1113

1214
## Goals
1315

@@ -24,6 +26,12 @@ Gateway API project and receive recognition.
2426

2527
- We want to avoid adding any infrastructure for the reporting mechanism if
2628
feasible.
29+
- For this iteration we don't want to add configuration files for conformance
30+
tests, instead leaving that to future iterations and working on the raw
31+
machinery here (see [alternatives considered](#alternatives-considered)).
32+
- For this iteration we don't want to add container images for conformance test
33+
runs, instead leaving that to future iterations (see
34+
[alternatives considered](#alternatives-considered).
2735

2836
## Introduction
2937

@@ -37,8 +45,304 @@ which implementations can prove they satisfy and be recognized for.
3745

3846
## API
3947

40-
TODO
48+
The API for conformance profiles will be a pipeline that implementations can
49+
opt into. The workflow is effectively:
50+
51+
1. select a [profile](#profiles)
52+
2. [integrate](#integration) tests in the downstream project
53+
3. [report results and get certified](#certification)
54+
55+
The goal is to make selecting a conformance profile as simple of a process as
56+
feasible and support both the existing command line integration approach (e.g. `go test`) as well
57+
as a [Golang][go] approach using the conformance suite as a library.
58+
59+
[go]:https://go.dev
60+
61+
### Profiles
62+
63+
Initially there will be three _categories_ of profiles which can be coupled with
64+
two _support levels_. These will be identified by short [CamelCase][camelcase]
65+
names. The initial categories we'll support will be:
66+
67+
- `Layer4`
68+
- `Layer7`
69+
- `Mesh`
70+
71+
> **NOTE**: these are simply the initial categories we're going to start with,
72+
> it's entirely reasonable for there to be more in the future.
73+
74+
Support levels will be limited to `Core` and `Extended`, since
75+
`ImplementationSpecific` isn't meaningful in conformance at the time of
76+
writing.
77+
78+
The initial profiles we provide certification for will be:
79+
80+
- `Layer4Core`
81+
- `Layer4Extended`
82+
- `Layer7Core`
83+
- `Layer7Extended`
84+
85+
> **NOTE**: `MeshCore` and `MeshExtended` are omitted _for now_ as of the time
86+
> of writing we don't actually have any mesh conformance tests. However if those
87+
> start to come online during initial implementation, it should be quite easy
88+
> to add them in given the precedent with the others.
89+
90+
The technical implementation of these profiles is very simple: they are static
91+
compilations of existing [SupportedFeatures][feat] which represent the named
92+
profile. Contributors will need to join any new features added in the future to
93+
the corresponding conformance profile.
94+
95+
Unlike `Core` the `Extended` support level presents a slight problem with the
96+
underlying basis on `SupportedFeatures` as implementations might support some
97+
(but not all) of a the `Extended` feature set. For implementations that intend
98+
to support _partial_ `Extended` features they will be required to _exclude_ any
99+
of the `Extended` features they don't support with test arguments (which we
100+
will cover in the [integration section](#integration)) and these exemptions
101+
will be reflected during the [reporting process](#reporting-process).
102+
103+
> **NOTE**: There are situations where an implementation might actually be able
104+
> to fulfill conformance tests for multiple profiles (e.g. `Layer4Core` and
105+
> `Layer7Extended`, or perhaps an implementation can literally do everything,
106+
> including `Mesh`!). Rather than bother with even higher level compilations at
107+
> this point, we will simply report these separately and the implementations
108+
> can indicate that they support multiples to their end-users in whatever way
109+
> best suits them. If over time we find that the community is really strongly
110+
> in favor of compilations at a higher level we are set up to add them in a
111+
> follow up iteration.
112+
113+
[camelcase]:https://en.wikipedia.org/wiki/Camel_case
114+
[feat]:https://github.com/kubernetes-sigs/gateway-api/blob/c61097edaa3b1fad29721e787fee4b02c35e3103/conformance/utils/suite/suite.go#L33
115+
116+
### Integration
117+
118+
Integrating the test suite into your implementation can be done using one of
119+
the following methods:
120+
121+
- The [go test][go-test] command line interface which enables projects of any
122+
language to run the test suite on any platform [Golang][go] supports.
123+
- Using the conformance test suite as a [Golang library][lib] within an already
124+
existing test suite.
125+
126+
> **NOTE**: Usage as a library is already an established colloquialism in the
127+
> community, this effort simply intends to make that more official.
128+
129+
Selected conformance profiles can be identified as arguments in either
130+
configuration. For instance using the command line interface method to run the
131+
`Layer7Core` profile to verify core feature conformance:
132+
133+
```console
134+
$ go test ./conformance/... -args -gateway-class=acme -conformance-profile=Layer7Core
135+
```
136+
137+
Or the equivalent configuration using the Golang library:
138+
139+
```go
140+
cSuite, err := suite.New(suite.Options{
141+
GatewayClassName: "acme",
142+
Profile: Layer7CoreProfile,
143+
// other options
144+
})
145+
require.NoError(t, err, "misconfigured conformance test suite")
146+
cSuite.Setup(t)
147+
148+
for i := 0; i < len(tests.ConformanceTests); i++ {
149+
test := tests.ConformanceTests[i]
150+
test.Run(t, cSuite)
151+
}
152+
```
153+
154+
> **NOTE**: In the `suite.Options` above it's still possible to add `SkipTests`
155+
> but when used in conjunction with `Profile` this will result in a report that
156+
> the profile is not valid for reporting. Implementations in this state may be
157+
> able to report themselves as "in progress", see the
158+
> [certification section](#certification) for details.
159+
160+
Alternatively for an `Extended` conformance profile where not all of the
161+
features are implemented (as described in the [profiles](#profiles) section
162+
above):
163+
164+
```console
165+
$ go test ./conformance/... -args \
166+
-gateway-class=acme \
167+
-conformance-profiles=Layer7Extended,Layer4Core \
168+
-unsupported-features=HTTPResponseHeaderModification,HTTPRouteMethodMatching,HTTPRouteQueryParamMatching,
169+
```
170+
171+
Or the equivalent configuration using the Golang library:
172+
173+
```go
174+
cSuite, err := suite.New(suite.Options{
175+
GatewayClassName: "acme",
176+
Profiles: sets.New(
177+
Layer7ExtendedProfile,
178+
Layer4Core,
179+
),
180+
UnsupportedFeatures: sets.New(
181+
suite.SupportHTTPResponseHeaderModification,
182+
suite.SupportHTTPRouteMethodMatching,
183+
suite.SupportHTTPRouteQueryParamMatching,
184+
),
185+
// other options
186+
})
187+
require.NoError(t, err, "misconfigured conformance test suite")
188+
cSuite.Setup(t)
189+
190+
for i := 0; i < len(tests.ConformanceTests); i++ {
191+
test := tests.ConformanceTests[i]
192+
test.Run(t, cSuite)
193+
}
194+
```
195+
196+
> **NOTE**: `UnsupportedFeatures` must match features that are extended. This
197+
> option is invalid when used on a `Core` type profile.
198+
199+
> **NOTE**: In the future we may consider expanding the options to configure
200+
> and run the conformance test suite using a container image and/or via a
201+
> configuration file, however it was decided to hold off from doing any of
202+
> those in this iteration to avoid it getting to big (see the
203+
> [alternatives](#alternatives-considered) section for more thoughts).
204+
205+
Once an implementation has integrated with the conformance test suite, they can
206+
move on to [certification](#certification) to report the results.
207+
208+
[go-test]:https://go.dev/doc/tutorial/add-a-test
209+
[go]:https://go.dev
210+
[lib]:https://pkg.go.dev/sigs.k8s.io/[email protected]/conformance/utils/suite
211+
212+
### Certification
213+
214+
Implementations will be able to report their conformance testing results using
215+
our [reporting process](#reporting-process). Implementations will be able to
216+
visibly demonstrate their conformance results on their downstream projects and
217+
repositories using our [certification process](#certification-process).
218+
219+
#### Reporting Process
220+
221+
When conformance tests complete a log message will be emitted:
222+
223+
```console
224+
Success! Conformance profiles "Layer7Extended" and "Layer4Core" passed for version v0.7.0
225+
The following extended features were flagged as not supported for "Layer7Extended":
226+
- HTTPResponseHeaderModification
227+
- HTTPRouteQueryParamMatching
228+
```
229+
230+
For the above output upstream Gateway API can manually generate or update a report
231+
in the following format:
232+
233+
```json
234+
kind: ConformanceReport
235+
implementation: acmeorg-acme
236+
profiles:
237+
- name: Layer7Extended
238+
successes:
239+
- tag: v0.7.0
240+
unsupportedFeatures:
241+
- HTTPResponseHeaderModification
242+
- HTTPRouteMethodMatching
243+
- tag: v0.6.2
244+
unsupportedFeatures:
245+
- HTTPResponseHeaderModification
246+
- HTTPRouteMethodMatching
247+
- HTTPRouteQueryParamMatching
248+
- tag: v0.6.1
249+
unsupportedFeatures:
250+
- HTTPResponseHeaderModification
251+
- HTTPRouteMethodMatching
252+
- HTTPRouteQueryParamMatching
253+
- name: Layer4Core
254+
successes:
255+
- tag: v0.7.0
256+
- tag: v0.6.2
257+
- tag: v0.6.1
258+
maintainers:
259+
- @kubernetes-sigs/gateway-api-maintainers
260+
```
261+
262+
> **NOTE**: In the above the `implementation` field is a combination of
263+
> `<organization>-<project>`. Organizations can be an open source organization,
264+
> an individual, a company, e.t.c.. Organizations can theoretically have more
265+
> than one `<project>` name and submit separate reports for each of them.
266+
267+
> **NOTE**: The above assumes that the `acmeorg-acme` project was already
268+
> passing for versions prior to `v0.7.0` and that the new passing version was
269+
> added to the top of the list. This allows implementations to report their
270+
> historical passes which will become relevant as we get into future major
271+
> releases. You can also see that there's been progression to add the
272+
> `HTTPRouteQueryParamMatching` feature this release.
273+
274+
> **NOTE**: The `maintainers` field indicates the Github usernames or team
275+
> names of those who are responsible for maintaining this file, so they can be
276+
> easily contacted when needed (e.g. for relevant release announcements
277+
> regarding conformance, e.t.c.).
278+
279+
The implementation can compile its `ConformanceReport` and upload it to the
280+
Gateway API project by creating a pull request. The file should be uploaded
281+
to the following sub-directory:
282+
283+
```console
284+
conformance/results/<organization>-<project>.yaml
285+
```
286+
287+
> **NOTE**: the first time an implementation adds their report, they need
288+
> to also add the Golang templating to compile that report into their section
289+
> of the implementations page if that's not already been handled.
290+
291+
This will start the certification process which will result in the
292+
implementation receiving a badge they can use on their project pages and/or
293+
repositories which indicates their conformance levels and link to their
294+
conformance reports in the upstream Gateway API website.
295+
296+
> **NOTE**: No verification process (to prevent intentionally incorrect
297+
> conformance results) will be implemented at this time. We expect that this wont
298+
> be an issue in our community and even if someone were to try and "cheat" on
299+
> the reporting the reputation loss for being caught would make them look very
300+
> bad and would not be worth it.
301+
302+
#### Certification Process
303+
304+
During the process of the pull request described in the reporting process the
305+
implementer will be expected to run a command to compile updates before the
306+
pull request will be allowed to merge:
307+
308+
```console
309+
make conformance.reports
310+
```
311+
312+
Any updates will be compiled (via Golang templates) and will then be submitted
313+
as a commit in the pull request.
314+
315+
The conformance report will emit a heading that displays the latest release
316+
version which conformance has been reported for, along with a note about any
317+
newer releases which have yet to be reported.
318+
319+
CI will provide [badges](https://shields.io) to contributors upon completion.
320+
321+
> **NOTE**: Badges which cover `Extended` support (such as `Layer7Extended`)
322+
> will look the same regardless of whether the implementation only fulfills
323+
> the extended feature set partially. The badge links will direct to the full
324+
> report, which will include a list of any feature exceptions.
325+
326+
## Alternatives Considered
327+
328+
### Conformance Test Configuration File
329+
330+
Conformance testing is currently done mainly through command line with
331+
`go test` or via use of the conformance test suite as a library in Golang
332+
projects. We considered whether adding the alternative to provide a
333+
configuration file to the test suite would be nice, but we haven't heard
334+
any specific demand for that from implementors yet so we're leaving the
335+
idea here for later iterations to consider.
336+
337+
### Conformance Test Container Image
338+
339+
Providing a container image which could be fed deployment instructions for an
340+
implementation was considered while working on this GET but it seemed like a
341+
scope all unto itself so we're avoiding it here and perhaps a later iteration
342+
could take a look at that if there are asks in the community.
41343

42344
## References
43345

44-
TODO
346+
- https://github.com/kubernetes-sigs/gateway-api/issues/1709
347+
- https://github.com/kubernetes-sigs/gateway-api/issues/1329
348+

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ nav:
7575
- geps/gep-1426.md
7676
- geps/gep-1324.md
7777
- geps/gep-1282.md
78+
- geps/gep-1709.md
7879
- Experimental:
7980
- geps/gep-1323.md
8081
- geps/gep-1016.md

0 commit comments

Comments
 (0)