OSDOCS-640: Adding docs for configuring proxy during installation #16635
Conversation
openshift-ci-robot
added
do-not-merge/work-in-progress
|
The preview will be available shortly at: |
bergerhoffer
force-pushed
the
OSDOCS-640
branch
from
55d0ca2 to
cf7464b
Compare
|
Thanks for the feedback @danehans! I've adjusted the field descriptions if you can take another look. Thanks! |
There was a problem hiding this comment.
I see @danehans asking for this restriction above, but I'm curious about why. My understanding is that you can use a HTTPS URI for httpProxy (e.g. to avoid leaking your proxy-connection creds), although you might bump into containers/image#699 if the proxy doesn't support any of the encryption options currently compiled into containers/image (and that would go for httpsProxy too).
There was a problem hiding this comment.
I once had a try with using https URI for httpProxy, network-operator will complain about it.
There was a problem hiding this comment.
It's a quite long time ago, so I just spin up a new cluster with such setting.
In network-operator pod log, I could see such message, also a clear prompt.
2019/09/27 02:42:21 Updated ClusterOperator with conditions:
- lastTransitionTime: "2019-09-27T02:42:21Z"
message: The configuration is invalid for proxy 'cluster' (httpProxy requires a
'http' URI scheme). Use 'oc edit proxy.config.openshift.io cluster' to fix.
reason: InvalidProxyConfig
There was a problem hiding this comment.
You also only need this (for proxy connections) if you're using an https proxy URI. No proxy cert to include if all your proxy connections are http.
bergerhoffer
added
peer-review-needed
bergerhoffer
force-pushed
the
OSDOCS-640
branch
2 times, most recently
from
32053b6 to
289a1ac
Compare
openshift-ci-robot
added
needs-rebase
bergerhoffer
force-pushed
the
OSDOCS-640
branch
from
289a1ac to
88d52e0
Compare
openshift-ci-robot
removed
the
needs-rebase
|
Preview bot isn't working, so posted a build to the file share site. The "Configuring the cluster-wide proxy during installation" file has been included in all appropriate provider/install types. Links to each install section are below:
* These two sections restructured all content under the "Creating the installation files for AWS" heading (broke up a single procedure into two procedures and inserted configuring proxy procedure in between). Needs to be reviewed carefully. |
bergerhoffer
changed the title
openshift-ci-robot
removed
the
do-not-merge/work-in-progress
bergerhoffer
added
the
peer-review-needed
bergerhoffer
force-pushed
the
OSDOCS-640
branch
from
88d52e0 to
7887219
Compare
There was a problem hiding this comment.
If restricted network is not going to be supported in 4.2, you should consider removing this.
There was a problem hiding this comment.
@kalexand-rh can confirm for me, but the docs for these restricted network are in for 4.2, so they should be supported. Also, this sentence will only appear in the restricted docs, so if they don't go out for some reason, then this won't appear.
There was a problem hiding this comment.
I'm a bit confused by this paragraph. First it talks about removing manifests, then it states to generate the manifests.
There was a problem hiding this comment.
First it talks about removing manifests, then it states to generate the manifests.
You: "Installer, make your usual manifests." Installer: "Here you go" You: "Forget about these. The rest look good. Carry on."
However we want to explain that workflow works for me.
There was a problem hiding this comment.
This was pulled from the original module that I broke up, but I agree that it is confusing.
@kalexand-rh - I don't have the background to know why the steps are the way that they are. Do you have any reword suggestions that would make this clearer, or know who might be good to ask on this?
There was a problem hiding this comment.
Sorry I missed your comment @wking before I added mine - thanks for the explanation!
@kalexand-rh let's look at this together tomorrow.
modules/installation-generate-aws-user-infra-install-config.adoc
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
@wking correct me if I'm wrong, but this section should be removed due to https://bugzilla.redhat.com/show_bug.cgi?id=1755073.
There was a problem hiding this comment.
I think so, but prob wait until openshift/installer#2402 lands or is otherwise sorted.
modules/installation-generate-aws-user-infra-install-config.adoc
Outdated
Show resolved
Hide resolved
|
@bergerhoffer this is coming along nicely. I provided a few comments. This PR covers proxy at install time. Will a separate PR address configuring |
|
@bergerhoffer UPI-based proxy installs require a newer ami from 4.1. Are the AMI references being updated as part of the general 4.2 docs? cc: @wking @ewolinetz |
If we haven't yet, 4.2 should definitely bump to https://github.com/openshift/installer/blob/release-4.2/data/data/rhcos.json |
bergerhoffer
force-pushed
the
OSDOCS-640
branch
from
8a23db6 to
aa39bf3
Compare
|
@vikram-redhat Do you know if it's on anyone's plate to update the list of AMIs for 4.2 [1] to [2] as suggested? [1] https://docs.openshift.com/container-platform/4.2/installing/installing_aws_user_infra/installing-aws-user-infra.html#installation-aws-user-infra-rhcos-ami_installing-aws-user-infra |
|
@danehans I've made a few updates. I'm going to go ahead and merge this PR because we have a round of reviews over the next few days that I'd like this to be included for. But we can continue to refine these docs as necessary in a separate PR. When you ask about day 2 configuration of the proxy, are you talking about enabling the proxy on an existing cluster, which was added by PR #16562 (and can be viewed here). Or are there some other configuration docs that we're missing? |
| spec: | ||
| httpProxy: http://<username>:<pswd>@<ip>:<port> <1> | ||
| httpsProxy: https://<username>:<pswd>@<ip>:<port> <2> | ||
| httpsProxy: http://<username>:<pswd>@<ip>:<port> <2> |
|
/cherrypick enterprise-4.2 |
|
@bergerhoffer: new pull request created: #16871 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@kalexand-rh would you want to add this to the last minute tasks? |
|
|
||
| include::modules/installation-aws-config-yaml.adoc[leveloffset=+2] | ||
|
|
||
| include::modules/installation-configure-proxy.adoc[leveloffset=+2] |
There was a problem hiding this comment.
I have a doubt whether we should add it here. As I know, only UPI way support proxy setting until IPI supports using a preconfigured VPC. This section is for IPI-on-AWS.
There was a problem hiding this comment.
You can use a proxy on AWS IPI, the proxy just has to live outside the VPC and still be reachable by the cluster. This is how CI works now. Not as nice as what's in flight with openshift/release#4719, but still something.
There was a problem hiding this comment.
@gpei Are you suggesting that this should not be included here for AWS IPI (and Azure and GCP)? I had confirmed with @katherinedube the appropriate supported list here: https://jira.coreos.com/browse/OSDOCS-640?focusedCommentId=130922&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-130922
There was a problem hiding this comment.
As Trevor mentioned, until we support deploying to a pre-existing VPC (planned for the next release), the proxy must live outside the VPC and be reachable by the nodes. While I agree it's not ideal, I'm not sure we should be saying it doesn't work unless it really doesn't work.
There was a problem hiding this comment.
yeah, I understand that we can use proxy for IPI install. But QE was told UPI must be used for proxy testing in 4.2, so all proxy related testing were scheduled with UPI installs in QE's 4.2 test matrix.
There was a problem hiding this comment.
I traced it back to a discussion Daneyon had with Xiaoli to remove the testing of ipi installs (at the time that it might have been the belief), but after ipi was tested, it wasn't the case.
@katherinedube IPI was being used for testing as a stop-gap until UPI CI was complete. It was identified early on by @derekwaynecarr that UPI is required due to the lack of existing VPC support for IPI. As I mention in my above comment, IPI can technically work, but I don't see the requirements for IPI proxy meeting real world use cases.
There was a problem hiding this comment.
@danehans Just to be clear, even if we're going to say upi support is required for proxy this should only effect public cloud provider ipi deployments. Meaning I don't know this is really a generic ipi problem across all of our ipi providers like OSP. As it is today in the text matrix, it's all ipi providers - so nothing is being tested with OSP (not sure this should have been excluded.) Otherwise, why couldn't a public cloud provider use proxy with ipi? Could you provider some additional details as to why it wouldn't be practical? Thanks!
There was a problem hiding this comment.
I totally agree with @danehans 's statement.
IPI's prerequisite is having internet connectivity. That means, cluster itself already have internet connection, why still need proxy? Even proxy is enabled, how to validate all components and operators are really getting outside via proxy but not via its own internet connectivity?
In QE's testing, we dropped internet connectivity, and enable proxy, we found the following bugs:
Bug 1753467 - [proxy] no proxy is set for kube-controller-manager
That indicates if we do not drop internet connectivity for proxy testing, our test result is NOT persuadable, or else, we would miss the above bugs. But in the whole IPI install process, we have no way to drop internet connectivity.
There was a problem hiding this comment.
In QE's testing, we dropped internet connectivity, and enable proxy, we found...
We definitely want to support proxy on blackholed UPI; I don't think anyone opposes that. Once we have bring-your-own-VPC, we all agree we want to support proxy there too. And testing in those environments makes me reasonably confident that at least OpenShift components will respect a voluntary proxy. The only contention seems to be whether we want to support proxy for installer-provided-everything. And the pushback seems to be mostly around "who wants this?". But I don't see anything about that case that would be a support burden, so I'd rather not add confusion by walling it off as an unsupported niche. Maybe customers want a proxy for MitM snooping on our egress (and they are also convinced from our blackhole support that we are playing nice). Or maybe they want a proxy for some other reason. But I much prefer the optics of "you may not care about this on IPI" to "we don't support this (suggesting it does not work reliably) on IPI". From a QE angle, we don't test all possible config permutations on all platforms. I'm fine leaving IPI+proxy covered just in CI, and relying on QE in the blackholed cases to turn up bugs like they're already doing.
There was a problem hiding this comment.
From a QE angle, we don't test all possible config permutations on all platforms.
Yeah, agree. QE did not run proxy testing in a blackholed IPI (due to in 4.2 IPI, we have no way to create a blackholed env), but only proxy testing in a blackholed UPI.
I like "we don't support this (suggesting it does not work reliably) on IPI".
|
|
||
| include::modules/installation-azure-config-yaml.adoc[leveloffset=+2] | ||
|
|
||
| include::modules/installation-configure-proxy.adoc[leveloffset=+2] |
There was a problem hiding this comment.
Same here, we only have IPI-on-Azure in 4.2
| include::modules/installation-gcp-config-yaml.adoc[leveloffset=+2] | ||
|
|
||
| include::modules/installation-configure-proxy.adoc[leveloffset=+2] | ||
|
|
There was a problem hiding this comment.
Same here, this section is for IPI-on-GCP
There was a problem hiding this comment.
And for proxy support on GCP, we still have a testblocker bug https://bugzilla.redhat.com/show_bug.cgi?id=1753930, which was targeted to 4.3. We might need to add it to known issue in the release note.
There was a problem hiding this comment.
openshift/installer#2405 is in the merge queue, which would unblock a 4.2 backport. But yeah, restricting proxy GCP to UPI in the meantime would be a good, conservative choice.
There was a problem hiding this comment.
The 4.3/master changes all landed. Hopefully we'll have the 4.2 backports soon.
There was a problem hiding this comment.
PRs for https://bugzilla.redhat.com/show_bug.cgi?id=1753930 have merged.
There was a problem hiding this comment.
https://bugzilla.redhat.com/show_bug.cgi?id=1753930 was already verified with 4.3 build.
But if the 4.2 backports were not merged into 4.2.0, customer still have the problem when 4.2.0 is shipped.
There was a problem hiding this comment.
FYI - with PR #17080 we're limiting support to UPI installs only, so support is only for UPI for GCP now.
@danehans Were the PRs for https://bugzilla.redhat.com/show_bug.cgi?id=1753930 backported to 4.2, or is this still a problem in 4.2 for GCP UPI?
If not, you mentioned there was a workaround of adding 'metadata.google.internal.' to noProxy - let me know if this is something that we need to document or not.
There was a problem hiding this comment.
@bergerhoffer The cloned bug for 4.2 https://bugzilla.redhat.com/show_bug.cgi?id=1759245 has been verified now, it's not an issue for 4.2.0 GCP UPI anymore, thanks!
There was a problem hiding this comment.
I once had a try with using https URI for httpProxy, network-operator will complain about it.
Drop the wording from aa39bf3 (OSDOCS-640: Adding docs for configuring proxy during installation, 2019-09-12, openshift#16635), because the network operator explicitly supports both the 'http' and 'https' schemes since openshift/cluster-network-operator@42dbcf8955 (Refactors PR to focus on http/https/no proxy reconciliation, 2019-08-06, openshift/cluster-network-operator#245), which landed before release-4.2 split off from the network operator's master.
|
My client is using a transparent proxy.. no url to put into http(s)Proxy.. however any outbound traffic needs a CA in the chain... additionalTrustedCA only sort of works for installation and post-install tasks. |
|
@danehans Can you address @therevoman's situation above? |
12 participants
Preview: https://osdocs-640--ocpdocs.netlify.com/openshift-enterprise/latest/installing/installing_aws/installing-aws-customizations.html#installation-configure-proxy_install-customizations-cloud