You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/executor_swagger.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -229,7 +229,7 @@ It will marshall back to string - marshalling is not symmetric.
229
229
| globalName | string|`string`||| GlobalName exports an output artifact to the global scope, making it available as</br>'{{workflow.outputs.artifacts.XXXX}} and in workflow.status.outputs.artifacts ||
| mode | int32 (formatted integer)|`int32`||| mode bits to use on this file, must be a value between 0 and 0777</br>set when loading input artifacts. ||
232
+
| mode | int32 (formatted integer)|`int32`||| mode bits to use on this file, must be a value between 0 and 0777.</br>Set when loading input artifacts. It is recommended to set the mode value</br>to ensure the artifact has the expected permissions in your container. ||
233
233
| name | string|`string`||| name of the artifact. must be unique within a template's inputs/outputs. ||
234
234
| optional | boolean|`bool`||| Make Artifacts optional, if Artifacts doesn't generate or exist ||
@@ -330,7 +330,7 @@ of a single workflow step, which the executor will use as a default location to
330
330
| globalName | string|`string`||| GlobalName exports an output artifact to the global scope, making it available as</br>'{{workflow.outputs.artifacts.XXXX}} and in workflow.status.outputs.artifacts ||
| mode | int32 (formatted integer)|`int32`||| mode bits to use on this file, must be a value between 0 and 0777</br>set when loading input artifacts. ||
333
+
| mode | int32 (formatted integer)|`int32`||| mode bits to use on this file, must be a value between 0 and 0777.</br>Set when loading input artifacts. It is recommended to set the mode value</br>to ensure the artifact has the expected permissions in your container. ||
334
334
| name | string|`string`||| name of the artifact. must be unique within a template's inputs/outputs. ||
335
335
| optional | boolean|`bool`||| Make Artifacts optional, if Artifacts doesn't generate or exist ||
Copy file name to clipboardExpand all lines: docs/fields.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2288,7 +2288,7 @@ Artifact indicates an artifact to place at a specified path
2288
2288
|`globalName`|`string`|GlobalName exports an output artifact to the global scope, making it available as '{{io.argoproj.workflow.v1alpha1.outputs.artifacts.XXXX}} and in workflow.status.outputs.artifacts|
|`mode`|`integer`|mode bits to use on this file, must be a value between 0 and 0777 set when loading input artifacts.|
2291
+
|`mode`|`integer`|mode bits to use on this file, must be a value between 0 and 0777. Set when loading input artifacts. It is recommended to set the mode value to ensure the artifact has the expected permissions in your container.|
2292
2292
|`name`|`string`|name of the artifact. must be unique within a template's inputs/outputs.|
2293
2293
|`optional`|`boolean`|Make Artifacts optional, if Artifacts doesn't generate or exist|
@@ -4798,7 +4798,7 @@ ArtifactPaths expands a step from a collection of artifacts
4798
4798
|`globalName`|`string`|GlobalName exports an output artifact to the global scope, making it available as '{{io.argoproj.workflow.v1alpha1.outputs.artifacts.XXXX}} and in workflow.status.outputs.artifacts|
|`mode`|`integer`|mode bits to use on this file, must be a value between 0 and 0777 set when loading input artifacts.|
4801
+
|`mode`|`integer`|mode bits to use on this file, must be a value between 0 and 0777. Set when loading input artifacts. It is recommended to set the mode value to ensure the artifact has the expected permissions in your container.|
4802
4802
|`name`|`string`|name of the artifact. must be unique within a template's inputs/outputs.|
4803
4803
|`optional`|`boolean`|Make Artifacts optional, if Artifacts doesn't generate or exist|
Copy file name to clipboardExpand all lines: docs/walk-through/artifacts.md
+71-18Lines changed: 71 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,11 +1,15 @@
1
1
# Artifacts
2
2
3
3
!!! Note
4
-
You will need to [configure an artifact repository](../configure-artifact-repository.md) to run this example.
4
+
You will need to [configure an artifact repository](../configure-artifact-repository.md) to run artifact examples.
5
5
6
-
When running workflows, it is very common to have steps that generate or consume artifacts. Often, the output artifacts of one step may be used as input artifacts to a subsequent step.
6
+
## Basic Example
7
7
8
-
The below workflow spec consists of two steps that run in sequence. The first step named `generate-artifact` will generate an artifact using the `hello-world-to-file` template that will be consumed by the second step named `print-message-from-file` that then consumes the generated artifact.
8
+
When running workflows, it is very common to have steps that generate or consume artifacts.
9
+
Often, the output artifacts of one step may be used as input artifacts to a subsequent step.
10
+
11
+
The below workflow spec consists of two steps that run in sequence.
12
+
The first step named `generate-artifact` will generate an artifact using the `hello-world-to-file` template that will be consumed by the second step named `print-message-from-file` that then consumes the generated artifact.
9
13
10
14
```yaml
11
15
apiVersion: argoproj.io/v1alpha1
@@ -53,13 +57,19 @@ spec:
53
57
args: ["cat /tmp/message"]
54
58
```
55
59
56
-
The `hello-world-to-file` template uses the `echo` command to generate a file named `/tmp/hello-world.txt`. It then `outputs` this file as an artifact named `hello-art`. In general, the artifact's `path` may be a directory rather than just a file. The `print-message-from-file` template takes an input artifact named `message`, unpacks it at the `path` named `/tmp/message` and then prints the contents of `/tmp/message` using the `cat` command.
57
-
The `artifact-example` template passes the `hello-art` artifact generated as an output of the `generate-artifact` step as the `message` input artifact to the `print-message-from-file` step. DAG templates use the tasks prefix to refer to another task, for example `{{tasks.generate-artifact.outputs.artifacts.hello-art}}`.
60
+
In this Workflow:
61
+
62
+
- The `hello-world-to-file` template uses the `echo` command to generate a file named `/tmp/hello-world.txt`.
63
+
- The `hello-world-to-file` template then `outputs` this file as an artifact named `hello-art`.
64
+
In general, the artifact's `path` may be a directory rather than just a file.
65
+
- The `print-message-from-file` template takes an input artifact named `message`, unpacks it at the `path` named `/tmp/message` and then prints the contents of `/tmp/message` using the `cat` command.
66
+
- The `artifact-example` template passes the `hello-art` artifact generated as an output of the `generate-artifact` step as the `message` input artifact to the `print-message-from-file` step.
67
+
DAG templates use the tasks prefix to refer to another task, for example `{{tasks.generate-artifact.outputs.artifacts.hello-art}}`.
58
68
59
69
Optionally, for large artifacts, you can set `podSpecPatch` in the workflow spec to increase the resource request for the init container and avoid any Out of memory issues.
60
70
61
71
```yaml
62
-
<... snipped ...>
72
+
# <... snipped ...>
63
73
- name: large-artifact
64
74
# below patch gets merged with the actual pod spec and increases the memory
65
75
# request of the init container.
@@ -78,13 +88,19 @@ Optionally, for large artifacts, you can set `podSpecPatch` in the workflow spec
78
88
image: alpine:latest
79
89
command: [sh, -c]
80
90
args: ["cat /tmp/large-file"]
81
-
<... snipped ...>
91
+
# <... snipped ...>
82
92
```
83
93
84
-
Artifacts are packaged as Tarballs and gzipped by default. You may customize this behavior by specifying an archive strategy, using the `archive` field. For example:
94
+
## Setting File Behavior
95
+
96
+
### Archive Strategy
97
+
98
+
Artifacts are packaged as Tarballs and gzipped by default.
99
+
You may customize this behavior by specifying an archive strategy, using the `archive` field.
@@ -108,14 +124,43 @@ Artifacts are packaged as Tarballs and gzipped by default. You may customize thi
108
124
tar:
109
125
# no compression (also accepts the standard gzip 1 to 9 values)
110
126
compressionLevel: 0
111
-
<... snipped ...>
127
+
# <... snipped ...>
112
128
```
113
129
130
+
### File Permissions (`mode`)
131
+
132
+
It is good practice to specify the `mode` of the input file, to ensure your container code can always interact with it (reading/writing/executing) as expected.
133
+
[This article](https://www.redhat.com/en/blog/linux-file-permissions-explained) explains file permissions and octal values.
134
+
135
+
For example, to allow the user to execute `/bin/kubectl`, we set `mode: 0755`.
136
+
137
+
```yaml
138
+
# <... snipped ...>
139
+
templates:
140
+
- name: executable-artifact
141
+
inputs:
142
+
artifacts:
143
+
# Download kubectl 1.8.0 and place it at /bin/kubectl
`0755`is a YAML-formatted octal value (`0o755` is also allowed in YAML spec 1.2).
154
+
When the Kubernetes API receives and validates the octal `mode` value, it will be stored in decimal.
155
+
Therefore you will see `mode: 493` in YAML on the cluster, and therefore also in the Argo UI.
156
+
The file permissions will still be correct as 493 in decimal is 0755 in octal.
157
+
114
158
## Artifact Garbage Collection
115
159
116
160
As of version 3.4 you can configure your Workflow to automatically delete Artifacts that you don't need (visit [artifact repository capability](../configure-artifact-repository.md) for the current supported store engine).
117
161
118
-
Artifacts can be deleted `OnWorkflowCompletion` or `OnWorkflowDeletion`. You can specify your Garbage Collection strategy on both the Workflow level and the Artifact level, so for example, you may have temporary artifacts that can be deleted right away but a final output that should be persisted:
162
+
Artifacts can be deleted `OnWorkflowCompletion` or `OnWorkflowDeletion`.
163
+
You can specify your Garbage Collection strategy on both the Workflow level and the Artifact level, so for example, you may have temporary artifacts that can be deleted right away but a final output that should be persisted:
119
164
120
165
```yaml
121
166
apiVersion: argoproj.io/v1alpha1
@@ -153,12 +198,16 @@ spec:
153
198
154
199
### Artifact Naming
155
200
156
-
Consider parameterizing your S3 keys by {{workflow.uid}}, etc (as shown in the example above) if there's a possibility that you could have concurrent Workflows of the same spec. This would be to avoid a scenario in which the artifact from one Workflow is being deleted while the same S3 key is being generated for a different Workflow.
201
+
Consider parameterizing your S3 keys by {{workflow.uid}}, etc (as shown in the example above) if there's a possibility that you could have concurrent Workflows of the same spec.
202
+
This would be to avoid a scenario in which the artifact from one Workflow is being deleted while the same S3 key is being generated for a different Workflow.
157
203
158
-
In the case of having a whole directory as S3 key, please pay attention to the key value. Here are two examples:
204
+
In the case of having a whole directory as S3 key, please pay attention to the key value.
205
+
Here are two examples:
159
206
160
-
- (A) When changing the default archive option to none, it is important that it ends with a "/". Otherwise, the directory will be created in S3 but the GC pod won't be able to remove it.
161
-
- (B) When keeping the default archive option to `.tgz`, in this case, it is important that it does NOT end with "/". Otherwise, Argo will fail to create the archive file.
207
+
- (A) When changing the default archive option to none, it is important that it ends with a "/".
208
+
Otherwise, the directory will be created in S3 but the GC pod won't be able to remove it.
209
+
- (B) When keeping the default archive option to `.tgz`, in this case, it is important that it does NOT end with "/".
210
+
Otherwise, Argo will fail to create the archive file.
162
211
163
212
Example (A) without packaging as `.tgz`
164
213
@@ -230,7 +279,9 @@ spec:
230
279
231
280
### Service Accounts and Annotations
232
281
233
-
Does your S3 bucket require you to run with a special Service Account or IAM Role Annotation? You can either use the same ones you use for creating artifacts or generate new ones that are specific for deletion permission. Generally users will probably just have a single Service Account or IAM Role to apply to all artifacts for the Workflow, but you can also customize on the artifact level if you need that:
282
+
Does your S3 bucket require you to run with a special Service Account or IAM Role Annotation?
283
+
You can either use the same ones you use for creating artifacts or generate new ones that are specific for deletion permission.
284
+
Generally users will probably just have a single Service Account or IAM Role to apply to all artifacts for the Workflow, but you can also customize on the artifact level if you need that:
234
285
235
286
```yaml
236
287
apiVersion: argoproj.io/v1alpha1
@@ -307,13 +358,15 @@ rules:
307
358
- patch
308
359
```
309
360
310
-
This is the `artifactgc` role if you installed using one of the quick-start manifest files. If you installed with the `install.yaml` file for the release then the same permissions are in the `argo-cluster-role`.
361
+
This is the `artifactgc` role if you installed using one of the quick-start manifest files.
362
+
If you installed with the `install.yaml` file for the release then the same permissions are in the `argo-cluster-role`.
311
363
312
364
If you don't use your own `ServiceAccount` and are just using `default` ServiceAccount, then the role needs a RoleBinding or ClusterRoleBinding to `default` ServiceAccount.
313
365
314
366
### What happens if Garbage Collection fails?
315
367
316
-
If deletion of the artifact fails for some reason (other than the Artifact already having been deleted which is not considered a failure), the Workflow's Status will be marked with a new Condition to indicate "Artifact GC Failure", a Kubernetes Event will be issued, and the Argo Server UI will also indicate the failure. For additional debugging, the user should find 1 or more Pods named `<wfName>-artgc-*` and can view the logs.
368
+
If deletion of the artifact fails for some reason (other than the Artifact already having been deleted which is not considered a failure), the Workflow's Status will be marked with a new Condition to indicate "Artifact GC Failure", a Kubernetes Event will be issued, and the Argo Server UI will also indicate the failure.
369
+
For additional debugging, the user should find 1 or more Pods named `<wfName>-artgc-*` and can view the logs.
317
370
318
371
If the user needs to delete the Workflow and its child CRD objects, they will need to patch the Workflow to remove the finalizer preventing the deletion:
0 commit comments