Skip to content

Commit f8dc601

Browse files
runesoerensenMalaxjoshwlewisschneems
authored
Migrate inventory code (#861)
* Add src files from inventory repo Copy all files currently living in https://github.com/Malax/inventory/tree/main/inventory/src without any changes. Co-Authored-By: Manuel Fuchs <manuel.fuchs@salesforce.com> Co-Authored-By: Josh W Lewis <josh.lewis@salesforce.com> * Merge lib.rs and inventory.rs * Declare inventory module and feature * Prefer PhantomData * Add must_use attribute * Remove unused import * Allow unwrap for now * Add sha2 feature We may want to consider another approach here (e.g. naming the feature `inventory-sha2` and/or pulling in the inventory dependency ["inventory", "dep:sha2"]). While the crate will be pulled in if a user enables the `sha2`, the inventory-specific sha2 code won't be compiled unless the `inventory` feature is also enabled. * Add semver feature * Include patch version in dependency requirements * Add inventory toml dependency * Add inventory thiserror dependency * Add changelog entry * Allow unreachable pub for re-exports Also see Malax/inventory#2 (comment) * Remove semver and sha2 re-exports These re-exports appear to be unnecessary, and doesn't have any impact on code that rely on the `inventory` module (and have the `semver` and/or `sha2` features enabled). Also see related prior discussion here Malax/inventory#2 (comment). The `semver` and `sha2` modules only contain implementations of public traits, so I don't think we need to re-export those (unlike bringing for instance a function or struct in to scope). If I understand how trait implementations work correctly, it doesn't matter where an implementation lives (only the visibility of the trait and the type it's implemented for is relevant) - in other words, the implementation will be available to any code that can access both the trait and the type, even across crate binaries. * Rename `semver` feature to `inventory-semver` * Rename `sha2` feature to `inventory-sha2` * [stacked] Add docs to inventory code (#864) * Add module docs and example * Document resolve and partial_resolve * Document more inventory methods * Document Artifact * Document ArtifactRequirement I also suggest we change the name of `inventory/version` to `inventory/artifact_requirement.rs` or `requirement.rs`. * Update example to compare Checksum instead of string * Apply suggestions from code review Co-authored-by: Rune Soerensen <rsoerensen@salesforce.com> * Update feature names * Rewrite example usage * Show how to display checksum in example --------- Co-authored-by: Rune Soerensen <rsoerensen@salesforce.com> * Group inventory features alphabetically --------- Co-authored-by: Manuel Fuchs <manuel.fuchs@salesforce.com> Co-authored-by: Josh W Lewis <josh.lewis@salesforce.com> Co-authored-by: Richard Schneeman <richard.schneeman+no-recruiters@gmail.com>
1 parent ba5e823 commit f8dc601

10 files changed

Lines changed: 695 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
## [Unreleased]
1111

12+
### Added
13+
14+
- `libherokubuildpack`:
15+
- Added `inventory` module. ([#861](https://github.com/heroku/libcnb.rs/pull/861))
1216

1317
## [0.23.0] - 2024-08-28
1418

‎libherokubuildpack/Cargo.toml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,13 @@ all-features = true
1818
workspace = true
1919

2020
[features]
21-
default = ["command", "download", "digest", "error", "log", "tar", "toml", "fs", "write", "buildpack_output"]
21+
default = ["command", "download", "digest", "error", "inventory", "log", "inventory-semver", "inventory-sha2", "tar", "toml", "fs", "write", "buildpack_output"]
2222
download = ["dep:ureq", "dep:thiserror"]
2323
digest = ["dep:sha2"]
2424
error = ["log", "dep:libcnb"]
25+
inventory = ["dep:hex", "dep:serde", "dep:thiserror", "dep:toml"]
26+
inventory-semver = ["dep:semver"]
27+
inventory-sha2 = ["dep:sha2"]
2528
log = ["dep:termcolor"]
2629
tar = ["dep:tar", "dep:flate2"]
2730
toml = ["dep:toml"]
@@ -38,8 +41,11 @@ crossbeam-utils = { version = "0.8.20", optional = true }
3841
# https://github.com/rust-lang/libz-sys/issues/93
3942
# As such we have to use the next best alternate backend, which is `zlib`.
4043
flate2 = { version = "1.0.33", default-features = false, features = ["zlib"], optional = true }
44+
hex = { version = "0.4.3", optional = true }
4145
libcnb = { workspace = true, optional = true }
4246
pathdiff = { version = "0.2.1", optional = true }
47+
semver = { version = "1.0.21", features = ["serde"], optional = true }
48+
serde = { version = "1.0.209", features = ["derive"], optional = true }
4349
sha2 = { version = "0.10.8", optional = true }
4450
tar = { version = "0.4.41", default-features = false, optional = true }
4551
termcolor = { version = "1.4.1", optional = true }
@@ -50,4 +56,5 @@ ureq = { version = "2.10.1", default-features = false, features = ["tls"], optio
5056
[dev-dependencies]
5157
indoc = "2.0.5"
5258
libcnb-test = { workspace = true }
59+
serde_test = "1.0.177"
5360
tempfile = "3.12.0"

‎libherokubuildpack/src/inventory.rs‎

Lines changed: 284 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,284 @@
1+
//! # Inventory
2+
//!
3+
//! Many buildpacks need to provide artifacts from different URLs. A helpful pattern
4+
//! is to provide a list of artifacts in a TOML file, which can be parsed and used by
5+
//! the buildpack to download the correct artifact. For example, a Ruby buildpack
6+
//! might need to download pre-compiled Ruby binaries hosted on S3.
7+
//!
8+
//! This module can be used to produce and consume such an inventory file.
9+
//!
10+
//! ## Features
11+
//!
12+
//! - Version lookup and comparison: To implement the inventory, you'll need to define how
13+
//! versions are compared. This allows the inventory code to find an appropriate artifact
14+
//! based on whatever custom version logic you need. If you don't need custom logic, you can
15+
//! use the included `inventory-semver` feature.
16+
//! - Architecture aware: Beyond version specifiers, buildpack authors may need to provide different
17+
//! artifacts for different computer architectures such as ARM64 or AMD64. The inventory encodes
18+
//! this information which is used to select the correct artifact.
19+
//! - Checksum validation: In addition to knowing the URL of an artifact, buildp authors
20+
//! want to be confident that the artifact they download is the correct one. To accomplish this
21+
//! the inventory contains a checksum of the download and can be used to validate the download
22+
//! has not been modified or tampered with. To use sha256 or sha512 checksums out of the box,
23+
//! enable the `inventory-sha2` feature
24+
//! - Extensible with metadata: The default inventory format covers a lot of common use cases,
25+
//! but if you need more, you can extend it by adding custom metadata to each artifact.
26+
//!
27+
//! ## Example usage
28+
//!
29+
//! This example demonstrates:
30+
//! * Creating an artifact using the `inventory-sha2` and `inventory-semver` features.
31+
//! * Adding the artifact to an inventory.
32+
//! * Serializing and deserializing the inventory [to](Inventory#method.fmt) and [from](Inventory::from_str) TOML.
33+
//! * [Resolving an inventory artifact](Inventory::resolve) specifying relevant OS, architecture, and version requirements.
34+
//! * Using the resolved artifact's checksum value to verify "downloaded" data.
35+
//!
36+
//! ```rust
37+
//! use libherokubuildpack::inventory::{artifact::{Arch, Artifact, Os}, Inventory, checksum::Checksum};
38+
//! use semver::{Version, VersionReq};
39+
//! use sha2::{Sha256, Digest};
40+
//!
41+
//! // Create an artifact with a SHA256 checksum and `semver::Version`
42+
//! let new_artifact = Artifact {
43+
//! version: Version::new(1, 0, 0),
44+
//! os: Os::Linux,
45+
//! arch: Arch::Arm64,
46+
//! url: "https://example.com/foo.txt".to_string(),
47+
//! checksum: "sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"
48+
//! .parse::<Checksum<Sha256>>()
49+
//! .unwrap(),
50+
//! metadata: None,
51+
//! };
52+
//!
53+
//! // Create an inventory and add the artifact
54+
//! let mut inventory = Inventory::<Version, Sha256, Option<()>>::new();
55+
//! inventory.push(new_artifact.clone());
56+
//!
57+
//! // Serialize the inventory to TOML
58+
//! let inventory_toml = inventory.to_string();
59+
//! assert_eq!(
60+
//! r#"[[artifacts]]
61+
//! version = "1.0.0"
62+
//! os = "linux"
63+
//! arch = "arm64"
64+
//! url = "https://example.com/foo.txt"
65+
//! checksum = "sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"
66+
//! "#,
67+
//! inventory_toml
68+
//! );
69+
//!
70+
//! // Deserialize the inventory from TOML
71+
//! let parsed_inventory = inventory_toml
72+
//! .parse::<Inventory<Version, Sha256, Option<()>>>()
73+
//! .unwrap();
74+
//!
75+
//! // Resolve the artifact by OS, architecture, and version requirement
76+
//! let version_req = VersionReq::parse("=1.0.0").unwrap();
77+
//! let resolved_artifact = parsed_inventory.resolve(Os::Linux, Arch::Arm64, &version_req).unwrap();
78+
//!
79+
//! assert_eq!(&new_artifact, resolved_artifact);
80+
//!
81+
//! // Verify checksum of the resolved artifact
82+
//! let downloaded_data = "foo"; // Example downloaded file content
83+
//! let downloaded_checksum = Sha256::digest(downloaded_data).to_vec();
84+
//!
85+
//! assert_eq!(downloaded_checksum, resolved_artifact.checksum.value);
86+
//! println!(
87+
//! "Successfully downloaded {} with checksum {}",
88+
//! resolved_artifact.url,
89+
//! hex::encode(&resolved_artifact.checksum.value)
90+
//! );
91+
//! ```
92+
pub mod artifact;
93+
pub mod checksum;
94+
pub mod version;
95+
96+
#[cfg(feature = "inventory-semver")]
97+
mod semver;
98+
#[cfg(feature = "inventory-sha2")]
99+
mod sha2;
100+
mod unit;
101+
102+
use crate::inventory::artifact::{Arch, Artifact, Os};
103+
use crate::inventory::checksum::Digest;
104+
use crate::inventory::version::ArtifactRequirement;
105+
use serde::de::DeserializeOwned;
106+
use serde::{Deserialize, Serialize};
107+
use std::cmp::Ordering;
108+
use std::fmt::Formatter;
109+
use std::str::FromStr;
110+
111+
/// Represents an inventory of artifacts.
112+
///
113+
/// An inventory can be read directly from a TOML file on disk and used by a buildpack to resolve
114+
/// requirements for a specific artifact to download.
115+
///
116+
/// The inventory can be manipulated in-memory and then re-serialized to disk to facilitate both
117+
/// reading and writing inventory files.
118+
#[derive(Debug, Serialize, Deserialize)]
119+
pub struct Inventory<V, D, M> {
120+
#[serde(bound = "V: Serialize + DeserializeOwned, D: Digest, M: Serialize + DeserializeOwned")]
121+
pub artifacts: Vec<Artifact<V, D, M>>,
122+
}
123+
124+
impl<V, D, M> Default for Inventory<V, D, M> {
125+
fn default() -> Self {
126+
Self { artifacts: vec![] }
127+
}
128+
}
129+
130+
impl<V, D, M> Inventory<V, D, M> {
131+
/// Creates a new empty inventory
132+
#[must_use]
133+
pub fn new() -> Self {
134+
Self::default()
135+
}
136+
137+
/// Add a new artifact to the in-memory inventory
138+
pub fn push(&mut self, artifact: Artifact<V, D, M>) {
139+
self.artifacts.push(artifact);
140+
}
141+
142+
/// Return a single artifact as the best match given the input constraints
143+
///
144+
/// If multiple artifacts match the constraints, the one with the highest version is returned.
145+
pub fn resolve<R>(&self, os: Os, arch: Arch, requirement: &R) -> Option<&Artifact<V, D, M>>
146+
where
147+
V: Ord,
148+
R: ArtifactRequirement<V, M>,
149+
{
150+
self.artifacts
151+
.iter()
152+
.filter(|artifact| {
153+
artifact.os == os
154+
&& artifact.arch == arch
155+
&& requirement.satisfies_version(&artifact.version)
156+
&& requirement.satisfies_metadata(&artifact.metadata)
157+
})
158+
.max_by_key(|artifact| &artifact.version)
159+
}
160+
161+
/// Resolve logic for Artifacts that implement `PartialOrd` rather than `Ord`
162+
///
163+
/// Some version implementations are only partially ordered. One example could be f32 which is not totally ordered
164+
/// because NaN is not comparable to any other number.
165+
pub fn partial_resolve<R>(
166+
&self,
167+
os: Os,
168+
arch: Arch,
169+
requirement: &R,
170+
) -> Option<&Artifact<V, D, M>>
171+
where
172+
V: PartialOrd,
173+
R: ArtifactRequirement<V, M>,
174+
{
175+
#[inline]
176+
fn partial_max_by_key<I, F, A>(iterator: I, f: F) -> Option<I::Item>
177+
where
178+
I: Iterator,
179+
F: Fn(&I::Item) -> A,
180+
A: PartialOrd,
181+
{
182+
iterator.fold(None, |acc, item| match acc {
183+
None => Some(item),
184+
Some(acc) => match f(&item).partial_cmp(&f(&acc)) {
185+
Some(Ordering::Greater | Ordering::Equal) => Some(item),
186+
None | Some(Ordering::Less) => Some(acc),
187+
},
188+
})
189+
}
190+
191+
partial_max_by_key(
192+
self.artifacts.iter().filter(|artifact| {
193+
artifact.os == os
194+
&& artifact.arch == arch
195+
&& requirement.satisfies_version(&artifact.version)
196+
&& requirement.satisfies_metadata(&artifact.metadata)
197+
}),
198+
|artifact| &artifact.version,
199+
)
200+
}
201+
}
202+
203+
#[derive(thiserror::Error, Debug)]
204+
pub enum ParseInventoryError {
205+
#[error("TOML parsing error: {0}")]
206+
TomlError(toml::de::Error),
207+
}
208+
209+
impl<V, D, M> FromStr for Inventory<V, D, M>
210+
where
211+
V: Serialize + DeserializeOwned,
212+
D: Digest,
213+
M: Serialize + DeserializeOwned,
214+
{
215+
type Err = ParseInventoryError;
216+
217+
fn from_str(s: &str) -> Result<Self, Self::Err> {
218+
toml::from_str(s).map_err(ParseInventoryError::TomlError)
219+
}
220+
}
221+
222+
impl<V, D, M> std::fmt::Display for Inventory<V, D, M>
223+
where
224+
V: Serialize + DeserializeOwned,
225+
D: Digest,
226+
M: Serialize + DeserializeOwned,
227+
{
228+
#![allow(clippy::unwrap_used)]
229+
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
230+
f.write_str(&toml::to_string(self).unwrap())
231+
}
232+
}
233+
234+
#[cfg(test)]
235+
mod test {
236+
use crate::inventory::artifact::{Arch, Artifact, Os};
237+
use crate::inventory::checksum::tests::BogusDigest;
238+
use crate::inventory::Inventory;
239+
240+
#[test]
241+
fn test_matching_artifact_resolution() {
242+
let mut inventory = Inventory::new();
243+
inventory.push(create_artifact("foo", Os::Linux, Arch::Arm64));
244+
245+
assert_eq!(
246+
"foo",
247+
&inventory
248+
.resolve(Os::Linux, Arch::Arm64, &String::from("foo"))
249+
.expect("should resolve matching artifact")
250+
.version,
251+
);
252+
}
253+
254+
#[test]
255+
fn test_dont_resolve_artifact_with_wrong_arch() {
256+
let mut inventory = Inventory::new();
257+
inventory.push(create_artifact("foo", Os::Linux, Arch::Arm64));
258+
259+
assert!(inventory
260+
.resolve(Os::Linux, Arch::Amd64, &String::from("foo"))
261+
.is_none());
262+
}
263+
264+
#[test]
265+
fn test_dont_resolve_artifact_with_wrong_version() {
266+
let mut inventory = Inventory::new();
267+
inventory.push(create_artifact("foo", Os::Linux, Arch::Arm64));
268+
269+
assert!(inventory
270+
.resolve(Os::Linux, Arch::Arm64, &String::from("bar"))
271+
.is_none());
272+
}
273+
274+
fn create_artifact(version: &str, os: Os, arch: Arch) -> Artifact<String, BogusDigest, ()> {
275+
Artifact {
276+
version: String::from(version),
277+
os,
278+
arch,
279+
url: "https://example.com".to_string(),
280+
checksum: BogusDigest::checksum("cafebabe"),
281+
metadata: (),
282+
}
283+
}
284+
}

0 commit comments

Comments
 (0)