Add unsafe-assume-privileged

cc #222

Author: taiki-e

Cargo.toml

@@ -38,6 +38,7 @@ float = []
 std = []
 require-cas = []
 unsafe-assume-single-core = []
+unsafe-assume-privileged = []
 s-mode = []
 force-amo = []
 disable-fiq = []

README.md

@@ -93,8 +93,11 @@ RUSTFLAGS="--cfg portable_atomic_unsafe_assume_single_core" cargo ...
 
   This enables atomic types with larger than the width supported by atomic instructions available on the current target. If the current target supports 128-bit atomics, this is no-op.
 
-  This uses lock-based fallback implementations by default. The following features/cfgs change this behavior:
-  - `unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg: Use fallback implementations that disabling interrupts instead of using locks.
+  This uses fallback implementation that using global locks by default. The following features/cfgs change this behavior:
+  - [`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg](#optional-features-unsafe-assume-single-core): Use fallback implementations that disabling interrupts instead of using global locks.
+    - If your target is single-core and calling interrupt disable instructions is safe, this is a safer and more efficient option.
+  - [`unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg](#optional-features-unsafe-assume-privileged): Use fallback implementations that using global locks with disabling interrupts.
+    - If your target is multi-core and calling interrupt disable instructions is safe, this is a safer option.
 
 - <a name="optional-features-float"></a>**`float` feature**<br>
   Provide `AtomicF{32,64}`.
@@ -145,6 +148,7 @@ RUSTFLAGS="--cfg portable_atomic_unsafe_assume_single_core" cargo ...
 >   ```
 >
 > - Enabling both this feature and `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) will result in a compile error.
+> - Enabling both this feature and `unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg) will result in a compile error.
 > - The MSRV when this feature is enabled depends on the MSRV of [critical-section].
 
 - <a name="optional-features-unsafe-assume-single-core"></a><a name="optional-cfg-unsafe-assume-single-core"></a>**`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg**<br>
@@ -156,28 +160,53 @@ RUSTFLAGS="--cfg portable_atomic_unsafe_assume_single_core" cargo ...
 > [!WARNING]
 > This feature/cfg is `unsafe`, and note the following safety requirements:
 > - Enabling this feature/cfg for multi-core systems is always **unsound**.
+>
 > - This uses privileged instructions to disable interrupts, so it usually doesn't work on unprivileged mode.
 >
 >   Enabling this feature/cfg in an environment where privileged instructions are not available, or if the instructions used are not sufficient to disable interrupts in the system, it is also usually considered **unsound**, although the details are system-dependent.
 >
 >   The following are known cases:
 >   - On pre-v6 Arm, this disables only IRQs by default. For many systems (e.g., GBA) this is enough. If the system need to disable both IRQs and FIQs, you need to enable the `disable-fiq` feature (or `portable_atomic_disable_fiq` cfg) together.
 >   - On RISC-V without A-extension, this generates code for machine-mode (M-mode) by default. If you enable the `s-mode` feature (or `portable_atomic_s_mode` cfg) together, this generates code for supervisor-mode (S-mode). In particular, `qemu-system-riscv*` uses [OpenSBI](https://github.com/riscv-software-src/opensbi) as the default firmware.
->
->   Consider using the [`critical-section` feature](#optional-features-critical-section) for systems that cannot use this feature/cfg.
->
->   See also the [`interrupt` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md).
+
+Consider using the [`unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg)](#optional-features-unsafe-assume-privileged) for multi-core systems with atomic CAS.
+
+Consider using the [`critical-section` feature](#optional-features-critical-section) for systems that cannot use this feature/cfg.
+
+See also the [`interrupt` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md).
 
 > [!NOTE]
 > - It is **very strongly discouraged** to enable this feature/cfg in libraries that depend on `portable-atomic`.
 >
 >   The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature/cfg. (However, it may make sense to enable this feature/cfg by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer targeting a single-core chip.)
 > - Enabling this feature/cfg for unsupported architectures will result in a compile error.
->   - Armv6-M (thumbv6m), pre-v6 Arm (e.g., thumbv4t, thumbv5te), RISC-V without A-extension, and Xtensa are currently supported. (Since all MSP430 and AVR are single-core, we always provide atomic CAS for them without this feature/cfg.)
+>   - Arm M-Profile architectures (e.g., thumbv6m), pre-v6 Arm (e.g., thumbv4t, thumbv5te), RISC-V, and Xtensa are currently supported. (Since all MSP430 and AVR are single-core, we always provide atomic CAS for them without this feature/cfg.)
+>   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target is not supported yet.
+> - Enabling this feature/cfg for targets where privileged instructions are obviously unavailable (e.g., Linux) will result in a compile error.
+>   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target supports privileged instructions but the build rejected.
+> - Enabling both this feature/cfg and `critical-section` feature will result in a compile error.
+> - When both this feature/cfg and `unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg) are enabled, this feature/cfg is preferred.
+
+- <a name="optional-features-unsafe-assume-privileged"></a><a name="optional-cfg-unsafe-assume-privileged"></a>**`unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg**<br>
+  Similar to `unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg, but only assumes about availability of privileged instructions required to disable interrupts.
+
+  - When both this feature/cfg and enabled-by-default `fallback` feature is enabled, this crate provides atomic types with larger than the width supported by native instructions by using global locks with disabling interrupts.
+
+> [!WARNING]
+> This feature/cfg is `unsafe`, and except for being sound in multi-core systems, this has the same safety requirements as [`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg](#optional-features-unsafe-assume-single-core).
+
+> [!NOTE]
+> - It is **very strongly discouraged** to enable this feature/cfg in libraries that depend on `portable-atomic`.
+>
+>   The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature/cfg. (However, it may make sense to enable this feature/cfg by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer.)
+> - Enabling this feature/cfg for unsupported targets will result in a compile error.
+>   - This requires atomic CAS (`cfg(target_has_atomic = "ptr")` or `cfg_no_atomic_cas!`).
+>   - Arm M-Profile architectures (e.g., thumbv6m), pre-v6 Arm (e.g., thumbv4t, thumbv5te), RISC-V, and Xtensa are currently supported.
 >   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target is not supported yet.
 > - Enabling this feature/cfg for targets where privileged instructions are obviously unavailable (e.g., Linux) will result in a compile error.
 >   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target supports privileged instructions but the build rejected.
 > - Enabling both this feature/cfg and `critical-section` feature will result in a compile error.
+> - When both this feature/cfg and `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) are enabled, `unsafe-assume-single-core` is preferred.
 
 - <a name="optional-cfg-no-outline-atomics"></a>**`portable_atomic_no_outline_atomics` cfg**<br>
   Disable dynamic dispatching by run-time CPU feature detection.

build.rs

@@ -20,6 +20,8 @@ fn main() {
 
     #[cfg(feature = "unsafe-assume-single-core")]
     println!("cargo:rustc-cfg=portable_atomic_unsafe_assume_single_core");
+    #[cfg(feature = "unsafe-assume-privileged")]
+    println!("cargo:rustc-cfg=portable_atomic_unsafe_assume_privileged");
     #[cfg(feature = "s-mode")]
     println!("cargo:rustc-cfg=portable_atomic_s_mode");
     #[cfg(feature = "force-amo")]
@@ -54,7 +56,7 @@ fn main() {
         // Custom cfgs set by build script. Not public API.
         // grep -F 'cargo:rustc-cfg=' build.rs | grep -Ev '^ *//' | sed -E 's/^.*cargo:rustc-cfg=//; s/(=\\)?".*$//' | LC_ALL=C sort -u | tr '\n' ',' | sed -E 's/,$/\n/'
         println!(
-            "cargo:rustc-check-cfg=cfg(portable_atomic_disable_fiq,portable_atomic_force_amo,portable_atomic_ll_sc_rmw,portable_atomic_atomic_intrinsics,portable_atomic_no_asm,portable_atomic_no_asm_maybe_uninit,portable_atomic_no_atomic_64,portable_atomic_no_atomic_cas,portable_atomic_no_atomic_load_store,portable_atomic_no_atomic_min_max,portable_atomic_no_cfg_target_has_atomic,portable_atomic_no_cmpxchg16b_intrinsic,portable_atomic_no_cmpxchg16b_target_feature,portable_atomic_no_const_mut_refs,portable_atomic_no_const_raw_ptr_deref,portable_atomic_no_const_transmute,portable_atomic_no_core_unwind_safe,portable_atomic_no_diagnostic_namespace,portable_atomic_no_strict_provenance,portable_atomic_no_stronger_failure_ordering,portable_atomic_no_track_caller,portable_atomic_no_unsafe_op_in_unsafe_fn,portable_atomic_pre_llvm_15,portable_atomic_pre_llvm_16,portable_atomic_pre_llvm_18,portable_atomic_pre_llvm_20,portable_atomic_s_mode,portable_atomic_sanitize_thread,portable_atomic_target_feature,portable_atomic_unsafe_assume_single_core,portable_atomic_unstable_asm,portable_atomic_unstable_asm_experimental_arch,portable_atomic_unstable_cfg_target_has_atomic,portable_atomic_unstable_isa_attribute)"
+            "cargo:rustc-check-cfg=cfg(portable_atomic_atomic_intrinsics,portable_atomic_disable_fiq,portable_atomic_force_amo,portable_atomic_ll_sc_rmw,portable_atomic_no_asm,portable_atomic_no_asm_maybe_uninit,portable_atomic_no_atomic_64,portable_atomic_no_atomic_cas,portable_atomic_no_atomic_load_store,portable_atomic_no_atomic_min_max,portable_atomic_no_cfg_target_has_atomic,portable_atomic_no_cmpxchg16b_intrinsic,portable_atomic_no_cmpxchg16b_target_feature,portable_atomic_no_const_mut_refs,portable_atomic_no_const_raw_ptr_deref,portable_atomic_no_const_transmute,portable_atomic_no_core_unwind_safe,portable_atomic_no_diagnostic_namespace,portable_atomic_no_strict_provenance,portable_atomic_no_stronger_failure_ordering,portable_atomic_no_track_caller,portable_atomic_no_unsafe_op_in_unsafe_fn,portable_atomic_pre_llvm_15,portable_atomic_pre_llvm_16,portable_atomic_pre_llvm_18,portable_atomic_pre_llvm_20,portable_atomic_s_mode,portable_atomic_sanitize_thread,portable_atomic_target_feature,portable_atomic_unsafe_assume_privileged,portable_atomic_unsafe_assume_single_core,portable_atomic_unstable_asm,portable_atomic_unstable_asm_experimental_arch,portable_atomic_unstable_cfg_target_has_atomic,portable_atomic_unstable_isa_attribute)"
         );
         // TODO: handle multi-line target_feature_fallback
         // grep -F 'target_feature_fallback("' build.rs | grep -Ev '^ *//' | sed -E 's/^.*target_feature_fallback\(//; s/",.*$/"/' | LC_ALL=C sort -u | tr '\n' ',' | sed -E 's/,$/\n/'

src/imp/fallback/seq_lock.rs

@@ -8,6 +8,8 @@ use core::{
 };
 
 use super::utils::Backoff;
+#[cfg(portable_atomic_unsafe_assume_privileged)]
+use crate::imp::interrupt::arch as interrupt;
 
 // See mod.rs for details.
 pub(super) type AtomicChunk = AtomicU64;
@@ -54,14 +56,24 @@ impl SeqLock {
     /// Grabs the lock for writing.
     #[inline]
     pub(super) fn write(&self) -> SeqLockWriteGuard<'_> {
+        // Get current interrupt state and disable interrupts when the user
+        // explicitly declares that privileged instructions are available.
+        #[cfg(portable_atomic_unsafe_assume_privileged)]
+        let interrupt_state = interrupt::disable();
+
         let mut backoff = Backoff::new();
         loop {
             let previous = self.state.swap(LOCKED, Ordering::Acquire);
 
             if previous != LOCKED {
                 atomic::fence(Ordering::Release);
 
-                return SeqLockWriteGuard { lock: self, state: previous };
+                return SeqLockWriteGuard {
+                    lock: self,
+                    state: previous,
+                    #[cfg(portable_atomic_unsafe_assume_privileged)]
+                    interrupt_state,
+                };
             }
 
             while self.state.load(Ordering::Relaxed) == LOCKED {
@@ -79,6 +91,10 @@ pub(super) struct SeqLockWriteGuard<'a> {
 
     /// The stamp before locking.
     state: State,
+
+    /// The interrupt state before disabling.
+    #[cfg(portable_atomic_unsafe_assume_privileged)]
+    interrupt_state: interrupt::State,
 }
 
 impl SeqLockWriteGuard<'_> {
@@ -93,6 +109,13 @@ impl SeqLockWriteGuard<'_> {
         //
         // Release ordering for synchronizing with `optimistic_read`.
         this.lock.state.store(this.state, Ordering::Release);
+
+        // Restore interrupt state.
+        // SAFETY: the state was retrieved by the previous `disable`.
+        #[cfg(portable_atomic_unsafe_assume_privileged)]
+        unsafe {
+            interrupt::restore(this.interrupt_state);
+        }
     }
 }
 
@@ -103,6 +126,13 @@ impl Drop for SeqLockWriteGuard<'_> {
         //
         // Release ordering for synchronizing with `optimistic_read`.
         self.lock.state.store(self.state.wrapping_add(2), Ordering::Release);
+
+        // Restore interrupt state.
+        // SAFETY: the state was retrieved by the previous `disable`.
+        #[cfg(portable_atomic_unsafe_assume_privileged)]
+        unsafe {
+            interrupt::restore(self.interrupt_state);
+        }
     }
 }
 

src/imp/fallback/seq_lock_wide.rs

@@ -8,6 +8,8 @@ use core::{
 };
 
 use super::utils::Backoff;
+#[cfg(portable_atomic_unsafe_assume_privileged)]
+use crate::imp::interrupt::arch as interrupt;
 
 // See mod.rs for details.
 pub(super) type AtomicChunk = AtomicU32;
@@ -85,6 +87,11 @@ impl SeqLock {
     /// Grabs the lock for writing.
     #[inline]
     pub(super) fn write(&self) -> SeqLockWriteGuard<'_> {
+        // Get current interrupt state and disable interrupts when the user
+        // explicitly declares that privileged instructions are available.
+        #[cfg(portable_atomic_unsafe_assume_privileged)]
+        let interrupt_state = interrupt::disable();
+
         let mut backoff = Backoff::new();
         loop {
             let previous = self.state_lo.swap(LOCKED, Ordering::Acquire);
@@ -94,7 +101,12 @@ impl SeqLock {
                 // the data at the critical section of `(state_hi, previous)`.
                 atomic::fence(Ordering::Release);
 
-                return SeqLockWriteGuard { lock: self, state_lo: previous };
+                return SeqLockWriteGuard {
+                    lock: self,
+                    state_lo: previous,
+                    #[cfg(portable_atomic_unsafe_assume_privileged)]
+                    interrupt_state,
+                };
             }
 
             while self.state_lo.load(Ordering::Relaxed) == LOCKED {
@@ -112,6 +124,10 @@ pub(super) struct SeqLockWriteGuard<'a> {
 
     /// The stamp before locking.
     state_lo: State,
+
+    /// The interrupt state before disabling.
+    #[cfg(portable_atomic_unsafe_assume_privileged)]
+    interrupt_state: interrupt::State,
 }
 
 impl SeqLockWriteGuard<'_> {
@@ -126,6 +142,13 @@ impl SeqLockWriteGuard<'_> {
         //
         // Release ordering for synchronizing with `optimistic_read`.
         this.lock.state_lo.store(this.state_lo, Ordering::Release);
+
+        // Restore interrupt state.
+        // SAFETY: the state was retrieved by the previous `disable`.
+        #[cfg(portable_atomic_unsafe_assume_privileged)]
+        unsafe {
+            interrupt::restore(this.interrupt_state);
+        }
     }
 }
 
@@ -146,6 +169,13 @@ impl Drop for SeqLockWriteGuard<'_> {
         //
         // Release ordering for synchronizing with `optimistic_read`.
         self.lock.state_lo.store(state_lo, Ordering::Release);
+
+        // Restore interrupt state.
+        // SAFETY: the state was retrieved by the previous `disable`.
+        #[cfg(portable_atomic_unsafe_assume_privileged)]
+        unsafe {
+            interrupt::restore(self.interrupt_state);
+        }
     }
 }