Docs: Update several docs regarding new native engine transactions, and new recommended storage configurations.

Author: jhuckaby

docs/config.md

@@ -811,7 +811,7 @@ Automatically recover incomplete transactions on startup (default: `true`).
 ### Storage.trans_dir
 <!-- Title: Transaction Directory Path -->
 
-Temp directory for transaction logs/journals (default: `data/_transactions`).
+Local directory for transaction rollback logs and recovery bookkeeping (default: `data/_transactions`).  Engines with native transaction support, such as SQLite, Postgres, and Redis, do not write rollback logs for successful commits, but the directory is still used for transaction housekeeping.
 
 ### Storage.log_event_types
 <!-- Title: Log Event Types -->

docs/hosting.md

@@ -135,7 +135,7 @@ See our [Command Line Guide](cli.md) for controlling the xyOps service via comma
 
 When you manually install xyOps, it creates a cluster of one, and promotes itself to primary.  To add backup conductors, follow these instructions.
 
-First, for multi-conductor setups, **you must have an external storage backend**, such as S3, or S3-compatible (MinIO, etc.).  See [Storage Engines](https://github.com/jhuckaby/pixl-server-storage#engines) for details.
+First, for multi-conductor setups, **you must have shared external storage**.  For live production, we recommend a Hybrid storage setup using Redis or Postgres for JSON data, plus S3 or an S3-compatible service for files.  See [Storage Setup](storage.md) for details.
 
 Once you have external storage setup and working, stop the xyOps service, and edit the `/opt/xyops/conf/masters.json` file:
 
@@ -265,7 +265,7 @@ For a load balanced multi-conductor setup with Nginx w/TLS, please read this sec
 
 A few prerequisites for this setup:
 
-- For multi-conductor setups, **you must have an external storage backend**, such as S3, or S3-compatible (MinIO, etc.).  See [Storage Engines](https://github.com/jhuckaby/pixl-server-storage#engines) for details.
+- For multi-conductor setups, **you must have shared external storage**.  For live production, we recommend a Hybrid storage setup using Redis or Postgres for JSON data, plus S3 or an S3-compatible service for files.  See [Storage Setup](storage.md) for details.
 - You will need a custom domain configured and TLS certs created and ready to attach.
 - You have your xyOps configuration file customized and ready to go ([config.json](https://github.com/pixlcore/xyops/blob/main/sample_conf/config.json)) (see below).
 
@@ -373,7 +373,7 @@ A few things to note here:
 
 ## External Storage
 
-For using an external storage system, you have [several to choose from](storage.md).  We currently recommend **MinIO**, because it has an open source community edition, it is S3 compatible, and it handles both data and files in the same engine.
+For using an external storage system, you have [several to choose from](storage.md).  For live production multi-conductor setups, we currently recommend **Redis + S3** or **Postgres + S3** via the Hybrid storage engine.  The S3 side can be AWS S3 or an S3-compatible service such as MinIO or RustFS.
 
 For more details, see the [Storage Setup Guide](storage.md).
 

docs/scaling.md

@@ -9,7 +9,7 @@ Running xyOps in live production with lots of servers and/or lots of running job
 - CPU cores: xyOps is multi-process and highly concurrent. More cores help the scheduler, web server, storage I/O, and log compression run smoothly under load.
 - RAM: Add headroom for the Node.js heap, in-process caches, storage engine caches, and OS page cache. RAM directly improves cache hit rates and reduces disk/remote I/O.
 - Storage: Prefer fast SSD/NVMe for local Filesystem/SQLite and log archives. Ensure enough IOPS for parallel job logs, snapshots, and uploads.
-- Network: For large fleets, ensure good NIC throughput and low latency between conductors and workers. If using external storage (S3, Redis, MinIO), place conductors close to it.
+- Network: For large fleets, ensure good NIC throughput and low latency between conductors, workers, and shared storage. For multi-conductor production, place conductors close to both the JSON data store (Redis or Postgres) and the file store (S3 or S3-compatible).
 - OS limits: Increase file descriptor and process limits for busy nodes (e.g. `ulimit -n`, systemd Limits). Ensure swap is configured conservatively to avoid heap thrash.
 
 ## Increase Node.js Memory
@@ -53,15 +53,15 @@ Or, you can set it globally in the main [satellite.config](config.md#satellite-c
 
 ## Multi-Conductor Setups
 
-Multi-conductor requires external shared storage so all conductors see the same state. See [Multi-Conductor with Nginx](hosting.md#multi-conductor-with-nginx).
+Multi-conductor requires shared external storage so all conductors see the same state. For live production, use a Hybrid storage setup with Redis or Postgres for JSON data, plus S3 or an S3-compatible service for files. Redis and Postgres support native storage transactions, so the database owns the commit or rollback decision if a conductor fails during a transaction.
 
 For details, see the [Storage Setup Guide](storage.md).
 
 ### NFS Warning
 
-While it is possible to use the [Filesystem](https://github.com/jhuckaby/pixl-server-storage#local-filesystem) engine with an NFS mount, this is **only** recommended if used in conjunction with the [Hybrid](https://github.com/jhuckaby/pixl-server-storage#hybrid) engine, where the Filesystem is set as the `binaryEngine` (to store only files), and the `docEngine` is set to a dedicated key/value store like [MinIO or RustFS](https://github.com/jhuckaby/pixl-server-storage#s3-compatible-services).
+While it is possible to use the [Filesystem](https://github.com/jhuckaby/pixl-server-storage#local-filesystem) engine with an NFS mount, this is **only** recommended if used in conjunction with the [Hybrid](https://github.com/jhuckaby/pixl-server-storage#hybrid) engine, where the Filesystem is set as the `binaryEngine` to store only files, and the `docEngine` is set to Redis or Postgres for JSON data.
 
-The reason is, xyOps reads and writes thousands upon thousands of tiny key/value records as part of its [database system](https://github.com/jhuckaby/pixl-server-unbase), and this is *extremely* difficult for NFS to deal with at scale.
+The reason is, xyOps reads and writes thousands upon thousands of tiny key/value records as part of its [database system](https://github.com/jhuckaby/pixl-server-unbase), and this is *extremely* difficult for NFS to deal with at scale. Also, NFS does not provide native pixl-server-storage transactions, so it should not be the document store for multi-conductor production.
 
 ## Automated Backups
 
@@ -204,5 +204,5 @@ This would limit traffic to 100 requests/sec per IP, utilizing up to 20MB of IP
 ## References
 
 - [xyOps Self-Hosting Guide](hosting.md)
-- [Storage engines and Hybrid](https://github.com/jhuckaby/pixl-server-storage#engines)
+- [xyOps Storage Setup Guide](storage.md)
 - [Web server documentation](https://github.com/jhuckaby/pixl-server-web)

docs/sso.md

@@ -577,7 +577,7 @@ For a load balanced multi-conductor setup with Nginx w/TLS and OAuth-Proxy for S
 
 A few prerequisites for this setup:
 
-- For multi-conductor setups, **you must have a central external storage backend**, such as [MinIO](storage.md#minio), [RustFS](storage.md#rustfs), or a [Hybrid storage](storage.md#hybrid-configurations) setup.  Do not use local Docker data volumes for multi-conductor.
+- For multi-conductor setups, **you must have shared external storage**.  For live production, we recommend a Hybrid storage setup using Redis or Postgres for JSON data, plus S3 or an S3-compatible service for files.  Do not use local Docker data volumes for multi-conductor.  See [Storage Setup](storage.md) for details.
 - You will need a custom domain configured and TLS certs created and ready to attach.
 - You have your xyOps configuration files customized and ready to go ([config.json](https://github.com/pixlcore/xyops/blob/main/sample_conf/config.json) and [sso.json](https://github.com/pixlcore/xyops/blob/main/sample_conf/sso.json)) (see below for details).
 - And of course you should have a pretested SSO configuration for OAuth2-Proxy, so you are confident that piece works before integrating it here.
@@ -645,7 +645,7 @@ Next is the OAuth2-Proxy setup (we use the official Docker image here).  Configu
 - `OAUTH2_PROXY_SET_XAUTHREQUEST` is set to `true`.  This returns the set of trusted headers in auth_request mode.
 - `OAUTH2_PROXY_SKIP_AUTH_ROUTES` has been removed, as OAuth2-Proxy doesn't actually do any routing in this configuration.
 
-Once you have those two components running, we can fire up the xyOps backend.  This is listed separately as you'll usually want to run these on dedicated servers.  Before starting multiple conductors, configure xyOps to use a central storage engine as described in the [Storage Setup Guide](storage.md).  Here is the multi-conductor configuration as a single Docker compose file.  For additional conductor servers you can duplicate this service, change the hostname, and point each conductor at the same central storage configuration:
+Once you have those two components running, we can fire up the xyOps backend.  This is listed separately as you'll usually want to run these on dedicated servers.  Before starting multiple conductors, configure xyOps to use shared external storage as described in the [Storage Setup Guide](storage.md).  Here is the multi-conductor configuration as a single Docker compose file.  For additional conductor servers you can duplicate this service, change the hostname, and point each conductor at the same shared storage configuration:
 
 ```yaml
 services:
@@ -669,10 +669,10 @@ A few things to note here:
 - We're using our official xyOps Docker image, but you can always [build your own from source](https://github.com/pixlcore/xyops/blob/main/Dockerfile).
 - All conductor server hostnames need to be listed in the `XYOPS_masters` environment variable, comma-separated.
 - All conductor servers need to be able to route to each other via their hostnames, so they can self-negotiate and hold elections.
-- All conductor servers must share the same central storage backend.  Do not mount separate local `/opt/xyops/data` volumes per conductor, as that will create split-brain data.
+- All conductor servers must use the same shared external storage configuration.  Do not mount separate local `/opt/xyops/data` volumes per conductor, as that will create split-brain data.
 - The timezone (`TZ`) should be set to your company's main timezone, so things like midnight log rotation and daily stat resets work as expected.
 
-For the xyOps container, we are bind mapping local host directories for configuration and logs (`./xyops-conf` and `./xyops-logs`).  Please change those paths to appropriate locations on the host where you want these files stored.  Launch the container once, and it will generate all the config files for you.  Then configure a central storage engine in `./xyops-conf/config.json` before bringing up multiple conductors.  See the [xyOps Configuration Guide](config.md) and [Storage Setup Guide](storage.md) for details.  Specifically though, let's talk about `sso.json` for this configuration.  This file is largely discussed above (see [Configuration](#configuration)), but the [Header Map](#header-map) in particular is going to be different for Nginx + OAuth2-Proxy:
+For the xyOps container, we are bind mapping local host directories for configuration and logs (`./xyops-conf` and `./xyops-logs`).  Please change those paths to appropriate locations on the host where you want these files stored.  Launch the container once, and it will generate all the config files for you.  Then configure shared external storage in `./xyops-conf/config.json` before bringing up multiple conductors.  See the [xyOps Configuration Guide](config.md) and [Storage Setup Guide](storage.md) for details.  Specifically though, let's talk about `sso.json` for this configuration.  This file is largely discussed above (see [Configuration](#configuration)), but the [Header Map](#header-map) in particular is going to be different for Nginx + OAuth2-Proxy:
 
 ```json
 "header_map": {