New Feature: Programmatic bucket data access, via new API: write_bucket_data.

Author: jhuckaby

docs/api.md

@@ -458,6 +458,43 @@ And an example response:
 
 Deletions are permanent and cannot be undone.
 
+### write_bucket_data
+
+```
+POST /api/app/write_bucket_data/v1
+```
+
+This API allows you to write bucket data into a storage bucket.  The [edit_buckets](privileges.md#edit_buckets) privilege is required, as well as a valid user session or API Key.  The input parameters are as follows:
+
+| Property Name | Type | Description |
+|---------------|------|-------------|
+| `bucket` | String | **(Required)** The alphanumeric ID of the bucket to write data to. |
+| `data` | Object | **(Required)** The data object to shallow-merge into the bucket data. |
+| `fetch` | Boolean | Optional flag requesting the entire data object be returned in the API response. |
+
+Here is an example request:
+
+```json
+{
+	"id": "bme4wi6pg35",
+	"fetch": true,
+	"data": { "foo": "bar" }
+}
+```
+
+And an example response:
+
+```json
+{
+	"code": 0,
+	"data": { "foo": "bar", "other": 12345 }
+}
+```
+
+Notably, data passed to this API is *shallow-merged* into the bucket data.  In this way multiple "clients" can read/write data to the same bucket without affecting each other (as long as they use unique property names).  Locking is used to ensure only one read/write operation occurs at a time.  If multiple clients write the same property names the latter prevails.
+
+This API is designed to be called from within jobs (i.e. Event Plugin scripts), so it does not update the bucket record itself, nor log a user transaction.
+
 ### upload_bucket_files
 
 ```
@@ -474,6 +511,8 @@ The file properties are automatically set based on the user files themselves, in
 
 Note that bucket files are automatically added or replaced based on their normalized filenames.  Normalization involves converting anything other than alphanumerics, dashes and periods to underscores, and converting the filename to lowercase.
 
+This API is designed to be called from within jobs (i.e. Event Plugin scripts), so it does not update the bucket record itself, nor log a user transaction.
+
 ### delete_bucket_file
 
 ```

docs/buckets.md

@@ -43,11 +43,11 @@ See [Privileges](privileges.md#buckets) for specifics. Listing and fetching typi
 
 ## Using Buckets In Jobs
 
-Buckets integrate with jobs through two action types: Fetch Bucket and Store Bucket. You attach these as job actions with conditions controlling when they run.
+Buckets integrate with jobs through two action types: [Fetch Bucket](actions.md#fetch-bucket) and [Store Bucket](actions.md#store-bucket). You attach these as job actions with conditions controlling when they run.
 
 ### Fetch At Job Start
 
-Use Fetch Bucket with the `start` condition to pull bucket content into the job's input context before launch:
+Use [Fetch Bucket](actions.md#fetch-bucket) with the `start` condition to pull bucket content into the job's input context before launch:
 
 - **Data**: Shallow-merged into the job's `input.data`. Avoid key collisions or namespace your keys deliberately.
 - **Files**: Selected files are added to the job's input file list and staged into the job's temp directory on the remote server before the Plugin starts.
@@ -67,7 +67,7 @@ Example (JSON):
 
 ### Store On Completion
 
-Use Store Bucket with a completion condition (e.g., `success`, `error`, `complete`) to persist job outputs:
+Use [Store Bucket](actions.md#store-bucket) with a completion condition (e.g., `success`, `error`, `complete`) to persist job outputs:
 
 - **Data**: The job can emit output data which is written into the bucket when `bucket_sync` includes `data`.
 - **Files**: The job's output files can be filtered by `bucket_glob` and stored in the bucket when `bucket_sync` includes `files`.
@@ -110,6 +110,44 @@ Every bucket file includes a `path` (e.g., `files/bucket/<bucket_id>/<hash>/<fil
 GET https://your.xyops.example.com/files/bucket/bme4wi6pg35/bdY8zZ9nKynfFUb4xH6fA/report.csv
 ```
 
+These URLs have built-in authentication and are "stable" (i.e. permalinks) even if the files are replaced in the bucket (however, not if they are deleted and then re-added).
+
+## Programmatic Access
+
+Using the [get_bucket](api.md#get_bucket), [write_bucket_data](api.md#write_bucket_data) and [upload_bucket_files](api.md#upload_bucket_files) APIs, you can programmatically read and write bucket data and files at any time, including during a job run.  Here is how to set that up:
+
+- First, create a storage bucket, and save the new [Bucket.id](data.md#bucket-id).
+- Next, create an [API Key](api.md#api-keys), and grant it the [edit_buckets](privileges.md#edit_buckets) privilege.  Save the API key secret when prompted.
+- Then, create a [Secret Vault](secrets.md), and add your API Key and Bucket ID as variables (e.g. `XYOPS_API_KEY` and `XYOPS_BUCKET_ID`).
+- Assign the secret vault to your Event, Category or Plugin (the scope is up to you).
+
+When your job runs, you will now have access to your secret variables, and also a special magic variable called [Job.base_url](data.md#job-base_url) (also available as the `JOB_BASE_URL` environment variable).  Using these variables, you can write bucket data like this:
+
+```sh
+#!/bin/sh
+JSON_PAYLOAD='{ "data": { "foo": "bar", "number": 1234 } }'
+API_URL="$JOB_BASE_URL/api/app/write_bucket_data/v1?id=$XYOPS_BUCKET_ID"
+
+curl -sS "$API_URL" \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: $XYOPS_API_KEY" \
+  -d "$JSON_PAYLOAD" >/dev/null
+```
+
+And read it back like this:
+
+```sh
+#!/bin/sh
+API_URL="$JOB_BASE_URL/api/app/get_bucket/v1?id=$XYOPS_BUCKET_ID"
+RESPONSE=$(curl -sS -H "X-API-Key: $XYOPS_API_KEY" "$API_URL")
+
+echo "Response: $RESPONSE"
+```
+
+The [write_bucket_data](api.md#write_bucket_data) API is designed to be rugged, and can easily handle getting bombarded with many jobs hitting it at once.  It uses locking to ensure the bucket data doesn't get corrupted.  Also, each request performs a "shallow merge" write into the data, so multiple "clients" (events / workflows) can read/write different properties in the same bucket at the same time.
+
+Of course, you can get similar functionality using the [Store Bucket](actions.md#store-bucket) and [Fetch Bucket](actions.md#fetch-bucket) actions, but this way you have complete control over when the data is read and written, and you're not limited to the start and completion of the job.
+
 ## Tips
 
 - **Namespacing**: Use distinct keys in bucket JSON to avoid shallow-merge collisions with job input.

docs/data.md

@@ -856,6 +856,10 @@ This is set to `true` when the job was fired from an event test.  This is used t
 
 If any secrets were assigned to the job, this object will *temporarily* hold the decrypted key/value pairs for the job code to use.  They are **not** stored anywhere, and only passed to the Event Plugin running the job for immediate use.
 
+### Job.base_url
+
+While the job is running, i.e. in the data passed to the Event Plugin, the job object will contain a `base_url` property.  This will be a fully-qualified URL pointing to the current primary conductor server, using the same protocol, hostname and port that xySat used to connect.  This is useful if your job needs to call any xyOps APIs directly.
+
 ### Job.workflow
 
 When the job is itself a workflow, or a sub-job inside a workflow, this object will contain additional information.  See [JobWorkflow](#jobworkflow) for details.

lib/api/buckets.js

@@ -284,6 +284,68 @@ class Buckets {
 		} ); // loadSession
 	}
 
+	api_write_bucket_data(args, callback) {
+		// write bucket data, safely with locks, and without updating bucket itself
+		// note: data is shallow-merged
+		var self = this;
+		var params = Tools.mergeHashes( args.params, args.query );
+		if (!this.requireMaster(args, callback)) return;
+		
+		if (!this.requireParams(params, {
+			id: /^[a-z0-9_]+$/
+		}, callback)) return;
+		
+		this.loadSession(args, function(err, session, user) {
+			if (err) return self.doError('session', err.message, callback);
+			if (!self.requireValidUser(session, user, callback)) return;
+			if (!self.requirePrivilege(user, 'edit_buckets', callback)) return;
+			
+			args.user = user;
+			args.session = session;
+			
+			var bucket = Tools.findObject(self.buckets, { id: params.id });
+			if (!bucket) {
+				return self.doError('bucket', "Bucket not found: " + params.id, callback);
+			}
+			
+			var bucket_path = 'buckets/' + bucket.id;
+			var bucket_data = null;
+			
+			// read and write using locking
+			self.logDebug(6, "Merging data into storage bucket: " + bucket.id, params);
+			
+			async.series([
+				function(callback) {
+					// lock bucket
+					self.storage.lock( bucket_path, true, callback );
+				},
+				function(callback) {
+					// load bucket data
+					self.storage.get( bucket_path + '/data', function(err, data) {
+						if (err) return callback(err);
+						bucket_data = data;
+						callback();
+					} );
+				},
+				function(callback) {
+					// shallow-merge data and save
+					Tools.mergeHashInto( bucket_data, params.data || {} );
+					
+					// write data back to storage
+					self.storage.put( bucket_path + '/data', bucket_data, callback );
+				}
+			],
+			function(err) {
+				// all done
+				self.storage.unlock( bucket_path );
+				if (err) return self.doError('bucket', "Failed to store data in bucket: " + (err.message || err), callback);
+				
+				// return success
+				callback({ code: 0, data: params.fetch ? bucket_data : null });
+			});
+		} ); // loadSession
+	}
+	
 	api_upload_bucket_files(args, callback) {
 		// upload one or more files to storage bucket
 		var self = this;