Skip to content

Updates for ThingSet Protocol draft v0.5 #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
martinjaeger merged 8 commits into from
Mar 16, 2022
Merged

Updates for ThingSet Protocol draft v0.5 #24

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
45b91eb
Add nested JSON support for statements
martinjaeger Feb 4, 2022
5699351
Tests: Change data to cover root level item
martinjaeger Feb 9, 2022
7ce2093
Use bit field for struct data_object to reduce memory footprint
martinjaeger Feb 9, 2022
a2910b5
Use flag to store if item was updated
martinjaeger Feb 9, 2022
727cb4d
Support nested JSON also for subsets
martinjaeger Feb 10, 2022
c69d111
Revert "Use flag to store if item was updated"
martinjaeger Mar 1, 2022
7546f94
Add update notification via callback from patch functions
martinjaeger Mar 1, 2022
cd5c514
Improve documentation of detail element in data object
martinjaeger Mar 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions platformio.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ build_flags =
-D TS_64BIT_TYPES_SUPPORT=1
-D TS_DECFRAC_TYPE_SUPPORT=1
-D TS_BYTE_STRING_TYPE_SUPPORT=1
-D TS_NESTED_JSON=1

# include src directory (otherwise unit-tests will only include lib directory)
test_build_project_src = true
Expand Down
18 changes: 17 additions & 1 deletion src/thingset.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,27 @@ int ts_process(struct ts_context *ts, const uint8_t *request, size_t request_len
}
}

void ts_set_authentication(struct ts_context *ts, uint16_t flags)
void ts_set_authentication(struct ts_context *ts, uint8_t flags)
{
ts->_auth_flags = flags;
}

bool ts_check_updated(struct ts_context *ts, const uint16_t subsets, bool clear)
{
bool updated = false;

for (unsigned int i = 0; i < ts->num_objects; i++) {
if (ts->data_objects[i].updated == 1U && (ts->data_objects[i].subsets & subsets)) {
updated = true;
if (clear) {
ts->data_objects[i].updated = 0U;
}
}
}

return updated;
}

struct ts_data_object *ts_get_object_by_name(struct ts_context *ts, const char *name,
size_t len, int32_t parent)
{
Expand Down
50 changes: 34 additions & 16 deletions src/thingset.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,15 +341,15 @@ static inline void *ts_array_to_void(struct ts_array_info *ptr) { return (void *
#define TS_ROLE_EXP (1U << 1) // expert user
#define TS_ROLE_MKR (1U << 2) // maker

#define TS_READ_MASK 0x00FF // read flags stored in 4 least-significant bits
#define TS_WRITE_MASK 0xFF00 // write flags stored in 4 most-significant bits
#define TS_READ_MASK 0x0F // read flags stored in 4 least-significant bits
#define TS_WRITE_MASK 0xF0 // write flags stored in 4 most-significant bits

#define TS_USR_MASK (TS_ROLE_USR << 8 | TS_ROLE_USR)
#define TS_EXP_MASK (TS_ROLE_EXP << 8 | TS_ROLE_EXP)
#define TS_MKR_MASK (TS_ROLE_MKR << 8 | TS_ROLE_MKR)
#define TS_USR_MASK (TS_ROLE_USR << 4 | TS_ROLE_USR)
#define TS_EXP_MASK (TS_ROLE_EXP << 4 | TS_ROLE_EXP)
#define TS_MKR_MASK (TS_ROLE_MKR << 4 | TS_ROLE_MKR)

#define TS_READ(roles) ((roles) & TS_READ_MASK)
#define TS_WRITE(roles) (((roles) << 8) & TS_WRITE_MASK)
#define TS_WRITE(roles) (((roles) << 4) & TS_WRITE_MASK)
#define TS_READ_WRITE(roles) (TS_READ(roles) | TS_WRITE(roles))

#define TS_USR_R TS_READ(TS_ROLE_USR)
Expand Down Expand Up @@ -401,25 +401,29 @@ struct ts_data_object {
/**
* One of TS_TYPE_INT32, _FLOAT, ...
*/
const uint8_t type;
const uint32_t type : 4;

/**
* Exponent (10^exponent = factor to convert to SI unit) for decimal fraction type,
Copy link
Copy Markdown
Collaborator

@b0661 b0661 Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you improve the documentation a little bit in the course of the change. It took me quite a time to understand that this are three different usages of the detail information depending on the type of data object. And I'm stiill not shure how to handle the exponent of the decimal fraction type. It seems you expect this to be a fixed value and not something that can be adapted acc. to the value.

For doxygen it would be nice to have a brief description and a detailed description.

Copy link
Copy Markdown
Contributor Author

@martinjaeger martinjaeger Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Will improve the docs here.

The idea was to have it fixed so that e.g. the firmware could store values in mV as integers, but send out the data in V without any numeric conversion to float. In that case the exponent would be -3. But so far I've never actually used the decimal fraction type (I can't even remember if it's implemented...).

Copy link
Copy Markdown
Contributor Author

@martinjaeger martinjaeger Mar 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated the documentation, please have a look.

* decimal digits to use for printing of floats in JSON strings or
* length of string buffer for string type
*/
const int16_t detail;
const int32_t detail : 12;

/**
* Flags to define read/write access
*/
const uint16_t access;
const uint32_t access : 8;

/**
* Flags to assign data item to different data item subsets (e.g. for publication messages)
*/
uint16_t subsets;
uint32_t subsets : 7;

/**
* Bit to store if value has been updated via ThingSet (e.g. for NVM stoarage)
*/
uint32_t updated : 1;
};

/* support for legacy code with old nomenclature */
Expand Down Expand Up @@ -479,7 +483,7 @@ struct ts_context {
/**
* Stores current authentication status (authentication as "normal" user as default)
*/
uint16_t _auth_flags;
uint8_t _auth_flags;
};

/**
Expand Down Expand Up @@ -529,7 +533,16 @@ void ts_dump_json(struct ts_context *ts, ts_object_id_t obj_id, int level);
* @param ts Pointer to ThingSet context.
* @param flags Flags to define authentication level (1 = access allowed)
*/
void ts_set_authentication(struct ts_context *ts, uint16_t flags);
void ts_set_authentication(struct ts_context *ts, uint8_t flags);

/**
* Checks if any of the data objects belonging to the subsets was updated.
*
* @param ts Pointer to ThingSet context.
* @param subsets Flags to select which subset(s) of data items should be considered
* @param clear If set to true, the updated status is cleared after checking.
*/
bool ts_check_updated(struct ts_context *ts, const uint16_t subsets, bool clear);
Copy link
Copy Markdown
Collaborator

@b0661 b0661 Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need that? Isn't there anyway a change callback that may be used by an application?

Copy link
Copy Markdown
Contributor Author

@martinjaeger martinjaeger Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change callback is currently assigned to a group. If any of the data objects in that group are changed, the callback is called. But if we change data in multiple groups, a separate callback for every group would have to be assigned and multiple callbacks would have to be called.

Maybe a better approach would be to assign a callback for every patch request instead. I'll think about this, thanks for the pointer.

Copy link
Copy Markdown
Collaborator

@b0661 b0661 Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a better approach would be to assign a callback for every patch request instead. I'll think about this, thanks for the pointer.

A ThingSet context may have an update counter. On every patch request the update counter may be incremented and every data object that is changed will have it's own update counter set to the context's update count. The general callback can then search for updated data objects by the update count. As the update count in the data objects should be of small size (e.g. 2 bits) all object update counts that are equal to the current one shall be invalidated before a patch request is applied. By this the last (e.g. 4) "transactions" on the object database can be retrieved by an application. This may help for multi patch request updates to the object database.

The general patch callback should have a void(*cb)(uintxx_t update_count, other???) signature to care for update count races in the execution of several patch requests (the context update count may already be incremented again when the callback is executed).

Note: Commented again - last comment got somehow lost in GitHub

Copy link
Copy Markdown
Contributor Author

@martinjaeger martinjaeger Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I understand correctly. How would the counter value be transferred over the wire?

I don't think we should support multiple patch requests in parallel. If one patch request is still in progress, the next requester will have to wait until the request is finished.

Copy link
Copy Markdown
Collaborator

@b0661 b0661 Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I understand correctly. How would the counter value be transferred over the wire?

Never !-) The counter is just a way to mark a data object was changed. This way the callback function can search the database for changed data objects. The counter just helps to in case of asynchronous callback function execution and a new patch executed at the same time the callback function is still active from the last patch request. If we serialise patch -> callback -> patch a data changed flag would be sufficient.

I don't think we should support multiple patch requests in parallel. If one patch request is still in progress, the next requester will have to wait until the request is finished.

Yes - patch requests to the database have to be serialised. The design decision is whether a patch callback that is executed after the patch is applied can block further patch request to be executed until it finishes. In my opinion this should not be the case.

Copy link
Copy Markdown
Contributor Author

@martinjaeger martinjaeger Mar 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted the ts_check_updated function and introduced a callback for the patch functions. This avoids to iterate through all data objects for each patch request and is quite similar to the previous approach where the callback was assigned per group.

Now the application can decide what to do after a patch request. It can either store the updated data in NVM synchronously in the callback (which would delay the ThingSet response) or it stores the data asynchronously and takes care of any race conditions. Handling asynchronous data processing inside the library will make it too complicated for very simple devices with low processing power, I think.


/**
* Retrieve data in JSON format for given subset(s).
Expand Down Expand Up @@ -676,7 +689,7 @@ int ts_bin_pub_can(struct ts_context *ts, int *start_pos, uint16_t subset, uint8
*
* @returns ThingSet status code
*/
int ts_bin_import(struct ts_context *ts, uint8_t *data, size_t len, uint16_t auth_flags,
int ts_bin_import(struct ts_context *ts, uint8_t *data, size_t len, uint8_t auth_flags,
uint16_t subsets);

/**
Expand Down Expand Up @@ -760,11 +773,16 @@ class ThingSet
ts_dump_json(&ts, obj_id, level);
};

inline void set_authentication(uint16_t flags)
inline void set_authentication(uint8_t flags)
{
ts_set_authentication(&ts, flags);
};

inline bool check_updated(const uint16_t subsets, bool clear)
{
return ts_check_updated(&ts, subsets, clear);
};

inline int txt_export(char *buf, size_t size, const uint16_t subsets)
{
return ts_txt_export(&ts, buf, size, subsets);
Expand All @@ -790,7 +808,7 @@ class ThingSet
return ts_bin_export(&ts, buf, size, subsets);
};

inline int bin_import(uint8_t *buf, size_t size, uint16_t auth_flags, const uint16_t subsets)
inline int bin_import(uint8_t *buf, size_t size, uint8_t auth_flags, const uint16_t subsets)
{
return ts_bin_import(&ts, buf, size, auth_flags, subsets);
};
Expand Down Expand Up @@ -880,7 +898,7 @@ class ThingSet
*
* @returns ThingSet status code
*/
inline int bin_sub(uint8_t *cbor_data, size_t len, uint16_t auth_flags, uint16_t subsets)
inline int bin_sub(uint8_t *cbor_data, size_t len, uint8_t auth_flags, uint16_t subsets)
__attribute__((deprecated))
{
return ts_bin_import(&ts, cbor_data + 1, len - 1, auth_flags, subsets);
Expand Down
10 changes: 6 additions & 4 deletions src/thingset_bin.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@ static int cbor_serialize_array_type(uint8_t *buf, size_t size,
const struct ts_data_object *data_obj);


static int cbor_deserialize_data_obj(const uint8_t *buf, const struct ts_data_object *data_obj)
static int cbor_deserialize_data_obj(const uint8_t *buf, struct ts_data_object *data_obj)
{
data_obj->updated = 1;

switch (data_obj->type) {
#if TS_64BIT_TYPES_SUPPORT
case TS_T_UINT64:
Expand Down Expand Up @@ -345,7 +347,7 @@ int ts_bin_fetch(struct ts_context *ts, const struct ts_data_object *parent, uin
}
}

int ts_bin_import(struct ts_context *ts, uint8_t *data, size_t len, uint16_t auth_flags,
int ts_bin_import(struct ts_context *ts, uint8_t *data, size_t len, uint8_t auth_flags,
uint16_t subsets)
{
uint8_t resp_tmp[1] = {}; // only one character as response expected
Expand All @@ -358,7 +360,7 @@ int ts_bin_import(struct ts_context *ts, uint8_t *data, size_t len, uint16_t aut
}

int ts_bin_patch(struct ts_context *ts, const struct ts_data_object *parent,
unsigned int pos_payload, uint16_t auth_flags, uint16_t subsets)
unsigned int pos_payload, uint8_t auth_flags, uint16_t subsets)
{
unsigned int pos_req = pos_payload;
uint16_t num_elements, element = 0;
Expand All @@ -383,7 +385,7 @@ int ts_bin_patch(struct ts_context *ts, const struct ts_data_object *parent,
}
pos_req += num_bytes;

const struct ts_data_object* object = ts_get_object_by_id(ts, id);
struct ts_data_object* object = ts_get_object_by_id(ts, id);
if (object) {
if ((object->access & TS_WRITE_MASK & auth_flags) == 0) {
if (object->access & TS_WRITE_MASK) {
Expand Down
4 changes: 2 additions & 2 deletions src/thingset_priv.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ int ts_txt_patch(struct ts_context *ts, const struct ts_data_object *parent);
* @param subsets Bitset to specifiy data item subsets to be considered, 0 to ignore
*/
int ts_bin_patch(struct ts_context *ts, const struct ts_data_object *parent,
unsigned int pos_payload, uint16_t auth_flags, uint16_t subsets);
unsigned int pos_payload, uint8_t auth_flags, uint16_t subsets);

/**
* POST request to append data.
Expand Down Expand Up @@ -200,7 +200,7 @@ int ts_json_serialize_name_value(struct ts_context *ts, char *buf, size_t size,
* @returns Number of tokens processed (always 1) or 0 in case of error
*/
int ts_json_deserialize_value(struct ts_context *ts, char *buf, size_t len, jsmntype_t type,
const struct ts_data_object *object);
struct ts_data_object *object);

#ifdef __cplusplus
} /* extern "C" */
Expand Down
83 changes: 77 additions & 6 deletions src/thingset_txt.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,21 @@ int ts_json_serialize_value(struct ts_context *ts, char *buf, size_t size,
pos = snprintf(&buf[pos], size - pos, "[");
for (unsigned int i = 0; i < ts->num_objects; i++) {
if (ts->data_objects[i].subsets & (uint16_t)object->detail) {
#if TS_NESTED_JSON
if (ts->data_objects[i].parent == 0) {
pos += snprintf(&buf[pos], size - pos, "\"%s\",", ts->data_objects[i].name);
}
else {
struct ts_data_object *parent_obj =
ts_get_object_by_id(ts, ts->data_objects[i].parent);
if (parent_obj != NULL) {
pos += snprintf(&buf[pos], size - pos, "\"%s/%s\",",
parent_obj->name, ts->data_objects[i].name);
}
}
#else
pos += snprintf(&buf[pos], size - pos, "\"%s\",", ts->data_objects[i].name);
#endif
}
}
if (pos > 1) {
Expand Down Expand Up @@ -423,7 +437,7 @@ int ts_txt_fetch(struct ts_context *ts, const struct ts_data_object *parent)
}

int ts_json_deserialize_value(struct ts_context *ts, char *buf, size_t len, jsmntype_t type,
const struct ts_data_object *object)
struct ts_data_object *object)
{
#if TS_DECFRAC_TYPE_SUPPORT
float tmp;
Expand Down Expand Up @@ -496,6 +510,8 @@ int ts_json_deserialize_value(struct ts_context *ts, char *buf, size_t len, jsmn
return 0;
}

object->updated = 1;

return 1; // value always contained in one token (arrays not yet supported)
}

Expand Down Expand Up @@ -529,7 +545,7 @@ int ts_txt_patch(struct ts_context *ts, const struct ts_data_object *parent)
return ts_txt_response(ts, TS_STATUS_BAD_REQUEST);
}

const struct ts_data_object* object = ts_get_object_by_name(ts,
struct ts_data_object* object = ts_get_object_by_name(ts,
ts->json_str + ts->tokens[tok].start,
ts->tokens[tok].end - ts->tokens[tok].start, parent_id);

Expand Down Expand Up @@ -583,7 +599,7 @@ int ts_txt_patch(struct ts_context *ts, const struct ts_data_object *parent)
// actually write data
while (tok + 1 < ts->tok_count) {

const struct ts_data_object *object =
struct ts_data_object *object =
ts_get_object_by_name(ts, ts->json_str + ts->tokens[tok].start,
ts->tokens[tok].end - ts->tokens[tok].start, parent_id);

Expand Down Expand Up @@ -683,10 +699,15 @@ int ts_txt_create(struct ts_context *ts, const struct ts_data_object *object)
}
else if (object->type == TS_T_SUBSET) {
if (ts->tokens[0].type == JSMN_STRING) {
struct ts_data_object *del_object = ts_get_object_by_name(ts, ts->json_str +
#if TS_NESTED_JSON
struct ts_data_object *add_object = ts_get_object_by_path(ts, ts->json_str +
ts->tokens[0].start, ts->tokens[0].end - ts->tokens[0].start);
#else
struct ts_data_object *add_object = ts_get_object_by_name(ts, ts->json_str +
ts->tokens[0].start, ts->tokens[0].end - ts->tokens[0].start, -1);
if (del_object != NULL) {
del_object->subsets |= (uint16_t)object->detail;
#endif
if (add_object != NULL) {
add_object->subsets |= (uint16_t)object->detail;
return ts_txt_response(ts, TS_STATUS_CREATED);
}
return ts_txt_response(ts, TS_STATUS_NOT_FOUND);
Expand All @@ -708,8 +729,13 @@ int ts_txt_delete(struct ts_context *ts, const struct ts_data_object *object)
}
else if (object->type == TS_T_SUBSET) {
if (ts->tokens[0].type == JSMN_STRING) {
#if TS_NESTED_JSON
struct ts_data_object *del_object = ts_get_object_by_path(ts, ts->json_str +
ts->tokens[0].start, ts->tokens[0].end - ts->tokens[0].start);
#else
struct ts_data_object *del_object = ts_get_object_by_name(ts, ts->json_str +
ts->tokens[0].start, ts->tokens[0].end - ts->tokens[0].start, -1);
#endif
if (del_object != NULL) {
del_object->subsets &= ~((uint16_t)object->detail);
return ts_txt_response(ts, TS_STATUS_DELETED);
Expand Down Expand Up @@ -769,6 +795,49 @@ int ts_txt_exec(struct ts_context *ts, const struct ts_data_object *object)
return ts_txt_response(ts, TS_STATUS_VALID);
}

#if TS_NESTED_JSON

/* currently only supporting nesting of depth 1 */
int ts_txt_export(struct ts_context *ts, char *buf, size_t buf_size, uint16_t subsets)
{
unsigned int len = 1;
buf[0] = '{';
uint16_t prev_parent = 0;
unsigned int depth = 0;

for (unsigned int i = 0; i < ts->num_objects; i++) {
if (ts->data_objects[i].subsets & subsets) {
const uint16_t parent_id = ts->data_objects[i].parent;
if (prev_parent != parent_id) {
if (prev_parent != 0) {
// close object of previous parent
buf[len-1] = '}'; // overwrite comma
buf[len++] = ',';
}
struct ts_data_object *parent = ts_get_object_by_id(ts, parent_id);
len += snprintf(&buf[len], buf_size - len, "\"%s\":{", parent->name);
prev_parent = parent_id;
depth = 1;
}
len += ts_json_serialize_name_value(ts, &buf[len], buf_size - len,
&ts->data_objects[i]);
}
if (len >= buf_size - 1 - depth) {
return 0;
}
}

buf[len-1] = '}'; // overwrite comma

if (depth == 1) {
buf[len++] = '}';
}

return len;
}

#else

int ts_txt_export(struct ts_context *ts, char *buf, size_t buf_size, uint16_t subsets)
{
unsigned int len = 1;
Expand All @@ -789,6 +858,8 @@ int ts_txt_export(struct ts_context *ts, char *buf, size_t buf_size, uint16_t su
return len;
}

#endif /* TS_NESTED_JSON */

int ts_txt_statement(struct ts_context *ts, char *buf, size_t buf_size,
struct ts_data_object *object)
{
Expand Down
20 changes: 20 additions & 0 deletions src/ts_config.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,4 +75,24 @@
#define TS_BYTE_STRING_TYPE_SUPPORT CONFIG_THINGSET_BYTE_STRING_TYPE_SUPPORT
#endif

/*
* The ThingSet specification v0.5 introduces a different data layout compared to previous
* versions where the data is grouped by entities of the device (like battery, actuator)
* instead of the data type (e.g. measurement, configuration). The data type is described by
* a single character prefix in the item name.
* The new grouping allows to have same item names in different groups, so for unambigous
* description of the data the nested structure has to be maintained in statements.
*
* With nested JSON enabled, requesting names for IDs will also return the entire path (as
* a JSON pointer) instead of just the data item name.
*
* This option is introduced to maintain compatibility with legacy firmware and will be
* enabled by default in the future.
*/
#if !defined(TS_NESTED_JSON) && !defined(CONFIG_THINGSET_NESTED_JSON)
#define TS_NESTED_JSON 0 // default: no nested JSON for legacy code
#elif !defined(TS_NESTED_JSON)
#define TS_NESTED_JSON CONFIG_THINGSET_NESTED_JSON
#endif

#endif /* TS_CONFIG_H_ */
Loading