API Functional Tests

[benhaynes]

Feature Request

Create API Functional Tests

What problem does this feature solve?

Knowing that the API is working as expected; more thoroughly testing the API after updates/fixes... and before releasing.

How do you think this should be implemented?

TBD. @shealavington has been working on this for a while, but we need to align on exactly how these tests will be structured. Here is what Shea wrote:

I've got some tests for the Items endpoint running. There's a few changes I'd like to make (Such as outputting successful tests to the console), but I guess the first iteration is almost ready to be master-merged. Though on that note, I'm currently testing the 'Demo' directly, therefore some fail (deletions) as it doesn't spawn a new instance every time haha.

• 'Items' Tests:
  • Create: Create a single item & Create multiple items in one payload.
  • Read: Get item, Get multiple items, Get all items (by default should max out at 200 items)
  • Update: Update item, Update multiple items with the same data, Update multiple items with different data.
  • Delete: Delete item, delete multiple items.
• Server: Ping should Pong
• Project: Expected keys should exist when authed, and shouldn't when not authed.

Status

Authentication

• Tokens
• Get Auth Token
• Refresh Auth Token
• Password Reset Request
• Password Reset
• SSO

Query Parameters

• Metadata
• Fields
• Limit
• Offset
• Single
• Sorting
• Status
• Filtering
• Language
• Search Query
• Groups
• Skip Activity Logging
• Activity Comment

Items

• Get Item
• Get Multiple Items
• Get All Items
• Create Item
• Create Items
• Update Item
• Update Items
• Delete Item
• Delete Items
• Get Item Revision
• Get Item Revisions
• Revert Item
• Revert Items?

System

• Activity
• Collections
• Collection Presets
• Fields
• Files
• Folders
• Permissions
• Relations
• Revisions
• Roles
• Settings
• Users

Utilities

• Hash String
• Match Hashed String
• Generate Random String

Mail

• Send Email

Extensions

• Get Extensions
• Get Interface
• Get Page
• Get Custom Endpoint

Server

• Information
• Ping

Projects

• Information
• Update
• Create Project
• Projects Auth Config
• Projects CORS Config
• Projects Cache Config
• Projects Storage Config
• Projects Mail Config
• Projects Rate Limit Config

Field Types

• Webhooks

Directus Objects

• Activity Object
• Activity Seen Object
• Collection Object
• Collection Preset Object
• Field Object
• File Object
• Folder Object
• Permission Object
• Relation Object
• Revision Object
• Role Object
• Setting Object
• User Object
• User Role Object

SCIM

• Overview
• Create SCIM User
• Get SCIM Users
• Get SCIM User
• Update SCIM User
• Create SCIM Group
• Get SCIM Groups
• Get SCIM Group
• Update SCIM Group
• Delete SCIM Group

[benhaynes]

We all want to keep this as simple as possible, and I know @shealavington had some very clean PoCs... but @WoLfulus raised a few points for using a proprietary cli command instead of relying on "off the shelf" stuff from the mocha docs:

• Use of docker was important
• So you could run the tests from anywhere and not just the test directory (global install)
• So you could create tests for your own use cases

We need to discuss this and find a good balance so that we can keep writing tests. Can we keep the core tests suite super simple (off the shelf) and then allow for more complex tests to happen through the CLI?

[rijkvanzanten]

I think we should start with a very KISS MVP

IMO having a single start command with a flag to set the API URL should be enough for now

[benhaynes]

Cool — simple is good!

@shealavington — does this work for you? Any recommendations to simplify things further?

[shealavington]

My recommendation for simplification is the confusion around directest which doesn't seem to want to work on one of the computers I use. I wonder if there's a way @WoLfulus could make this easier. I don't truly understand the intricacies of what was done with that.

Other than that, I agree... We should KISS 😉 to start with.

[WoLfulus]

I'm ok with whatever you guys decide, but having a way to parametrize the target url/api installation would be awesome (since I can use it to test docker images later)

[sayore]

If nobody is currently on to it, i would be making some tests for the API. With that in mind i would go this route:

To make a simple Testsuite, there probably should be some kind of way to make a collection via API Calls, and destroying it after testing.

I think a testing Script via NodeJS or plain JS, using node-fetch could already give a very quick and dirty way to test everything.

The steps should simply be:

1. Create a Testcollection
2. Run each API Call in a simple configuration
3. Delete Testcollection

The Tests can just be
[Partial Pseudocode]

var url = args[1]; //unsure rn what the correct way of fetching the args was

var Tests = [ { query, result, [testname, testgroup, description]:optional }... ]
Tests.map(e => { fetch(e.url).then((result)=>{ e.result = result; return e; } ) })
Tests.foreach(e => { /* Handle Results*/ });

The results could be printed in console, or as an HTML File, using a simple UI Framework to make a Clickable SlideDown Menu.

Edit: Note to self:
If working on this maybe "query" should be either string or function OR add validation-function into array.

[benhaynes]

@edenprojectde — that would be awesome! We would love all the help we can get! Right now @shealavington (with guidance from our Tech lead, @rijkvanzanten) has been leading this effort.

Shea, any thoughts on this approach? Could we get all of this standardized and started so that it's easier for contributors to add more tests one-by-one?

[rijkvanzanten]

Also feel free to reach out on Slack, is a little easier to discuss over there 🙂