[adapter-commons] filters and operators

[bwgjoseph]

Hi,

Continuing the discussion from Slack

Conversation from Slack

Hey @daffl, I see that the filterQuery supports additional filters, and operators but how do I go about using this? https://github.com/feathersjs/feathers/blob/dove/packages/adapter-commons/src/filter-query.ts#L103

I have a query that is

// can be as a options to ignore all fields
service.find({ query: { name: 'POST', description: 'hello', $ignoreCase: true } });
// or can be per field
service.find({ query: { name: { $eq: 'POST', $ignoreCase: true }, description: 'hello' } });

Is there a way for filterQuery to extract the split out to filters and so on correct by passing in the additional options ? I hope there is at least ways pass in as the options (first query above) with filterQuery splitting out as filters/operators. as for per field, I think I can still do post-processing after filterQuery split out the query

this question is for database service adapter..

Thanks

---

I think I got it working for the filters

const { filters, paginate, query } = this.filterQuery(params, { filters: ['$ignoreCase'] });

Initially, I was confused because I saw that you set the default as {} https://github.com/feathersjs/feathers/blob/3f4db68d6700c7d9023ecd17d0d39893f75a19fd/packages/adapter-commons/src/filter-query.ts#L105

So I'm not sure how to pass it in.. So I just tried to pass in as a array, and it seem to work but not sure if that is correct. If you could let me know that would be great! As I'm not quite sure why additionalFilters is default to {} and additionalOperators is default to []

---

And it seem like this is all I need for to support both filters and operators

for as a option, and as individual field

const { filters, paginate, query } = this.filterQuery(params, { filters: ['$ignoreCase'], operators: ['$eq', '$ignoreCase'] });

---

ok, I'm probably thinking too complex, for additional operators. can just pass in when init the service with whitelist options. So it's not the adapter that needs to allow this explicitly.

But I still need to pass in $ignoreCase as the additional filters since this can't be passed in via the service options.

---

Daffl

I think some of this is actually broken or at least not working very well (e.g. discussion in https://github.com/feathersjs/feathers/issues/1971) in what order these are happening, what specifically the difference between operators and filters is etc.

So your questions are totally valid. Maybe we should move this into an issue to see what can be done to improve the semantics and API for v5.

Summary

This issue is those who use filterQuery from adapter-commons package. And I assume, mostly used by database-adapter-service package (e.g feathers-mongoose).

filterQuery provides a way to extract out the known operators and filters and also allows to pass in additional filters and operators. For example, we can pass in whitelist in the service options to the service to allow that service to accept the database-specific operators. But there is no way to pass in additional filters from the service options. Hence, if the database-adapter does not support it either by allowing it explicitly in the adapter or provide a way to let the client pass in via options then the adapter could be considered a missing feature. Should that be the job of the adapter to provide the feature out of the box, or should it always allow the client to determine what to pass through?

There is also confusion around what operators and filters should be

Related

#1971
feathersjs-ecosystem/feathers-mongoose#391

---

Problem Statement

I think the issue here would be:

1. To identify what does filter and operator really mean/is?
2. To provide a way to pass in additional filters and operators via service options so that the adapter can pass down to filterQuery
3. To identify if it's the responsibility of the client or database adapter to provide database-specific operators/filters functionality (see adapter commons whitelist skips filter items? #1971)

---

filters and operators meant (roughly) the same thing right now, and I think there is a need to properly differentiate it. If we look at .find method in general, it probably would look like this

.find(filters, options);

Some example/variation of find would look like this

.find({ name: 'joseph'}, { limit: 10 });

.find({ name: { $in: ['joseph', 'david'] }, { limit: 10 });

So, filters are just a more general term to indicate how you like to search for your data set, and you can use operators to help you with that. And then we pass in the options to affect (think sort, limit) the final look (of the result) before returning back to the caller.

However, in feathers world, we pass in everything within query and filterQuery method will interpret and breakdown before passing to the actual driver API call through whichever ODM/ORM you are using (correct me if I'm wrong)

This is how we will write it in term of feathers query

.find({ query: { name: { $in: ['joseph', 'david'] }, $limit: 10 });

Which breaks down to

query: { name: { $in: ['joseph', 'david'] }
filters: { $limit: 10 }

This is different from how I describe above where it would look like this

filters: { name: { $in: ['joseph', 'david'] } // operators is $in
options: { $limit: 10 }

I guess you can have operators for both filters and options which translate to

filters: { name: { $in: ['joseph', 'david'] } // operators is $in
options: { $limit: 10 } // operators is $limit

Proposal

First, with regards to how the actual term and usage of filters, operators, or options will need further discussion to see how there could be a better way to term this or come to a common understanding of what it means at least to Feathers Database Common API. Since that (maybe) different ODM/ORM has different terminology of those, it might be hard to define one that is valid across all different types of ODM/ORM.

The feathers database common API can then describe what are the default list of operators it allows for both filters and options like the list that is in querying right now

My suggestion is to call it filters and options but also keep the operators term.

.find(filters, options)

.find({ query: { name: { $isNull: true }, $ignoreCase: true });

// filters
{ name: { $isNull: true }
// options
{ $ignoreCase: true }

operators can be classify as filterOperators and optionOperators

// service options
const options = {
  optionOperators: ['$ignoreCase'],
  filterOperators: ['$isMissing', '$inNull'],
};

Second, to allow to pass in additionalFilters via the service options, because, as far as I understand, whitelist in the service options is only meant for Operators and not Filters

const options = {
  Model: getModel('posts'),
  // could be `adapter` to be generic
  ottoman: {
    optionOperators: ['$ignoreCase'],
    filterOperators: ['$inMissing', '$inNull'],
    lean: true,
    consistency: SearchConsistency.GLOBAL,
  },
  events: ['testing'],
};

However, doing so has a disadvantage which is putting the responsibility on the client, and where not all clients are advanced users and know the different operators/filters to allow/disallow via the service options. On the other hand, if the client needs to use such operator/filter, then it's not difficult for the client to just allow it via the service options rather than the adapter providing it out of the box. So long the documentation states clearly.

Bonus

I was wondering if there is a way to standardize how database-adapter accept options when initializing service.

For example, in feathers-ottoman,

const options = {
  Model: getModel('posts'),
  // could be `adapter` to be generic
  ottoman: {
    lean: true,
    consistency: SearchConsistency.GLOBAL,
  },
  events: ['testing'],
};

Any adapter specifics options to go into ottoman object. This way, it makes it easier to identify/differentiate the options for feathers-service and for the adapters

---

Not sure if this is more of a discussion, if so, do move over to discussion and we can continue from there.

Let me know if any points are unclear, and I can clear those up, or re-words those.