Import async function

[moics30]

I need to import an async function but have an error because when the function is evaluated return a promise.
Is there any way to do that?

Thanks for advance.

[harrysarson]

Would it be possible to share a small code snippet showing what you are trying to do?

[moics30]

Sure!

const math = require('mathjs');
math.import({
  getExchangeRate: function () {
    return new Promise(resolve => resolve(3.45)); // e.g. Connect to database, or any other async request.
  },
});

const expr = "a = b*getExchangeRate()"; // expression to evaluate

When eval "getExchangeRate", the error occur because the get value is [object Promise]

Thanks for advance.

[josdejong]

@moics30 non of the functions of math.js support promises, so if one of the values of multiply is a promise, the function has no clue what to do. You will have to first resolve the promise, store the result in a variable, and use that in your calculations (clearly separate sync and async).

[atulmy]

I'm trying to do something like this:

parser.set('select', async params => {
  const count = await Model.count()

  return count
}

However, when calling the parser const response = parser.evaluate('select()') throws following error:

TypeError: Value has unknown type. Value: [object Promise]

Can we create async functions? Any help on this would be highly appreciated.

CC: @josdejong

[harrysarson]

Async functions are not something that mathjs supports. Saying that your example should work - I made a quick example: https://repl.it/repls/GargantuanAcclaimedCoins

const mathjs = require('mathjs');

const parser = new mathjs.parser();

parser.set('select', async params => {
  await Promise.resolve();
  return 17;
})

parser.evaluate('select()')
  .then(console.log);

which does what you would expect it to do :)

[atulmy]

That was super quick @harrysarson, thanks!

I tried your example and it does work. However in case of parser.evaluate('select() + 10') it actually fails with the same issue. Any idea how can we achieve that?

Updated example: https://repl.it/repls/UnselfishBrownAlphatest

[josdejong]

The parser has no support for async responses. An async function should be invoke with async (or by resolving the returned promise), and this is not what the parser does.

I think you can simply do something like this:

const mathjs = require('mathjs')

async function run () {
  const parser = new mathjs.parser()

  const count = await Model.count() // first resolve (async)
  parser.set('select', count)       // then set the results (sync)

  const result = parser.evaluate('select()')
  console.log(result)
}

run()

So first you resolve your async value and after that put the results in the parser.

Be careful in your examples not to run any evaluation before your await Model.count() is resolved, until then it will of course be undefined.

[atulmy]

Okay, will use a work around. Thanks for the quick help and building this awesome library @josdejong!

[josdejong]

I do not see this as a workaround but as a proper, valid solution. The Parser simply is not async, and adding support for it would not add anything. I don't think it would make your code simpler or better. It may even make it more tricky and complicated when you mix sync and async on various levels that way: as you can see from my example you have to await for the count to resolve before you can evaluate. It would be very easy to make the mistake to call set without await and then call evaluate as Harry demonstrates with his example :D

[josdejong]

Sorry my previous comment sounds a bit offensive, that was not my intention. Anyway, I think there is no need at this point to add async support and it's easy to write your own wrapper if needed. Will close this question now, feel free to reopen if needed.

[atulmy]

I think async support would really help when we are working with cases where user creates own formula with different params to it and which needs to be resolved from database.

[josdejong]

Atul can you give a few examples of your idea of using async functions inside the parser? Some use-cases that would be very hard, verbose, cumbersome, or simply impossible to achieve without having async support in mathjs itself?

[atulmy]

Hey @josdejong , thank you for considering the request.

I'm building a web app where users can create lists (eg: Cars or Bikes) enter key and values for each item in list (c1 = 10, b1 = 20). Users can later write formula along with passing parameters (keys) for computation:

const formula = `select(cars("c1", "c2"), bikes("b1", "b2")) + 5`

In above formula, c1, c2, b1, b2 are typed by user inside the formula itself which points to some values in database for example (c1 = 10, c2 = 15, b1 = 20, b2 = 25)

Custom function cars:

parser.set('cars', async function () {
  const cars = arguments // holds the parameters passed by user "c1", "c2"
  
  let sum = 0

  for(let c of cars) {
    sum += await Cars.findOne({ name: c }).value
  }

  return sum // will be 10 + 15 = 25
})

Custom function bikes:

parser.set('bikes', function () {
  const bikes = arguments // holds the parameters passed by user "b1", "b2"
  
  let sum = 0

  for(let b of bikes) {
    sum += await Bikes.findOne({ name: b }).value
  }

  return sum // will be 20 + 25 = 45
})

Custom function select:

parser.set('select', function () {
  const sum = 0

  for(let a of arguments) {
    sum += a
  }

  return sum // will be 25 + 45 = 70
})

Evaluating the result:

const parser = mathjs.parser()

const result = parser.evaluate(formula)

console.log(result) // will be 70 + 5 = 75

Looking forward to hearing your thoughts on this.

[josdejong]

Ah, now I understand you Atul. I thought you wanted to resolve the result of an async function and load it in the parser via parser.set, but you have async functions inside the expressions you want to use. That is a valid use case.

I think there is a straightforward way to add async support to mathjs functions. It will be a lot of work though. We can add a new data type Promise to mathjs, and extend every function of mathjs to support Promise input. It's only a few lines, but we need to implement it for every function, and need to add tests and extend the docs.

Here is a working proof of concept (you can achieve this by extending mathjs):

https://runkit.com/josdejong/5e1219882bec72001a472a21

const { create, all, factory } = require('mathjs')

const mathjs = create(all)

const createPromise = factory('Promise', ['typed'], ({ typed }) => {
  // add a new type Promise in mathjs
  typed.addType({
    name: 'Promise',
    test: function (x) {
      return x
        ? typeof x.then === 'function' && typeof x.catch === 'function'
        : false
    }
  })

  // create a conversion to convert from any value to a Promise,
  // so we can mix promise and non-promise inputs to our functions
  typed.addConversion({
    from: 'any',
    to: 'Promise',
    convert: x => Promise.resolve(x)
  })

  return Promise
})

// Create a factory which will extend the function addScalar with support for Promise
// To make this work for real, we should create a Promise implementation for ALL functions
// of mathjs
const createAddPromise = factory('addScalar', ['typed', 'Promise'], ({ typed, Promise }) => {
  return typed('addScalar', {
    'Promise, Promise': function (xPromise, yPromise) {
      return Promise.all([xPromise, yPromise]).then(([x, y]) =>
        mathjs.addScalar(x, y)
      )
    }
  })
})

mathjs.import([createPromise, createAddPromise])

async function getDelayedResult (result, delay = 2000) {
  const resolvedResult = await result // in case result is a Promise

  return new Promise(resolve => {
    setTimeout(() => {
      resolve(resolvedResult)
    }, delay)
  })
}

mathjs.import({
  getDelayedResult
})

function log (...args) {
  console.log([new Date().toISOString(), ...args].join(' '))
}

async function run () {
  log('Starting async calculations...')

  const result1 = await mathjs.add(getDelayedResult(2), 3)
  log('async result1:', result1)

  const result2 = await mathjs.evaluate('20 + getDelayedResult(30)')
  log('async result2:', result2)
  
  const result3 = mathjs.evaluate('5 + 7')
  log('sync result3:', result3)  
  
  const result4 = await mathjs.evaluate('1 + getDelayedResult(getDelayedResult(1))')
  log('async result4:', result4)
}

run()

Let's give it some more thought if this is something we want built-in in mathjs, and investigate how much work it would require.

[ericman314]

You added a conversion from 'any' to 'Promise'. Is there any way to make that conversion work the other direction? For instance, by changing typed-function so that when a promise is encountered in an argument it will wait until it resolves before evaluating the function? I can imagine that that would cause every other function that uses typed-function to become async, such as math.evaluate, so maybe it wouldn't work. But if it did, you could avoid changing every function. Or perhaps you could release a typed-function-async and allow custom builds of math.js to choose whether to include async support, so that it is not a breaking change.

[josdejong]

First benchmarks show there is no performance penalty 🎉 . So that means we will not need to create an option for this in mathjs, and simply can provide async support on all functions for free. Still a lot of things to work out, I took some shortcuts to be able to do these benchmarks and test integration in mathjs.

[vishal767]

it fails in ResultSet if the equation is in multiline. WHich dependency should I overwrite?

[josdejong]

@vishal767 support for async functions is not yet finished.

[vishal767]

I made it work ,by overwriting the BlockNode ,and some other modules..but haven't tested for all operators.. awesome package.. happy to contribute if you need any help in adding async

[josdejong]

I would love to see you you're implementing this. I made a proof of concept some time ago (josdejong/typed-function#34) but this still has a couple of tricky edge cases that needs to be worked out.

[f1nality]

Looking forward to have it implemented. In my case it is needed to run different API queries based on IF function result.

[josdejong]

Short status update:

The current proof of concept is in the following feature branches:
https://github.com/josdejong/mathjs/tree/feature/async_support
https://github.com/josdejong/typed-function/tree/async_support

Status:

• It's a working proof of concept
• In typed-function, what needs to be worked out properly is:
  • async support for the any type in combination with conversions
  • optimization: should remove signatures created from promises which are not reachable: where none of the params is a 'Promise'
• In the mathjs feature branch:
  • think through the API we want in the end. Think though if we want to set async support globally somehow or individually per function
  • enable async support for all functions, and add unit tests for every function

[bernarma]

Short status update:

The current proof of concept is in the following feature branches:
https://github.com/josdejong/mathjs/tree/feature/async_support
https://github.com/josdejong/typed-function/tree/async_support

Status:

• It's a working proof of concept

• In typed-function, what needs to be worked out properly is:

  • async support for the any type in combination with conversions
  • optimization: should remove signatures created from promises which are not reachable: where none of the params is a 'Promise'

• In the mathjs feature branch:

  • think through the API we want in the end. Think though if we want to set async support globally somehow or individually per function
  • enable async support for all functions, and add unit tests for every function

I know it is currently a Proof of Concept but today I came across this issue when starting to retrieve data from the data store (asynchronously) as part of a custom function and the variables passed in. I have checked out the async branches for both typed-function and mathjs... Got it working but I did come across the same issue when a "ResultSet" is returned...

{ mathjs: 'ResultSet', entries: [ 1, Promise { <pending> } ] }

I understand it is a POC at this stage but would you happen to know if this is a known issue and if so, are there any workarounds available?

When a single value is returned the expression is resolved successfully. As a side note, being able to define "async" functions is a must as the scopes are largely populated with data from the database based on expressions passed in - the size of the data in the database is not able to be pre-loaded due to the size.

[josdejong]

I understand it is a POC at this stage but would you happen to know if this is a known issue and if so, are there any workarounds available?

I think it simply wasn't worked out for ResultSet. As a workaound it may be possible to split a multiline expression yourself in separate lines, and evaluate them one by one (passing the same scope every time).

being able to define "async" functions is a must

Yes, the use case is definitely clear to me, I would love to have this available. There is simply a lack of time on my side to pick up all and everything, my spare time is limited.

[bernarma]

I have since been using the async branches for typed-function and mathjs incorporating your suggestions RE: the ResultSet. I have been quite successful in calling the library with large amounts of data from a DB using optimised routines such as "sum", "mean", etc. leveraging the database for the heavy lifting of the operation as opposed to pulling the data to the application and performing the operation.

One problem I have encountered (probably an edge case) is the extending of a built-in function to handle different parameters (i.e. custom types).

Example

1. I have extended the built-in "subtract" function with the following operation "LazyData, LazyData : _lazyDataHandler"
2. The extended and have specified createAsync: true in the factory function (see code below for the example)

`import { factory } from 'mathjs';

const name = 'subtract';
const dependencies = ['typed', 'LazyData'];

export const createSubtract = /* #PURE */ factory(
name,
dependencies,
({ typed, DataFunction, DataSeries }) => {
return typed(
name,
{
'LazyData, LazyData': _lazyDataHandler,
},
{ createAsync: true },
);

async function _lazyDataHandler(lh, rh) {
  // TODO: implement
  console.log({ lh, rh });
  throw new Error('Not Implemented');
}

},
);
`

Upon execution of the following expression, the below error is produced:

Expression> lazydata([[1],[1]]) - lazydata([[2],[6]]) * where lazydata is a function that returns a promise evaluating to the custom type "LazyData".

Error: Signature "Promise,Promise" is defined twice at validateUnique (/Users/berniemac/Develop/projekt1/typed-function/typed-function.js:1573:17) at extractSignatures (/Users/berniemac/Develop/projekt1/typed-function/typed-function.js:1588:15) at Function....Function (/Users/berniemac/Develop/projekt1/typed-function/typed-function.js:1621:50) at Function.preprocessRestParams (/Users/berniemac/Develop/projekt1/typed-function/typed-function.js:898:28) at Function.generic (/Users/berniemac/Develop/projekt1/typed-function/typed-function.js:1317:27) at Function.typed (/Users/berniemac/Develop/projekt1/typed-function/typed-function.js:1348:24) at lazyTyped (/Users/berniemac/Develop/projekt1/mathjs/lib/cjs/core/create.js:194:23) at resolver (/Users/berniemac/Develop/projekt1/mathjs/lib/cjs/core/function/import.js:303:16) at Object.get [as subtract] (/Users/berniemac/Develop/projekt1/mathjs/lib/cjs/utils/object.js:262:18) at /Users/berniemac/Develop/projekt1/mathjs/lib/cjs/core/function/import.js:323:29

I am assuming when extending the subtract function it rightly detects that a new 2 parameter function needs to be handled but is trying to create a Promise, Promise resolver when one already exists.

Before I get too deep into the mathjs and typed-function code, do you know if there is an easy fix or I am doing something wrong when creating the extension to the subtract function?

I was going to start modifying the mathjs code directly to build the function directly into the library as opposed to extending it but that became a larger exercise than I would have like to do (i.e. prefer extend over customise for future proofing). I also found that adding dependencies to the mathjs subtract function, it complained that those dependencies weren't found - it would appear that there are 2 sets of dependency injections, "dependenciesNumber.generated" and "dependenciesAny.generated" and I am unsure as to the architecture as to the differences between these dependencies.

[josdejong]

Thanks for your feedback. This can very well be caused by not all edge cases being handled properly yet. It can be hard to debug though, the code and generation quite complex.

[sachsgiri]

@josdejong Hi, is there any update on this? This library is awesome, but real use case for us is to use functions that fetch data from the external data stores.

[josdejong]

There was no progress on async support. Would be nice indeed to pick it up again. In the mean time typed-function@3 was released recently, which has quite some (mostly internal) changes, so I think the POC needs a re-do.

[jonathanstanley]

i wrote some code that helps with this (very different approach from above). it's rather hacky, but may help in certain circumstances.

let scope = {};

/** ensures all promised assignments in scope are resolved */
const resolveScope = async () => {
	let entries = Object.entries(scope);
	let promisedMap = entries.map(async ([k, v]) => [k, await v]);
	let resolvedEntries = await Promise.all(promisedMap);
	let resolvedScope = Object.fromEntries(resolvedEntries);
	Object.assign(scope, resolvedScope);
};

/** splits a formula to resolve each line one at a time (asynchronously) */
const asyncEvaluate = async (formula:string) => {
	let result: any;
	// some systems use `\r\n`, but mathjs expects only `\n`
	let lines = formula.replaceAll('\r', '').split('\n');
	for (const line of lines) {
		// assume the result is a promise
		let promise = evaluate(line, scope);
		// wait for the promised value (completes immediately if not a promise)
		result = await Promise.resolve(promise);
		// ensure any promised assignments are resolved
		await resolveScope();
	}
	return result;
};

[josdejong]

Thanks for sharing Jonathan, that makes sense 👍

[Venerons]

Any progress on this?

It would be great to add support for async functions for all the reasons stated on the comments above, and make something like this possibile:

import { create, all } from 'mathjs';

const math = create(all);

math.import({
  getValue: async function (timeseriesID, timestamp) {
    const value = await getValueFromDatabase(timeseriesID, timestamp);
    return value;
  }
});

const scope = new Map();
scope.set('t', Date.now());

const result = await math.asyncEvaluate('getValue("id1", t) + getValue("id2", t)');

[josdejong]

Yes it would be really useful. There is no progress though since this first POC: #2665 (comment)

[Venerons]

Is there any reason for this feature to be blocked? If it's just a matter of making it done, me and my team might help I suppose. Or is there any blocking issue that prevents this to be worked on?

[josdejong]

I made a POC back, ran into some tricky edge cases. Then it stopped due to a lack of time and priority.

[josdejong]

Help would definitely be welcome 😄

[Venerons]

Nice, I will see in the next days if we manage to get some work on this