Search API

[elshize]

Still work in progress

[elshize]

@JMMackenzie can you try checking out this branch and running tox? Idk why I'm getting Python.h: No such file or directory... Same happens for python setup.py install. Can you check if you're getting the same?

[elshize]

Actually nvm, I think I was missing the devel python package...

[elshize]

Pushed some code, but I can't get it to work yet. I get bad_alloc of all things... I tried debugging but something wierd was happening that I didn't understand. @JMMackenzie I'll try again in a spare moment, but if you have any ideas about what i messed up, let me know.

[JMMackenzie]

Pushed some code, but I can't get it to work yet. I get bad_alloc of all things... I tried debugging but something wierd was happening that I didn't understand. @JMMackenzie I'll try again in a spare moment, but if you have any ideas about what i messed up, let me know.

Thanks for this. Could you briefly show how you're building/testing? I just want to make sure we're doing the same thing. No rush, thanks!

[elshize]

I just run python setup.py install --user (you can use virtualenv or whatever you like), and then try running it. Here's an example script you might run:

import pypisa

results = pypisa.search(
    "data/robust04.block_simdbp.idx",
    "block_simdbp",
    ["103"],
    "maxscore",
    10,
    "data/robust04.bmw",
    False,
    "bm25",
    "data/robust04.termlex",
    "porter2"
)
print(results)

I used the Robus04-queries-only index from https://github.com/osirrc/ciff/, converted it with ciff tool, then compressed it and created lexicon with pypisa.

[JMMackenzie]

Just a quick note for debugging as discussed earlier. The query processor seems to be losing the scorer resulting in a segfault so we may need to alter the lambda captures in resolve_query_processor.hpp to avoid this.

[JMMackenzie]

Quick update. It seems that the problem could be caused by the ScorerParams getting destroyed (somehow) before it gets into the resolve_query_processor call. You can access the members of that struct inside the engine::processor(...) call correctly, and that function actually calls resolve_query_processor, so it is somehow getting lost through there.

[JMMackenzie]

And final update for now: passing ScorerParams through as a reference seems to work but then I get a bad_alloc 😂 So that's enough for today...