Skip to content

Failed to open: ../registry/data/social_iqa/few_shot.jsonl with custom registry #1394

@LoryPack

Description

@LoryPack

Describe the bug

I have created a new eval within a custom registry and installed evals from pypi (using a recent checkout from github). Calling oaievals works fine if I do not use few_show samples, but it does not if you include few shot examples.

As can be seen in the error thread below, the issue is caused by calling self._prefix_registry_path(self.few_shot_jsonl) (see _prefix_registry_path) which transforms the path string into a Path object. Then, open_by_file_pattern (here) still treats it as a string, unaware of the fact that it is actually a Path

To Reproduce

  1. Create a new eval with few-shot samples.
  2. Call oaieval, for instance:
oaieval text-ada-001 social_iqa_few_shot --registry_path=../registry
  1. You'll get an error thread as follows:
[2023-11-03 14:40:52,931] [registry.py:254] Loading registry from /home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/registry/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from /home/lorenzo/.evals/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from ../registry/evals
[2023-11-03 14:40:55,007] [oaieval.py:189] �[1;35mRun started: 2311031440552KUYNQGH�[0m
[2023-11-03 14:40:55,010] [data.py:90] Fetching ../registry/data/social_iqa/few_shot.jsonl
Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 54, in open_by_file_pattern
    if filename.endswith(".gz"):
AttributeError: 'PosixPath' object has no attribute 'endswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/bin/oaieval", line 8, in <module>
    sys.exit(main())
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 274, in main
    run(args)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 215, in run
    eval: Eval = eval_class(
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/elsuite/basic/match.py", line 28, in __init__
    self.few_shot = evals.get_jsonl(self._prefix_registry_path(self.few_shot_jsonl))
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 129, in get_jsonl
    return _get_jsonl_file(path)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 91, in _get_jsonl_file
    with open_by_file_pattern(path, mode="r") as f:
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 75, in open_by_file_pattern
    raise RuntimeError(f"Failed to open: {filename}") from e
RuntimeError: Failed to open: ../registry/data/social_iqa/few_shot.jsonl

Code snippets

No response

OS

Ubuntu 20.04

Python version

python 3.9

Library version

git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions