Add 2d_movement eval

elh · elh · commit 7830edaf8769 · 2023-03-27T00:45:14.000-07:00
Eval if GPT can keep track of cardinal directions and rotations
while walking on a 2D grid
diff --git a/evals/registry/data/2d_movement/samples.jsonl b/evals/registry/data/2d_movement/samples.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a599e52030f1dd1bf110785c54ed27a297546412974b8dbe1e88a50250d007e7
+size 60724
diff --git a/evals/registry/evals/2d_movement.yaml b/evals/registry/evals/2d_movement.yaml
@@ -0,0 +1,8 @@
+2d_movement:
+  id: 2d_movement.dev.v0
+  metrics: [accuracy]
+
+2d_movement.dev.v0:
+  class: evals.elsuite.basic.match:Match
+  args:
+    samples_jsonl: 2d_movement/samples.jsonl

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:a599e52030f1dd1bf110785c54ed27a297546412974b8dbe1e88a50250d007e7`
	`3`	`+size 60724`