Issue finding AWS credentials #161
Description
Setting up cookbook on a new laptop, and have been encountering issues with AWS credentials when launching evals. I'm using the same launch commands that are working on my other laptop, so as far as I know the necessary secrets are present and correct in the workspace.
Did some troubleshooting with @undfined --
- ~/.aws/config and ~/.aws/credentials are present and as far as I can tell identical between laptops
- beaker user is the same on both laptops
- command doesn't appear to have any issues or typos (pasted below).
It looks like I was able to solve the issue for now by setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY directly in my local environment. But as far as I can tell I do not have these set in my environment on the other laptop, so still unclear what was causing the failure.
@undfined suggested cutting issue.
Command
Command:
/Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/bin/python oe_eval/launch.py --beaker-workspace 'ai2/oe-data' --beaker-budget 'ai2/oe-base' --beaker-priority high --cluster 'ai2/saturn-cirrascale,ai2/ceres-cirrascale,ai2/jupiter-cirrascale-2' --gpus 1 --datalake-tags 'dashboard=allysone-microanneals,checkpoint=webv18-reddit-lowthresh-nonMC-5B-olmo3-4T-microanneal-2bc12e7f_step2385-hf' --push-datalake --model webv18-reddit-lowthresh-nonMC-5B-olmo3-4T-microanneal-2bc12e7f_step2385-hf --model-args 'model_path=weka://oe-training-default/ai2-llm/checkpoints/allysone/webv18-reddit-lowthresh-nonMC-5B-olmo3-4T-microanneal-2bc12e7f/step2385-hf,add_bos_token=false,gpu_memory_utilization=0.8,chat_template=basic_answer,trust_remote_code=true,max_length=8192' --model-type vllm --task agi_eval_aqua-rat:0shot_cot::olmo3:thinker agi_eval_gaokao-english:0shot_cot::olmo3:thinker agi_eval_logiqa-en:0shot_cot::olmo3:thinker agi_eval_lsat-ar:0shot_cot::olmo3:thinker agi_eval_lsat-lr:0shot_cot::olmo3:thinker agi_eval_lsat-rc:0shot_cot::olmo3:thinker agi_eval_sat-en-without-passage:0shot_cot::olmo3:thinker agi_eval_sat-en:0shot_cot::olmo3:thinker --remote-output-dir 's3://ai2-llm/evaluation/allysone-microanneals/webv18-reddit-lowthresh-nonMC-5B-olmo3-4T-microanneal-2bc12e7f_step2385-hf/agi_eval_aqua-rat_0shot_cot_olmo3_thinker-a-1e4ce2' --beaker-image oe-eval-beaker/oe_eval_olmo2_retrofit_auto --use-gantry --gantry-args '{"env": "VLLM_USE_V1=1", "env-secret": "OPENAI_API_KEY=openai_api_key"}' --task-args '{"chat_overrides": {"generation_kwargs": {"stop_sequences": ["Problem:", "Answer:", "Question:", "</s>", "<|eot_id|>"]}}}'
From:
/private/var/folders/1f/6p0v_gw12fs2842l79t46nbw0000gp/T/tmprht_7n35
Traceback.
/Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/bin/gantry:8 in <module> │
│ │
│ 5 from gantry.__main__ import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/click │
│ /core.py:1442 in __call__ │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/click │
│ /core.py:1363 in main │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/click │
│ /core.py:1830 in invoke │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/click │
│ /core.py:1226 in invoke │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/click │
│ /core.py:794 in invoke │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/gantr │
│ y/commands/run.py:558 in run │
│ │
│ 555 │ name_prefix = name │
│ 556 │ while True: │
│ 557 │ │ try: │
│ ❱ 558 │ │ │ experiment = beaker.experiment.create(name, spec) │
│ 559 │ │ │ break │
│ 560 │ │ except ExperimentConflict: │
│ 561 │ │ │ name = ( │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/beake │
│ r/services/experiment.py:216 in create │
│ │
│ 213 │ │ spec.validate() │
│ 214 │ │ json_spec = spec.to_json() │
│ 215 │ │ workspace = self.resolve_workspace(workspace) │
│ ❱ 216 │ │ self._validate_spec(spec, workspace) │
│ 217 │ │ experiment_data = self.request( │
│ 218 │ │ │ f"workspaces/{workspace.id}/experiments", │
│ 219 │ │ │ method="POST", │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/beake │
│ r/services/experiment.py:971 in _validate_spec │
│ │
│ 968 │ │ │ # Make sure secrets in env variables exist. │
│ 969 │ │ │ for env_var in task.env_vars or []: │
│ 970 │ │ │ │ if env_var.secret is not None: │
│ ❱ 971 │ │ │ │ │ self.beaker.secret.get(env_var.secret, workspace=workspace) │
│ 972 │ │ │ # Make sure cluster exists. │
│ 973 │ │ │ if task.context.cluster: │
│ 974 │ │ │ │ self.beaker.cluster.get(task.context.cluster) │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/beake │
│ r/services/secret.py:30 in get │
│ │
│ 27 │ │ """ │
│ 28 │ │ workspace = self.resolve_workspace(workspace, read_only_ok=True) │
│ 29 │ │ return Secret.from_json( │
│ ❱ 30 │ │ │ self.request( │
│ 31 │ │ │ │ f"workspaces/{workspace.id}/secrets/{self.url_quote(secret)}", │
│ 32 │ │ │ │ method="GET", │
│ 33 │ │ │ │ exceptions_for_status={404: SecretNotFound(secret)}, │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/beake │
│ r/services/service_client.py:148 in request │
│ │
│ 145 │ │ │ make_request = retriable()(make_request) │
│ 146 │ │ │
│ 147 │ │ if self.beaker._session is not None: │
│ ❱ 148 │ │ │ return make_request(self.beaker._session) │
│ 149 │ │ else: │
│ 150 │ │ │ with self.beaker._make_session() as session: │
│ 151 │ │ │ │ return make_request(session) │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/beake │
│ r/util.py:195 in retriable_method │
│ │
│ 192 │ │ │ retries = 0 │
│ 193 │ │ │ while True: │
│ 194 │ │ │ │ try: │
│ ❱ 195 │ │ │ │ │ return func(*args, **kwargs) │
│ 196 │ │ │ │ except recoverable_errors as err: │
│ 197 │ │ │ │ │ if retries < Beaker.MAX_RETRIES: │
│ 198 │ │ │ │ │ │ if on_failure is not None: │
│ │
│ /Users/allysone/Desktop/code/olmo-cookbook/.venv/oe-eval-venv/lib/python3.11/site-packages/beake │
│ r/services/service_client.py:127 in make_request │
│ │
│ 124 │ │ │ │ │ status_code = 409 │
│ 125 │ │ │ │ │
│ 126 │ │ │ │ if exceptions_for_status is not None and status_code in exceptions_for_s │
│ ❱ 127 │ │ │ │ │ raise exceptions_for_status[status_code] │
│ 128 │ │ │ │ │
│ 129 │ │ │ │ if msg is not None and status_code is not None and 400 <= status_code < │
│ 130 │ │ │ │ │ # Raise a BeakerError if we're misusing the API (4xx error code). │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
SecretNotFound: aws_access_key_id_read_only
Metadata
Metadata
Assignees
Labels
No labels