-
Notifications
You must be signed in to change notification settings - Fork 7
Description
Issue 1: Unit Conversion Error (Seconds → Minutes)
Task ID: 68993ef3cf3e953b8ab83fa3
Prompt: Get walking time from The Salty Donut to Vizcaya Museum (Miami, FL)
Tool Result (OSM Routing API)
{
"summary": {
"distance": 8496.7,
"duration": 687.3,
"mode": "foot"
}
}API Documentation: OSM routing API returns duration in seconds, distance in meters.
Claim
"The Walking Time from The Salty Donut to Vizcaya Museum is 687.3 minutes (11 hours and 27 minutes)."
The claim treats seconds as minutes. Walking 8.5km does not take 11 hours (admittedly it probably takes more than 11 minutes so I'm a bit confused here)
Issue 2: Temporal Impossibility (2022 Vehicle in 2017)
Task ID: 68940d8355faa0b720f0b482
Prompt: Find the customer with a Mustang who filed a complaint on August 7, 2017, and retrieve the vehicle year.
Claim
"The mustang from the august 7 2017 complaint vehicle's year is 2022."
A 2022 model year vehicle cannot exist in 2017.
Issue 3: Weather API Does Not Support Historical Dates
Task ID: 68940d8355faa0b720f0b4ba
Prompt: What was the moon phase the day Shirley Jackson died (August 8, 1965) but in the book's publication year (1959)?
Tool Call
{
"q": "United States of America",
"dt": "1959-08-08"
}Tool Result
{
"location": {
"name": "Washington",
"localtime": "2025-08-10 16:32"
},
"astronomy": {
"astro": {
"moon_phase": "Waxing Crescent"
}
}
}Claim
"The moon phase on August 8, 1959 was waxing crescent."
The tool was queried for 1959-08-08 but returned data for 2025-08-10. Another task (689e0b1d9c8e2ac413c1f249) queried dt: "2010-11-20" and also received 2025 data (localtime: "2025-08-15"). It seems that the weather API cannot return historical astronomical data?