[FIX] Executor Queue for Agentic extraction#1893
[FIX] Executor Queue for Agentic extraction#1893chandrasekharan-zipstack merged 127 commits intomainfrom
Conversation
Conflicts resolved: - docker-compose.yaml: Use main's dedicated dashboard_metric_events queue for worker-metrics - PromptCard.jsx: Keep tool_id matching condition from our async socket feature - PromptRun.jsx: Merge useEffect import from main with our branch - ToolIde.jsx: Keep fire-and-forget socket approach (spinner waits for socket event) - SocketMessages.js: Keep both session-store and socket-custom-tool imports + updateCusToolMessages dep - SocketContext.js: Keep simpler path-based socket connection approach - usePromptRun.js: Keep Celery fire-and-forget with socket delivery over polling - setupProxy.js: Accept main's deletion (migrated to Vite)
for more information, see https://pre-commit.ci
… into feat/execution-backend
for more information, see https://pre-commit.ci
… into feat/execution-backend
WalkthroughRemoved a prompt output option, adjusted Celery retry settings, added an agentic executor queue and surfaced it in configs/health checks, refined NA handling and email extraction in the legacy executor, tightened error handling/logging in destination and file-structure tasks, and updated tests to match NA behavior. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| workers/executor/executors/legacy_executor.py | Two targeted fixes: (1) _sanitize_null_values no longer converts top-level "NA" strings to None; (2) email type conversion short-circuits on "NA" instead of making an LLM call. Minor inconsistency: DATE type still converts "NA" → None via _convert_scalar_answer while EMAIL now preserves it. |
| workers/executor/worker.py | Health check now reads queues dynamically from CELERY_QUEUES_EXECUTOR env var via split(","); minor risk of whitespace in token names if env var has spaces around commas. |
| workers/shared/enums/worker_enums_base.py | Adds EXECUTOR_AGENTIC = "celery_executor_agentic" to QueueName enum cleanly — no duplicate names, no import issues. |
| workers/shared/infrastructure/config/registry.py | Adds EXECUTOR_AGENTIC as an additional queue for the executor worker type; task routing is unchanged (agentic tasks are expected to be dispatched with an explicit queue, not via route config). |
| docker/docker-compose.yaml | Default CELERY_QUEUES_EXECUTOR now includes celery_executor_agentic alongside celery_executor_legacy. |
| workers/tests/test_answer_prompt.py | Tests updated to assert "NA" strings are preserved at the top level instead of being converted to None; accurately reflects the new _sanitize_null_values behavior. |
| backend/prompt_studio/prompt_studio_core_v2/static/select_choices.json | Removes deprecated line_item output type from select choices; straightforward cleanup. |
| backend/workflow_manager/workflow_v2/workflow_helper.py | Removes autoretry_for and related retry options from the execute_bin Celery task decorator, consistent with max_retries=0 intent. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Agentic task dispatched\nqueue: celery_executor_agentic] --> B{Executor Worker\nlistens on both queues}
B --> C[celery_executor_legacy\nprimary queue]
B --> D[celery_executor_agentic\nnew additional queue]
C --> E[LegacyExecutor._handle_answer_prompt]
D --> E
E --> F{output_type?}
F -->|EMAIL| G{answer == 'NA'?}
G -->|yes| H[preserve answer as 'NA' string]
G -->|no| I[run_completion LLM call]
F -->|DATE| J[_convert_scalar_answer\n'NA' → None unchanged]
F -->|TEXT / other| K[store raw answer]
K --> L[_sanitize_null_values]
H --> L
I --> L
J --> L
L --> M{value is list?}
M -->|yes| N[convert 'NA' items → None]
M -->|no - dict| O[_sanitize_dict_values]
M -->|no - string top-level| P[preserve as-is NEW BEHAVIOR]
Prompt To Fix All With AI
This is a comment left during a code review.
Path: workers/executor/worker.py
Line: 46-48
Comment:
**Potential whitespace in queue names from env var**
`split(",")` on a value like `"celery_executor_legacy, celery_executor_agentic"` (with a space after the comma) will produce `[" celery_executor_agentic"]` with a leading space. While this only affects the health check response here, it's good practice to strip individual tokens so the output is reliably clean:
```suggestion
"queues": [
q.strip()
for q in os.environ.get(
"CELERY_QUEUES_EXECUTOR", "celery_executor_legacy"
).split(",")
if q.strip()
],
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: workers/executor/executors/legacy_executor.py
Line: 1699-1711
Comment:
**Inconsistent NA handling between EMAIL and DATE types**
After this change, the two types diverge in how they handle an `"NA"` answer:
- **EMAIL** (updated here): `"NA"` → `"NA"` string preserved
- **DATE** (unchanged, line 1723): still calls `_convert_scalar_answer`, which returns `None` for `"NA"` answers
`_convert_scalar_answer` (lines 1268–1269):
```python
if answer.lower() == "na":
return None
```
If the intent behind the `_sanitize_null_values` fix is that `"NA"` values in structured output should be preserved as strings (the PR description says "Top-level 'NA' values were being silently converted to None"), then DATE has the same problem through a different code path. Was the inconsistency intentional (i.e., `None` is semantically correct for missing dates but not for missing emails)?
How can I resolve this? If you propose a fix, please make it concise.Reviews (6): Last reviewed commit: "Revert ETL destination pipeline changes ..." | Re-trigger Greptile
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
for more information, see https://pre-commit.ci
- Widen except clause in structure_tool_task to catch FileOperationError and log write paths for diagnostics - Add diagnostic logging at all silent return-None points in destination connector so missing INFILE/METADATA paths are visible in logs - Raise RuntimeError instead of silently skipping when no tool execution result is available for DB/FS destinations, preventing false success - Remove dead retry config from execute_bin task (max_retries=0) - Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
chandrasekharan-zipstack
left a comment
There was a problem hiding this comment.
Change the PR title to follow the format
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
workers/executor/executors/legacy_executor.py (1)
1056-1070: Docstring is now inconsistent with actual behavior.The docstring on line 1060 states "Replace 'NA' strings with None in structured output" but the implementation now only sanitizes "NA" within nested lists and dicts, preserving top-level "NA" strings. This could mislead future maintainers.
📝 Proposed docstring update
`@staticmethod` def _sanitize_null_values( structured_output: dict[str, Any], ) -> dict[str, Any]: - """Replace 'NA' strings with None in structured output.""" + """Replace 'NA' strings with None in nested structures only. + + Top-level 'NA' values are preserved. Only 'NA' strings inside + lists or nested dicts are converted to None. + """ for k, v in structured_output.items():🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@workers/executor/executors/legacy_executor.py` around lines 1056 - 1070, Docstring for LegacyExecutor._sanitize_null_values is inaccurate: it claims to "Replace 'NA' strings with None in structured output" but the method only replaces "NA" within nested lists/dicts (using LegacyExecutor._sanitize_dict_values) and leaves top-level "NA" strings untouched; update the docstring to accurately describe the behavior (e.g., "Sanitize nested list/dict values by replacing 'NA' strings with None; top-level string values are not modified") and reference the helper LegacyExecutor._sanitize_dict_values in the docstring so maintainers know nested sanitization is delegated.workers/executor/worker.py (1)
47-49: Prefer the resolved worker config as the health-check source of truth.If this is meant to report the queues this process actually started with,
config.queue_config.to_queue_list()is safer than re-splittingCELERY_QUEUES_EXECUTORand carrying a separate legacy-only fallback.♻️ Proposed refactor
- "queues": os.environ.get( - "CELERY_QUEUES_EXECUTOR", "celery_executor_legacy" - ).split(","), + "queues": ( + config.queue_config.to_queue_list() + if config + else WorkerRegistry.get_queue_config( + WorkerType.EXECUTOR + ).to_queue_list() + ),🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@workers/executor/worker.py` around lines 47 - 49, The health-check should report the actual queues the process started with rather than re-splitting the CELERY_QUEUES_EXECUTOR env var: replace the current use of os.environ.get(...).split(",") in the "queues" field with the resolved worker config call config.queue_config.to_queue_list() (or equivalent accessor on your QueueConfig) so the health-check uses the authoritative, possibly-processed queue list the worker actually started with; keep a minimal fallback only if config.queue_config may be unset (use an empty list or existing legacy default).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@backend/workflow_manager/workflow_v2/workflow_helper.py`:
- Around line 598-603: The `@celery_app.task` decorator call for async_execute_bin
is broken because the closing parenthesis comes before retry_jitter=True; fix it
by moving retry_jitter=True inside the decorator argument list and removing the
stray closing paren so the decorator reads something like
`@celery_app.task`(name="async_execute_bin", max_retries=0, retry_jitter=True)
directly above the async_execute_bin function; ensure indentation is corrected
so the decorator encloses all arguments and the function registers properly.
In `@workers/executor/executors/legacy_executor.py`:
- Around line 1698-1711: Add a clarifying inline comment near the EMAIL branch
in legacy_executor.py explaining that for PSKeys.EMAIL the literal string "NA"
is intentionally preserved (checked via answer.lower() == "na" and assigned
directly to structured_output[prompt_name]) whereas other types (NUMBER,
BOOLEAN, DATE) map "NA" to None; reference the output_type == PSKeys.EMAIL
branch, the answer.lower() == "na" check, and the use of
answer_prompt_svc.run_completion so future maintainers understand this
deliberate discrepancy and its impact on downstream consumers.
In `@workers/shared/enums/worker_enums_base.py`:
- Around line 27-31: Remove the duplicate IDE_CALLBACK member from the
WorkerType enum so it's defined only once, remove the extra repeated
`@classmethod` decorator on the from_directory_name method so it has a single
`@classmethod`, and deduplicate entries in the port_mapping dict so each
WorkerType (notably WorkerType.EXECUTOR and WorkerType.IDE_CALLBACK) appears
only once with the correct port value; update port_mapping to keep the intended
port for each unique key and remove the redundant mapping(s).
---
Nitpick comments:
In `@workers/executor/executors/legacy_executor.py`:
- Around line 1056-1070: Docstring for LegacyExecutor._sanitize_null_values is
inaccurate: it claims to "Replace 'NA' strings with None in structured output"
but the method only replaces "NA" within nested lists/dicts (using
LegacyExecutor._sanitize_dict_values) and leaves top-level "NA" strings
untouched; update the docstring to accurately describe the behavior (e.g.,
"Sanitize nested list/dict values by replacing 'NA' strings with None; top-level
string values are not modified") and reference the helper
LegacyExecutor._sanitize_dict_values in the docstring so maintainers know nested
sanitization is delegated.
In `@workers/executor/worker.py`:
- Around line 47-49: The health-check should report the actual queues the
process started with rather than re-splitting the CELERY_QUEUES_EXECUTOR env
var: replace the current use of os.environ.get(...).split(",") in the "queues"
field with the resolved worker config call config.queue_config.to_queue_list()
(or equivalent accessor on your QueueConfig) so the health-check uses the
authoritative, possibly-processed queue list the worker actually started with;
keep a minimal fallback only if config.queue_config may be unset (use an empty
list or existing legacy default).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 335d2886-4446-4367-9349-e179ae7cb57e
📒 Files selected for processing (8)
backend/prompt_studio/prompt_studio_core_v2/static/select_choices.jsonbackend/workflow_manager/workflow_v2/workflow_helper.pydocker/docker-compose.yamlworkers/executor/executors/legacy_executor.pyworkers/executor/worker.pyworkers/shared/enums/worker_enums_base.pyworkers/shared/infrastructure/config/registry.pyworkers/tests/test_answer_prompt.py
💤 Files with no reviewable changes (1)
- backend/prompt_studio/prompt_studio_core_v2/static/select_choices.json
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@workers/shared/workflow/destination_connector.py`:
- Around line 661-672: The error message for the missing tool execution result
is misleading because the code raises a RuntimeError (failing the operation)
rather than merely "skipping" the filesystem copy; update the message used in
logger.error, log_file_error (with exec_ctx.workflow_log and
exec_ctx.file_execution_id), and the RuntimeError to reflect failure (e.g.,
"filesystem copy failed" or "cannot perform filesystem copy") and keep the
existing context from result.tool_execution_result and file_ctx.execution_error
when constructing the message.
- Around line 711-721: The error message says "database insertion skipped" but
the code still raises RuntimeError; update the behavior in the block that builds
error_msg (references: file_ctx.file_name, logger, log_file_error,
exec_ctx.file_execution_id) so it matches the message: do not raise RuntimeError
— instead log the error and return/exit the enclosing function early (or
explicitly return None) to skip database insertion, or if an exception is
desired change the message to reflect a raised error; prefer removing the raise
RuntimeError(error_msg) and returning to mirror the filesystem handler's
"skipped" behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 49bfbde0-46e7-44f6-be53-72fb0dab43bb
📒 Files selected for processing (5)
backend/workflow_manager/workflow_v2/workflow_helper.pydocker/docker-compose.yamlworkers/file_processing/structure_tool_task.pyworkers/shared/enums/worker_enums_base.pyworkers/shared/workflow/destination_connector.py
💤 Files with no reviewable changes (1)
- backend/workflow_manager/workflow_v2/workflow_helper.py
Reverts diagnostic logging and error-raising changes in structure_tool_task.py and destination_connector.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
* [FIX] Executor Queue for Agentic extraction (#1893) * Execution backend - revamp * async flow * Streaming progress to FE * Removing multi hop in Prompt studio ide and structure tool * UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item * Added executors for agentic prompt studio * Added executors for agentic prompt studio * Removed redundant envs * Removed redundant envs * Removed redundant envs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed redundant envs * adding worker for callbacks * adding worker for callbacks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pluggable apps and plugins to fit the new async prompt execution architecture * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pluggable apps and plugins to fit the new async prompt execution architecture * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pluggable apps and plugins to fit the new async prompt execution architecture * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * fix: write output files in agentic extraction pipeline Agentic extraction returned early without writing INFILE (JSON) or METADATA.json, causing destination connectors to read the original PDF and fail with "Expected tool output type: TXT, got: application/pdf". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850) * UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants in all affected test files to avoid world-writable directory vulnerabilities. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update docs * UN-3266 fix: remove dead code with undefined names in fetch_response Remove unreachable code block after the async callback return in fetch_response that still referenced output_count_before and response from the old synchronous implementation, causing ruff F821 errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Un 3266 fix security hotspot tmp paths (#1851) * UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants in all affected test files to avoid world-writable directory vulnerabilities. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve ruff linting failures across multiple files - B026: pass url positionally in worker_celery.py to avoid star-arg after keyword - N803: rename MockAsyncResult to mock_async_result in test_tasks.py - E501/I001: fix long line and import sort in llm_whisperer helper - ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers - F841: remove unused workflow_id and result assignments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849 - S2259: guard against None after _discover_plugins() in loader.py to satisfy static analysis on the dict[str,type]|None field type - S1244: replace float equality checks with pytest.approx() in test_answer_prompt.py and test_phase2h.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve SonarCloud code smells in PR #1849 - S5799: Merge all implicit string concatenations in log messages (legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py, registry.py, variable_replacement.py, structure_tool_task.py) - S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in dispatcher.py - S1871: Merge identical elif/else branches in tasks.py and test_sanity_phase6j.py - S1186: Add comment to empty stub method in test_sanity_phase6a.py - S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j and test_phase5d.py - S117: Rename PascalCase local variables to snake_case in test_sanity_phase3/5/6i.py - S5655: Broaden tool type annotation to StreamMixin in IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config - docker:S7031: Merge consecutive RUN instructions in worker-unified.Dockerfile - javascript:S1128: Remove unused pollForCompletion import in usePromptRun.js Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: wrap long log message in dispatcher.py to fix E501 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve remaining SonarCloud S117 naming violations Rename PascalCase local variables to snake_case to comply with S117: - legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results (AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc, VariableReplacementService→variable_replacement_svc, LLM→llm_cls, EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and update all downstream usages including _apply_type_conversion and _handle_summarize - test_phase1_log_streaming.py: rename Mock* local variables to mock_* snake_case equivalents - test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls and MockShim→mock_shim_cls across all 10 test methods - test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test; fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls in _mock_prompt_deps helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849 - test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase local variables in _mock_prompt_deps/_mock_deps to snake_case (RetrievalService→retrieval_svc, VariableReplacementService→ variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls, AnswerPromptService→answer_prompt_svc_cls) — fixes S117 - test_sanity_phase3.py: remove unused local variable "result" — fixes S1481 - structure_tool_task.py: remove redundant json.JSONDecodeError from except clause (subclass of ValueError) — fixes S5713 - shared/workflow/execution/service.py: replace generic Exception with RuntimeError for structure tool failure — fixes S112 - run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and replace 10 literal "executor" occurrences — fixes S1192 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations - Reduce cognitive complexity in answer_prompt.py: - Extract _build_grammar_notes, _run_webhook_postprocess helpers - _is_safe_public_url: extracted _resolve_host_addresses helper - handle_json: early-return pattern eliminates nesting - construct_prompt: delegates grammar loop to _build_grammar_notes - Reduce cognitive complexity in legacy_executor.py: - Extract _execute_single_prompt, _run_table_extraction helpers - Extract _run_challenge_if_enabled, _run_evaluation_if_enabled - Extract _inject_table_settings, _finalize_pipeline_result - Extract _convert_number_answer, _convert_scalar_answer - Extract _sanitize_dict_values helper - _handle_answer_prompt CC reduced from 50 to ~7 - Reduce CC in structure_tool_task.py: guard-clause refactor - Reduce CC in backend: dto.py, deployment_helper.py, api_deployment_views.py, prompt_studio_helper.py - Fix S117: rename PascalCase local vars in test_answer_prompt.py - Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh - Fix S1172: remove unused params from structure_tool_task.py - Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py - Fix S112/S5727 in test_execution.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: rename UsageHelper params to lowercase (N803) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192 - Add @staticmethod to _sanitize_null_values (fixes S2325 missing self) - Reduce _execute_single_prompt params from 25 to 11 (S107) by grouping services as deps tuple and extracting exec params from context.executor_params - Add NOSONAR suppression for raise exc in test helper (S112) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: remove unused locals in _handle_answer_prompt (F841) execution_id, file_hash, log_events_id, custom_data are now extracted inside _execute_single_prompt from context.executor_params. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve Biome linting errors in frontend source files Auto-fixed 48 lint errors across 56 files: import ordering, block statements, unused variable prefixing, and formatting issues. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: replace dynamic import of SharePermission with static import in Workflows Resolves vite build warning about SharePermission.jsx being both dynamically and statically imported across the codebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve SonarCloud warnings in frontend components - Remove unnecessary try-catch around PostHog event calls - Flip negated condition in PromptOutput.handleTable for clarity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address PR #1849 review comments: fix null guards, dead code, and test drift - Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid) - URL-encode DB_USER in worker_celery.py result backend connection string - Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app - Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist) - Move profile_manager/default_profile null checks before first dereference - Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py - Handle ProfileManager.DoesNotExist as warning, not hard failure - Wrap PostHog analytics in try/catch so failures don't block prompt execution - Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status) - Reset formData when metadata is missing in ConfigureDs.jsx - Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only) - Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix missing llm_usage_reason for summarize LLM usage tracking Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so summarization costs appear under summarize_llm in API response metadata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor - Route _handle_structure_pipeline to _handle_single_pass_extraction when is_single_pass=True (was always calling _handle_answer_prompt) - Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry, falling back to _handle_answer_prompt if plugin not installed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fixing API depployment response mismatches * Fix single-pass extraction showing only one prompt result in real-time - Fix accumulation bug in usePromptOutput updateCoverage() where each loop iteration spread the original promptOutputs instead of the accumulating updatedPromptOutputs, causing only the last prompt to render - Improve index success toast to show document name - Strip adapter names from index key configs for consistent hashing - Update sdk1 uv.lock Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move summarize from sync Django plugin to executor worker for IDE index When "Summarize" is enabled and a user indexes a new document in Prompt Studio, the backend returned a 500 because the sync Django plugin tried to read the extracted .txt file before extraction happened in the worker. Fix: defer summarization to the executor worker's _handle_ide_index (extract → summarize → index), build summarize_params in the payload, and track the summarize index in the ide_index_complete callback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address PR #1849 review comments: null guards, thread safety - Fix null guard ordering in build_index_payload: add DefaultProfileError check before calling validators on default_profile - Fix null guard ordering in _fetch_single_pass_response: move check before .llm.id access and validators (was dead code after dereference) - Add threading.Lock to worker_celery singleton to prevent race under concurrent gunicorn threads Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add documentation to ExecutionResponse DTO describing result structure Documents the ExecutionResponse dataclass fields, especially the result attribute's per-file dict structure (output, metadata, metrics, error keys) as requested in PR review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup - Strip result payload from task_status to prevent IDOR data leak - Move null guard before validators in build_fetch_response_payload - Roll back indexing flag on broker failure in index_document - Use explicit IDs for spinner clearing instead of ORM serialization format - Catch broad exceptions in summarize tracking to prevent false failures - Guard prompt_id lookup in fetch_response with 400/404 responses Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix CI, tests, and add async prompt studio improvements - Fix biome import ordering in frontend test files - Fix 22 stale backend test assertions in test_tasks.py (patch targets, return values, view dispatch pattern, remove TestCeleryConfig) - Add ide_prompt_complete callback tests - Add frontend vitest config and regression tests for null guards, stale closures, and single-pass loading guard - Add LLMCompat llama-index compatibility wrapper in SDK1 - Add litellm cohere embed timeout monkey-patch (v1.82.3) - Improve S3 filesystem helper (region_name, empty credential handling) - Add RetrieverLLM and lazy LLM creation for retrievers - Improve worker cleanup: api_client.close() in finally blocks, early return on setup failure in API deployment tasks - Add workers conftest.py for test environment setup Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix pre-existing biome CI errors: import ordering and formatting Auto-fix 18 pre-existing organizeImports and formatting errors across 17 frontend files that were blocking biome ci on the entire codebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix ruff F821: add missing transaction import in prompt_studio_helper Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add input validation guards to bulk_fetch_response endpoint Validate prompt_ids (non-empty), document_id (required), and handle DoesNotExist for both prompts and document to return proper 400/404 instead of dispatching no-op tasks or raising unhandled 500 errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * IDE Call backs * Sonar issues fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update bun.lock to match package.json dependency ranges Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move ExecutionContext import into TYPE_CHECKING block Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix SonarQube issues: duplication, naming, nesting, unused var - Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to deduplicate 5 identical JSON parsing blocks in internal_views.py - Rename `User` local variable to `user_model` (naming convention) - Merge _emit_result/_emit_error into unified _emit_event() in ide_callback tasks to reduce code duplication - Extract _get_task_error() helper to deduplicate AsyncResult error retrieval in ide_index_error and ide_prompt_error - Remove unused `mock_ar_cls` variable in test_ide_callback.py - Add security note documenting why @csrf_exempt is safe on internal endpoints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Replace worker-ide-callback Dockerfile with worker-unified The IDE callback worker should use the unified worker image (worker-unified.Dockerfile) consistent with all other v2 workers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add celery_executor_agentic queue to executor worker The executor worker only consumed from celery_executor_legacy, but the agentic prompt studio dispatches tasks to celery_executor_agentic. This caused agentic operations to sit in RabbitMQ with no consumer, resulting in timeouts and stuck-in-progress states. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * FIxing email enforce type * Removing line-item from select choices * Update workers/shared/enums/worker_enums_base.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> * Update backend/workflow_manager/workflow_v2/workflow_helper.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix false success logs and silent failures in ETL destination pipelines - Widen except clause in structure_tool_task to catch FileOperationError and log write paths for diagnostics - Add diagnostic logging at all silent return-None points in destination connector so missing INFILE/METADATA paths are visible in logs - Raise RuntimeError instead of silently skipping when no tool execution result is available for DB/FS destinations, preventing false success - Remove dead retry config from execute_bin task (max_retries=0) - Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert ETL destination pipeline changes — deferring to next cut Reverts diagnostic logging and error-raising changes in structure_tool_task.py and destination_connector.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com> Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * [MISC] Fix blank LLM profile edit form and settings menu bold style (#1896) - Replace form.resetFields() with form.setFieldsValue() to fix blank edit form caused by Ant Design's initialValues being a one-time snapshot - Remove Strict Mode-incompatible cleanup that cleared editLlmProfileId - Clear editLlmProfileId in parent when adding new profile - Remove unconditional bold on first settings popover menu item Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [FEAT] Port LINE_ITEM extraction to pluggable executor architecture LINE_ITEM enforce_type prompts previously raised an unconditional error in the new workers executor. Wire the structure-tool path to delegate to a `line_item` executor plugin (mirroring how TABLE delegates to the `table_extractor` plugin), so API deployments / ETL pipelines can run LINE_ITEM extraction once the cloud `line_item_extractor` plugin is installed. - legacy_executor.py: replace the LINE_ITEM error site in `_execute_single_prompt` with a delegation call and add a new `_run_line_item_extraction()` method structurally identical to `_run_table_extraction()`. - test_line_item_extraction.py: 5 new tests covering plugin missing, success, sub-context construction, failure path, and end-to-end Celery eager-mode delegation. - test_sanity_phase6d.py: update the existing LINE_ITEM guard test to match the new "install the line_item_extractor plugin" hint and patch `ExecutorRegistry.get` to simulate the missing plugin. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com> Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
* [FIX] Executor Queue for Agentic extraction (#1893) * Execution backend - revamp * async flow * Streaming progress to FE * Removing multi hop in Prompt studio ide and structure tool * UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item * Added executors for agentic prompt studio * Added executors for agentic prompt studio * Removed redundant envs * Removed redundant envs * Removed redundant envs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * Removed redundant envs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed redundant envs * adding worker for callbacks * adding worker for callbacks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pluggable apps and plugins to fit the new async prompt execution architecture * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pluggable apps and plugins to fit the new async prompt execution architecture * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pluggable apps and plugins to fit the new async prompt execution architecture * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * adding worker for callbacks * fix: write output files in agentic extraction pipeline Agentic extraction returned early without writing INFILE (JSON) or METADATA.json, causing destination connectors to read the original PDF and fail with "Expected tool output type: TXT, got: application/pdf". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850) * UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants in all affected test files to avoid world-writable directory vulnerabilities. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update docs * UN-3266 fix: remove dead code with undefined names in fetch_response Remove unreachable code block after the async callback return in fetch_response that still referenced output_count_before and response from the old synchronous implementation, causing ruff F821 errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Un 3266 fix security hotspot tmp paths (#1851) * UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants in all affected test files to avoid world-writable directory vulnerabilities. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve ruff linting failures across multiple files - B026: pass url positionally in worker_celery.py to avoid star-arg after keyword - N803: rename MockAsyncResult to mock_async_result in test_tasks.py - E501/I001: fix long line and import sort in llm_whisperer helper - ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers - F841: remove unused workflow_id and result assignments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849 - S2259: guard against None after _discover_plugins() in loader.py to satisfy static analysis on the dict[str,type]|None field type - S1244: replace float equality checks with pytest.approx() in test_answer_prompt.py and test_phase2h.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve SonarCloud code smells in PR #1849 - S5799: Merge all implicit string concatenations in log messages (legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py, registry.py, variable_replacement.py, structure_tool_task.py) - S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in dispatcher.py - S1871: Merge identical elif/else branches in tasks.py and test_sanity_phase6j.py - S1186: Add comment to empty stub method in test_sanity_phase6a.py - S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j and test_phase5d.py - S117: Rename PascalCase local variables to snake_case in test_sanity_phase3/5/6i.py - S5655: Broaden tool type annotation to StreamMixin in IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config - docker:S7031: Merge consecutive RUN instructions in worker-unified.Dockerfile - javascript:S1128: Remove unused pollForCompletion import in usePromptRun.js Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: wrap long log message in dispatcher.py to fix E501 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve remaining SonarCloud S117 naming violations Rename PascalCase local variables to snake_case to comply with S117: - legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results (AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc, VariableReplacementService→variable_replacement_svc, LLM→llm_cls, EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and update all downstream usages including _apply_type_conversion and _handle_summarize - test_phase1_log_streaming.py: rename Mock* local variables to mock_* snake_case equivalents - test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls and MockShim→mock_shim_cls across all 10 test methods - test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test; fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls in _mock_prompt_deps helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849 - test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase local variables in _mock_prompt_deps/_mock_deps to snake_case (RetrievalService→retrieval_svc, VariableReplacementService→ variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls, AnswerPromptService→answer_prompt_svc_cls) — fixes S117 - test_sanity_phase3.py: remove unused local variable "result" — fixes S1481 - structure_tool_task.py: remove redundant json.JSONDecodeError from except clause (subclass of ValueError) — fixes S5713 - shared/workflow/execution/service.py: replace generic Exception with RuntimeError for structure tool failure — fixes S112 - run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and replace 10 literal "executor" occurrences — fixes S1192 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations - Reduce cognitive complexity in answer_prompt.py: - Extract _build_grammar_notes, _run_webhook_postprocess helpers - _is_safe_public_url: extracted _resolve_host_addresses helper - handle_json: early-return pattern eliminates nesting - construct_prompt: delegates grammar loop to _build_grammar_notes - Reduce cognitive complexity in legacy_executor.py: - Extract _execute_single_prompt, _run_table_extraction helpers - Extract _run_challenge_if_enabled, _run_evaluation_if_enabled - Extract _inject_table_settings, _finalize_pipeline_result - Extract _convert_number_answer, _convert_scalar_answer - Extract _sanitize_dict_values helper - _handle_answer_prompt CC reduced from 50 to ~7 - Reduce CC in structure_tool_task.py: guard-clause refactor - Reduce CC in backend: dto.py, deployment_helper.py, api_deployment_views.py, prompt_studio_helper.py - Fix S117: rename PascalCase local vars in test_answer_prompt.py - Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh - Fix S1172: remove unused params from structure_tool_task.py - Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py - Fix S112/S5727 in test_execution.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: rename UsageHelper params to lowercase (N803) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192 - Add @staticmethod to _sanitize_null_values (fixes S2325 missing self) - Reduce _execute_single_prompt params from 25 to 11 (S107) by grouping services as deps tuple and extracting exec params from context.executor_params - Add NOSONAR suppression for raise exc in test helper (S112) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * UN-3266 fix: remove unused locals in _handle_answer_prompt (F841) execution_id, file_hash, log_events_id, custom_data are now extracted inside _execute_single_prompt from context.executor_params. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve Biome linting errors in frontend source files Auto-fixed 48 lint errors across 56 files: import ordering, block statements, unused variable prefixing, and formatting issues. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: replace dynamic import of SharePermission with static import in Workflows Resolves vite build warning about SharePermission.jsx being both dynamically and statically imported across the codebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve SonarCloud warnings in frontend components - Remove unnecessary try-catch around PostHog event calls - Flip negated condition in PromptOutput.handleTable for clarity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address PR #1849 review comments: fix null guards, dead code, and test drift - Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid) - URL-encode DB_USER in worker_celery.py result backend connection string - Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app - Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist) - Move profile_manager/default_profile null checks before first dereference - Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py - Handle ProfileManager.DoesNotExist as warning, not hard failure - Wrap PostHog analytics in try/catch so failures don't block prompt execution - Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status) - Reset formData when metadata is missing in ConfigureDs.jsx - Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only) - Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix missing llm_usage_reason for summarize LLM usage tracking Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so summarization costs appear under summarize_llm in API response metadata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor - Route _handle_structure_pipeline to _handle_single_pass_extraction when is_single_pass=True (was always calling _handle_answer_prompt) - Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry, falling back to _handle_answer_prompt if plugin not installed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fixing API depployment response mismatches * Fix single-pass extraction showing only one prompt result in real-time - Fix accumulation bug in usePromptOutput updateCoverage() where each loop iteration spread the original promptOutputs instead of the accumulating updatedPromptOutputs, causing only the last prompt to render - Improve index success toast to show document name - Strip adapter names from index key configs for consistent hashing - Update sdk1 uv.lock Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move summarize from sync Django plugin to executor worker for IDE index When "Summarize" is enabled and a user indexes a new document in Prompt Studio, the backend returned a 500 because the sync Django plugin tried to read the extracted .txt file before extraction happened in the worker. Fix: defer summarization to the executor worker's _handle_ide_index (extract → summarize → index), build summarize_params in the payload, and track the summarize index in the ide_index_complete callback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address PR #1849 review comments: null guards, thread safety - Fix null guard ordering in build_index_payload: add DefaultProfileError check before calling validators on default_profile - Fix null guard ordering in _fetch_single_pass_response: move check before .llm.id access and validators (was dead code after dereference) - Add threading.Lock to worker_celery singleton to prevent race under concurrent gunicorn threads Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add documentation to ExecutionResponse DTO describing result structure Documents the ExecutionResponse dataclass fields, especially the result attribute's per-file dict structure (output, metadata, metrics, error keys) as requested in PR review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup - Strip result payload from task_status to prevent IDOR data leak - Move null guard before validators in build_fetch_response_payload - Roll back indexing flag on broker failure in index_document - Use explicit IDs for spinner clearing instead of ORM serialization format - Catch broad exceptions in summarize tracking to prevent false failures - Guard prompt_id lookup in fetch_response with 400/404 responses Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix CI, tests, and add async prompt studio improvements - Fix biome import ordering in frontend test files - Fix 22 stale backend test assertions in test_tasks.py (patch targets, return values, view dispatch pattern, remove TestCeleryConfig) - Add ide_prompt_complete callback tests - Add frontend vitest config and regression tests for null guards, stale closures, and single-pass loading guard - Add LLMCompat llama-index compatibility wrapper in SDK1 - Add litellm cohere embed timeout monkey-patch (v1.82.3) - Improve S3 filesystem helper (region_name, empty credential handling) - Add RetrieverLLM and lazy LLM creation for retrievers - Improve worker cleanup: api_client.close() in finally blocks, early return on setup failure in API deployment tasks - Add workers conftest.py for test environment setup Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix pre-existing biome CI errors: import ordering and formatting Auto-fix 18 pre-existing organizeImports and formatting errors across 17 frontend files that were blocking biome ci on the entire codebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix ruff F821: add missing transaction import in prompt_studio_helper Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add input validation guards to bulk_fetch_response endpoint Validate prompt_ids (non-empty), document_id (required), and handle DoesNotExist for both prompts and document to return proper 400/404 instead of dispatching no-op tasks or raising unhandled 500 errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * IDE Call backs * Sonar issues fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update bun.lock to match package.json dependency ranges Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move ExecutionContext import into TYPE_CHECKING block Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix SonarQube issues: duplication, naming, nesting, unused var - Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to deduplicate 5 identical JSON parsing blocks in internal_views.py - Rename `User` local variable to `user_model` (naming convention) - Merge _emit_result/_emit_error into unified _emit_event() in ide_callback tasks to reduce code duplication - Extract _get_task_error() helper to deduplicate AsyncResult error retrieval in ide_index_error and ide_prompt_error - Remove unused `mock_ar_cls` variable in test_ide_callback.py - Add security note documenting why @csrf_exempt is safe on internal endpoints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Replace worker-ide-callback Dockerfile with worker-unified The IDE callback worker should use the unified worker image (worker-unified.Dockerfile) consistent with all other v2 workers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add celery_executor_agentic queue to executor worker The executor worker only consumed from celery_executor_legacy, but the agentic prompt studio dispatches tasks to celery_executor_agentic. This caused agentic operations to sit in RabbitMQ with no consumer, resulting in timeouts and stuck-in-progress states. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * FIxing email enforce type * Removing line-item from select choices * Update workers/shared/enums/worker_enums_base.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> * Update backend/workflow_manager/workflow_v2/workflow_helper.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix false success logs and silent failures in ETL destination pipelines - Widen except clause in structure_tool_task to catch FileOperationError and log write paths for diagnostics - Add diagnostic logging at all silent return-None points in destination connector so missing INFILE/METADATA paths are visible in logs - Raise RuntimeError instead of silently skipping when no tool execution result is available for DB/FS destinations, preventing false success - Remove dead retry config from execute_bin task (max_retries=0) - Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert ETL destination pipeline changes — deferring to next cut Reverts diagnostic logging and error-raising changes in structure_tool_task.py and destination_connector.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com> Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * [MISC] Fix blank LLM profile edit form and settings menu bold style (#1896) - Replace form.resetFields() with form.setFieldsValue() to fix blank edit form caused by Ant Design's initialValues being a one-time snapshot - Remove Strict Mode-incompatible cleanup that cleared editLlmProfileId - Clear editLlmProfileId in parent when adding new profile - Remove unconditional bold on first settings popover menu item Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [FEAT] Port LINE_ITEM extraction to pluggable executor architecture LINE_ITEM enforce_type prompts previously raised an unconditional error in the new workers executor. Wire the structure-tool path to delegate to a `line_item` executor plugin (mirroring how TABLE delegates to the `table_extractor` plugin), so API deployments / ETL pipelines can run LINE_ITEM extraction once the cloud `line_item_extractor` plugin is installed. - legacy_executor.py: replace the LINE_ITEM error site in `_execute_single_prompt` with a delegation call and add a new `_run_line_item_extraction()` method structurally identical to `_run_table_extraction()`. - test_line_item_extraction.py: 5 new tests covering plugin missing, success, sub-context construction, failure path, and end-to-end Celery eager-mode delegation. - test_sanity_phase6d.py: update the existing LINE_ITEM guard test to match the new "install the line_item_extractor plugin" hint and patch `ExecutorRegistry.get` to simulate the missing plugin. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add line-item to prompt studio enforce_type select choices The `line-item` enforce_type is defined in the `ToolStudioPrompt` model but was missing from the `select_choices.json` dropdown options served to the prompt studio frontend, so users could not select it. Add it now that LINE_ITEM extraction is wired through the new pluggable executor architecture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com> Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com> Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fixing API depployment response mismatches
* Fix single-pass extraction showing only one prompt result in real-time
- Fix accumulation bug in usePromptOutput updateCoverage() where each
loop iteration spread the original promptOutputs instead of the
accumulating updatedPromptOutputs, causing only the last prompt to
render
- Improve index success toast to show document name
- Strip adapter names from index key configs for consistent hashing
- Update sdk1 uv.lock
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move summarize from sync Django plugin to executor worker for IDE index
When "Summarize" is enabled and a user indexes a new document in Prompt
Studio, the backend returned a 500 because the sync Django plugin tried
to read the extracted .txt file before extraction happened in the worker.
Fix: defer summarization to the executor worker's _handle_ide_index
(extract → summarize → index), build summarize_params in the payload,
and track the summarize index in the ide_index_complete callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address PR #1849 review comments: null guards, thread safety
- Fix null guard ordering in build_index_payload: add DefaultProfileError
check before calling validators on default_profile
- Fix null guard ordering in _fetch_single_pass_response: move check
before .llm.id access and validators (was dead code after dereference)
- Add threading.Lock to worker_celery singleton to prevent race under
concurrent gunicorn threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add documentation to ExecutionResponse DTO describing result structure
Documents the ExecutionResponse dataclass fields, especially the
result attribute's per-file dict structure (output, metadata, metrics,
error keys) as requested in PR review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup
- Strip result payload from task_status to prevent IDOR data leak
- Move null guard before validators in build_fetch_response_payload
- Roll back indexing flag on broker failure in index_document
- Use explicit IDs for spinner clearing instead of ORM serialization format
- Catch broad exceptions in summarize tracking to prevent false failures
- Guard prompt_id lookup in fetch_response with 400/404 responses
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI, tests, and add async prompt studio improvements
- Fix biome import ordering in frontend test files
- Fix 22 stale backend test assertions in test_tasks.py (patch targets,
return values, view dispatch pattern, remove TestCeleryConfig)
- Add ide_prompt_complete callback tests
- Add frontend vitest config and regression tests for null guards,
stale closures, and single-pass loading guard
- Add LLMCompat llama-index compatibility wrapper in SDK1
- Add litellm cohere embed timeout monkey-patch (v1.82.3)
- Improve S3 filesystem helper (region_name, empty credential handling)
- Add RetrieverLLM and lazy LLM creation for retrievers
- Improve worker cleanup: api_client.close() in finally blocks,
early return on setup failure in API deployment tasks
- Add workers conftest.py for test environment setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix pre-existing biome CI errors: import ordering and formatting
Auto-fix 18 pre-existing organizeImports and formatting errors across
17 frontend files that were blocking biome ci on the entire codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix ruff F821: add missing transaction import in prompt_studio_helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add input validation guards to bulk_fetch_response endpoint
Validate prompt_ids (non-empty), document_id (required), and handle
DoesNotExist for both prompts and document to return proper 400/404
instead of dispatching no-op tasks or raising unhandled 500 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* IDE Call backs
* Sonar issues fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update bun.lock to match package.json dependency ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move ExecutionContext import into TYPE_CHECKING block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix SonarQube issues: duplication, naming, nesting, unused var
- Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to
deduplicate 5 identical JSON parsing blocks in internal_views.py
- Rename `User` local variable to `user_model` (naming convention)
- Merge _emit_result/_emit_error into unified _emit_event() in
ide_callback tasks to reduce code duplication
- Extract _get_task_error() helper to deduplicate AsyncResult error
retrieval in ide_index_error and ide_prompt_error
- Remove unused `mock_ar_cls` variable in test_ide_callback.py
- Add security note documenting why @csrf_exempt is safe on internal endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace worker-ide-callback Dockerfile with worker-unified
The IDE callback worker should use the unified worker image
(worker-unified.Dockerfile) consistent with all other v2 workers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add celery_executor_agentic queue to executor worker
The executor worker only consumed from celery_executor_legacy, but the
agentic prompt studio dispatches tasks to celery_executor_agentic. This
caused agentic operations to sit in RabbitMQ with no consumer, resulting
in timeouts and stuck-in-progress states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* FIxing email enforce type
* Removing line-item from select choices
* Update workers/shared/enums/worker_enums_base.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* Update backend/workflow_manager/workflow_v2/workflow_helper.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix false success logs and silent failures in ETL destination pipelines
- Widen except clause in structure_tool_task to catch FileOperationError
and log write paths for diagnostics
- Add diagnostic logging at all silent return-None points in destination
connector so missing INFILE/METADATA paths are visible in logs
- Raise RuntimeError instead of silently skipping when no tool execution
result is available for DB/FS destinations, preventing false success
- Remove dead retry config from execute_bin task (max_retries=0)
- Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert ETL destination pipeline changes — deferring to next cut
Reverts diagnostic logging and error-raising changes in
structure_tool_task.py and destination_connector.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix false success logs and missing data in ETL destination pipelines
The Celery-based structure tool execution never created the COPY_TO_FOLDER
directory that FS destination connectors expect, causing "file successfully
sent" logs with no data written. DB destinations silently skipped insertion
when tool_execution_result was None, also producing false success.
- Write structured output to COPY_TO_FOLDER for both regular and agentic paths
- Widen exception handling to catch FileOperationError from fs.json_dump
- Raise RuntimeError instead of silently skipping in FS/DB destination handlers
- Add warning logs at silent return-None points in result retrieval
- Surface METADATA.json write failures with error-level logging
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix missing context_retrieval metric for single pass extraction
The cloud single_pass_extraction plugin handles retrieval internally but
does not report context_retrieval timing in its metrics. This adds a
post-execution file-read measurement that injects the metric, matching
what RetrievalService.retrieve_complete_context provides for the normal
answer_prompt path. Also forces chunk-size=0 for single pass so the OSS
fallback path uses full-context retrieval instead of vector DB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Unstructured IO adapter PermissionError on remote storage
Rewrite UnstructuredHelper.process_document() to read file bytes via
fs.read() and wrap in BytesIO, removing the hardcoded local FileStorage
download/builtin open() pattern. The previous code assumed
input_file_path was a local filesystem path, which broke in the executor
worker container where files live in MinIO and the local staging path
isn't mounted. Now matches the pattern used by LLMWhisperer v2 and works
for both local and remote storage backends.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Defer subscription usage tracking to IDE callback workers
Move subscription usage commit from the synchronous Prompt Studio entry
points (index_document, prompt_responder) into the async IDE callback
tasks (ide_index_complete, ide_prompt_complete). With the new executor
flow, the sync entry points return before the actual execution finishes,
so the dashboard usage decorator was firing prematurely and missing real
usage data. Tracking now runs after the executor signals success via the
callback, ensuring dashboard usage metrics reflect completed runs only.
Errors from the subscription plugin are caught and logged so they never
fail the callback. Plugin is optional (OSS mode is a no-op).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix missing embedding metadata in API deployment with chunking
In the structure pipeline indexing path, _run_pipeline_index built the
INDEX executor_params without usage_kwargs. This caused EmbeddingCompat
to register its UsageHandler callback with empty kwargs, so audit DB
rows for embedding usage were stored without run_id/file_execution_id.
get_usage_by_model() in the API deployment then returned no embedding
data, leaving the embedding key absent from the response metadata.
Propagate usage_kwargs from extract_params through _run_pipeline_index
into the INDEX ExecutionContext so the embedding adapter's callback
records audit rows against the correct file_execution_id.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix email enforce type returning "NA" string and surface null in FE
The EMAIL branch in LegacyExecutor stored the literal string "NA" when
the LLM could not extract an address — first because the early-return
short-circuited before _convert_scalar_answer ran, and second because
_convert_scalar_answer never re-checked the second-pass LLM result.
The top-level NA → None sanitizer was also missing, so any "NA" that
slipped through type conversion reached the FE as a string.
Backend (workers/executor/executors/legacy_executor.py):
- _convert_scalar_answer now also returns None when the LLM
extraction itself yields "NA" (benefits both EMAIL and DATE)
- EMAIL branch in _convert_answer_by_type now delegates to
_convert_scalar_answer instead of inlining the if/else
- _sanitize_null_values re-adds the top-level scalar "NA" → None
branch as a defensive backstop, with .strip() symmetry across
dict, list, and nested helpers
Frontend (DisplayPromptResult.jsx + PromptCard.css):
- Split the null/undefined early-return: undefined still shows
"Yet to run", but null now renders an italic-grey "null" literal
so the user can distinguish "ran but produced no value" from
"never ran"
Tests:
- workers/tests/test_answer_prompt.py: TestNullSanitization NA
cases assert None instead of preserved strings; new
TestConvertScalarAnswer covers first-pass / second-pass /
success paths
- DisplayPromptResult.test.jsx: null test asserts the literal
"null" is rendered (and "Yet to run" is not)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Feat/line item executor plugin (#1899)
* [FIX] Executor Queue for Agentic extraction (#1893)
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fixing API depployment response mismatches
* Fix single-pass extraction showing only one prompt result in real-time
- Fix accumulation bug in usePromptOutput updateCoverage() where each
loop iteration spread the original promptOutputs instead of the
accumulating updatedPromptOutputs, causing only the last prompt to
render
- Improve index success toast to show document name
- Strip adapter names from index key configs for consistent hashing
- Update sdk1 uv.lock
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move summarize from sync Django plugin to executor worker for IDE index
When "Summarize" is enabled and a user indexes a new document in Prompt
Studio, the backend returned a 500 because the sync Django plugin tried
to read the extracted .txt file before extraction happened in the worker.
Fix: defer summarization to the executor worker's _handle_ide_index
(extract → summarize → index), build summarize_params in the payload,
and track the summarize index in the ide_index_complete callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address PR #1849 review comments: null guards, thread safety
- Fix null guard ordering in build_index_payload: add DefaultProfileError
check before calling validators on default_profile
- Fix null guard ordering in _fetch_single_pass_response: move check
before .llm.id access and validators (was dead code after dereference)
- Add threading.Lock to worker_celery singleton to prevent race under
concurrent gunicorn threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add documentation to ExecutionResponse DTO describing result structure
Documents the ExecutionResponse dataclass fields, especially the
result attribute's per-file dict structure (output, metadata, metrics,
error keys) as requested in PR review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup
- Strip result payload from task_status to prevent IDOR data leak
- Move null guard before validators in build_fetch_response_payload
- Roll back indexing flag on broker failure in index_document
- Use explicit IDs for spinner clearing instead of ORM serialization format
- Catch broad exceptions in summarize tracking to prevent false failures
- Guard prompt_id lookup in fetch_response with 400/404 responses
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI, tests, and add async prompt studio improvements
- Fix biome import ordering in frontend test files
- Fix 22 stale backend test assertions in test_tasks.py (patch targets,
return values, view dispatch pattern, remove TestCeleryConfig)
- Add ide_prompt_complete callback tests
- Add frontend vitest config and regression tests for null guards,
stale closures, and single-pass loading guard
- Add LLMCompat llama-index compatibility wrapper in SDK1
- Add litellm cohere embed timeout monkey-patch (v1.82.3)
- Improve S3 filesystem helper (region_name, empty credential handling)
- Add RetrieverLLM and lazy LLM creation for retrievers
- Improve worker cleanup: api_client.close() in finally blocks,
early return on setup failure in API deployment tasks
- Add workers conftest.py for test environment setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix pre-existing biome CI errors: import ordering and formatting
Auto-fix 18 pre-existing organizeImports and formatting errors across
17 frontend files that were blocking biome ci on the entire codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix ruff F821: add missing transaction import in prompt_studio_helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add input validation guards to bulk_fetch_response endpoint
Validate prompt_ids (non-empty), document_id (required), and handle
DoesNotExist for both prompts and document to return proper 400/404
instead of dispatching no-op tasks or raising unhandled 500 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* IDE Call backs
* Sonar issues fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update bun.lock to match package.json dependency ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move ExecutionContext import into TYPE_CHECKING block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix SonarQube issues: duplication, naming, nesting, unused var
- Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to
deduplicate 5 identical JSON parsing blocks in internal_views.py
- Rename `User` local variable to `user_model` (naming convention)
- Merge _emit_result/_emit_error into unified _emit_event() in
ide_callback tasks to reduce code duplication
- Extract _get_task_error() helper to deduplicate AsyncResult error
retrieval in ide_index_error and ide_prompt_error
- Remove unused `mock_ar_cls` variable in test_ide_callback.py
- Add security note documenting why @csrf_exempt is safe on internal endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace worker-ide-callback Dockerfile with worker-unified
The IDE callback worker should use the unified worker image
(worker-unified.Dockerfile) consistent with all other v2 workers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add celery_executor_agentic queue to executor worker
The executor worker only consumed from celery_executor_legacy, but the
agentic prompt studio dispatches tasks to celery_executor_agentic. This
caused agentic operations to sit in RabbitMQ with no consumer, resulting
in timeouts and stuck-in-progress states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* FIxing email enforce type
* Removing line-item from select choices
* Update workers/shared/enums/worker_enums_base.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* Update backend/workflow_manager/workflow_v2/workflow_helper.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix false success logs and silent failures in ETL destination pipelines
- Widen except clause in structure_tool_task to catch FileOperationError
and log write paths for diagnostics
- Add diagnostic logging at all silent return-None points in destination
connector so missing INFILE/METADATA paths are visible in logs
- Raise RuntimeError instead of silently skipping when no tool execution
result is available for DB/FS destinations, preventing false success
- Remove dead retry config from execute_bin task (max_retries=0)
- Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert ETL destination pipeline changes — deferring to next cut
Reverts diagnostic logging and error-raising changes in
structure_tool_task.py and destination_connector.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* [MISC] Fix blank LLM profile edit form and settings menu bold style (#1896)
- Replace form.resetFields() with form.setFieldsValue() to fix blank
edit form caused by Ant Design's initialValues being a one-time snapshot
- Remove Strict Mode-incompatible cleanup that cleared editLlmProfileId
- Clear editLlmProfileId in parent when adding new profile
- Remove unconditional bold on first settings popover menu item
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [FEAT] Port LINE_ITEM extraction to pluggable executor architecture
LINE_ITEM enforce_type prompts previously raised an unconditional error
in the new workers executor. Wire the structure-tool path to delegate to
a `line_item` executor plugin (mirroring how TABLE delegates to the
`table_extractor` plugin), so API deployments / ETL pipelines can run
LINE_ITEM extraction once the cloud `line_item_extractor` plugin is
installed.
- legacy_executor.py: replace the LINE_ITEM error site in
`_execute_single_prompt` with a delegation call and add a new
`_run_line_item_extraction()` method structurally identical to
`_run_table_extraction()`.
- test_line_item_extraction.py: 5 new tests covering plugin missing,
success, sub-context construction, failure path, and end-to-end
Celery eager-mode delegation.
- test_sanity_phase6d.py: update the existing LINE_ITEM guard test
to match the new "install the line_item_extractor plugin" hint
and patch `ExecutorRegistry.get` to simulate the missing plugin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
* Guard against undefined connector type in PostHog event lookup
ConfigureDs called type.toUpperCase() unconditionally, which threw a
TypeError when type was undefined and broke the connector save flow.
Use optional chaining and only fire the PostHog event when the key
maps to a known connector.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add worker-executor-v2 service to docker-compose under workers-v2 profile
Defines a second executor worker container that runs the unified
worker image with WORKER_TYPE=executor on port 8092. Gated behind
the workers-v2 compose profile so it is opt-in and does not affect
default deployments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Feat/line item executor plugin (#1900)
* [FIX] Executor Queue for Agentic extraction (#1893)
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthro…
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fixing API depployment response mismatches
* Fix single-pass extraction showing only one prompt result in real-time
- Fix accumulation bug in usePromptOutput updateCoverage() where each
loop iteration spread the original promptOutputs instead of the
accumulating updatedPromptOutputs, causing only the last prompt to
render
- Improve index success toast to show document name
- Strip adapter names from index key configs for consistent hashing
- Update sdk1 uv.lock
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move summarize from sync Django plugin to executor worker for IDE index
When "Summarize" is enabled and a user indexes a new document in Prompt
Studio, the backend returned a 500 because the sync Django plugin tried
to read the extracted .txt file before extraction happened in the worker.
Fix: defer summarization to the executor worker's _handle_ide_index
(extract → summarize → index), build summarize_params in the payload,
and track the summarize index in the ide_index_complete callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address PR #1849 review comments: null guards, thread safety
- Fix null guard ordering in build_index_payload: add DefaultProfileError
check before calling validators on default_profile
- Fix null guard ordering in _fetch_single_pass_response: move check
before .llm.id access and validators (was dead code after dereference)
- Add threading.Lock to worker_celery singleton to prevent race under
concurrent gunicorn threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add documentation to ExecutionResponse DTO describing result structure
Documents the ExecutionResponse dataclass fields, especially the
result attribute's per-file dict structure (output, metadata, metrics,
error keys) as requested in PR review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup
- Strip result payload from task_status to prevent IDOR data leak
- Move null guard before validators in build_fetch_response_payload
- Roll back indexing flag on broker failure in index_document
- Use explicit IDs for spinner clearing instead of ORM serialization format
- Catch broad exceptions in summarize tracking to prevent false failures
- Guard prompt_id lookup in fetch_response with 400/404 responses
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI, tests, and add async prompt studio improvements
- Fix biome import ordering in frontend test files
- Fix 22 stale backend test assertions in test_tasks.py (patch targets,
return values, view dispatch pattern, remove TestCeleryConfig)
- Add ide_prompt_complete callback tests
- Add frontend vitest config and regression tests for null guards,
stale closures, and single-pass loading guard
- Add LLMCompat llama-index compatibility wrapper in SDK1
- Add litellm cohere embed timeout monkey-patch (v1.82.3)
- Improve S3 filesystem helper (region_name, empty credential handling)
- Add RetrieverLLM and lazy LLM creation for retrievers
- Improve worker cleanup: api_client.close() in finally blocks,
early return on setup failure in API deployment tasks
- Add workers conftest.py for test environment setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix pre-existing biome CI errors: import ordering and formatting
Auto-fix 18 pre-existing organizeImports and formatting errors across
17 frontend files that were blocking biome ci on the entire codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix ruff F821: add missing transaction import in prompt_studio_helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add input validation guards to bulk_fetch_response endpoint
Validate prompt_ids (non-empty), document_id (required), and handle
DoesNotExist for both prompts and document to return proper 400/404
instead of dispatching no-op tasks or raising unhandled 500 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* IDE Call backs
* Sonar issues fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update bun.lock to match package.json dependency ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move ExecutionContext import into TYPE_CHECKING block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix SonarQube issues: duplication, naming, nesting, unused var
- Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to
deduplicate 5 identical JSON parsing blocks in internal_views.py
- Rename `User` local variable to `user_model` (naming convention)
- Merge _emit_result/_emit_error into unified _emit_event() in
ide_callback tasks to reduce code duplication
- Extract _get_task_error() helper to deduplicate AsyncResult error
retrieval in ide_index_error and ide_prompt_error
- Remove unused `mock_ar_cls` variable in test_ide_callback.py
- Add security note documenting why @csrf_exempt is safe on internal endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace worker-ide-callback Dockerfile with worker-unified
The IDE callback worker should use the unified worker image
(worker-unified.Dockerfile) consistent with all other v2 workers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add celery_executor_agentic queue to executor worker
The executor worker only consumed from celery_executor_legacy, but the
agentic prompt studio dispatches tasks to celery_executor_agentic. This
caused agentic operations to sit in RabbitMQ with no consumer, resulting
in timeouts and stuck-in-progress states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* FIxing email enforce type
* Removing line-item from select choices
* Update workers/shared/enums/worker_enums_base.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* Update backend/workflow_manager/workflow_v2/workflow_helper.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix false success logs and silent failures in ETL destination pipelines
- Widen except clause in structure_tool_task to catch FileOperationError
and log write paths for diagnostics
- Add diagnostic logging at all silent return-None points in destination
connector so missing INFILE/METADATA paths are visible in logs
- Raise RuntimeError instead of silently skipping when no tool execution
result is available for DB/FS destinations, preventing false success
- Remove dead retry config from execute_bin task (max_retries=0)
- Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert ETL destination pipeline changes — deferring to next cut
Reverts diagnostic logging and error-raising changes in
structure_tool_task.py and destination_connector.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix false success logs and missing data in ETL destination pipelines
The Celery-based structure tool execution never created the COPY_TO_FOLDER
directory that FS destination connectors expect, causing "file successfully
sent" logs with no data written. DB destinations silently skipped insertion
when tool_execution_result was None, also producing false success.
- Write structured output to COPY_TO_FOLDER for both regular and agentic paths
- Widen exception handling to catch FileOperationError from fs.json_dump
- Raise RuntimeError instead of silently skipping in FS/DB destination handlers
- Add warning logs at silent return-None points in result retrieval
- Surface METADATA.json write failures with error-level logging
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix missing context_retrieval metric for single pass extraction
The cloud single_pass_extraction plugin handles retrieval internally but
does not report context_retrieval timing in its metrics. This adds a
post-execution file-read measurement that injects the metric, matching
what RetrievalService.retrieve_complete_context provides for the normal
answer_prompt path. Also forces chunk-size=0 for single pass so the OSS
fallback path uses full-context retrieval instead of vector DB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Unstructured IO adapter PermissionError on remote storage
Rewrite UnstructuredHelper.process_document() to read file bytes via
fs.read() and wrap in BytesIO, removing the hardcoded local FileStorage
download/builtin open() pattern. The previous code assumed
input_file_path was a local filesystem path, which broke in the executor
worker container where files live in MinIO and the local staging path
isn't mounted. Now matches the pattern used by LLMWhisperer v2 and works
for both local and remote storage backends.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Defer subscription usage tracking to IDE callback workers
Move subscription usage commit from the synchronous Prompt Studio entry
points (index_document, prompt_responder) into the async IDE callback
tasks (ide_index_complete, ide_prompt_complete). With the new executor
flow, the sync entry points return before the actual execution finishes,
so the dashboard usage decorator was firing prematurely and missing real
usage data. Tracking now runs after the executor signals success via the
callback, ensuring dashboard usage metrics reflect completed runs only.
Errors from the subscription plugin are caught and logged so they never
fail the callback. Plugin is optional (OSS mode is a no-op).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix missing embedding metadata in API deployment with chunking
In the structure pipeline indexing path, _run_pipeline_index built the
INDEX executor_params without usage_kwargs. This caused EmbeddingCompat
to register its UsageHandler callback with empty kwargs, so audit DB
rows for embedding usage were stored without run_id/file_execution_id.
get_usage_by_model() in the API deployment then returned no embedding
data, leaving the embedding key absent from the response metadata.
Propagate usage_kwargs from extract_params through _run_pipeline_index
into the INDEX ExecutionContext so the embedding adapter's callback
records audit rows against the correct file_execution_id.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix email enforce type returning "NA" string and surface null in FE
The EMAIL branch in LegacyExecutor stored the literal string "NA" when
the LLM could not extract an address — first because the early-return
short-circuited before _convert_scalar_answer ran, and second because
_convert_scalar_answer never re-checked the second-pass LLM result.
The top-level NA → None sanitizer was also missing, so any "NA" that
slipped through type conversion reached the FE as a string.
Backend (workers/executor/executors/legacy_executor.py):
- _convert_scalar_answer now also returns None when the LLM
extraction itself yields "NA" (benefits both EMAIL and DATE)
- EMAIL branch in _convert_answer_by_type now delegates to
_convert_scalar_answer instead of inlining the if/else
- _sanitize_null_values re-adds the top-level scalar "NA" → None
branch as a defensive backstop, with .strip() symmetry across
dict, list, and nested helpers
Frontend (DisplayPromptResult.jsx + PromptCard.css):
- Split the null/undefined early-return: undefined still shows
"Yet to run", but null now renders an italic-grey "null" literal
so the user can distinguish "ran but produced no value" from
"never ran"
Tests:
- workers/tests/test_answer_prompt.py: TestNullSanitization NA
cases assert None instead of preserved strings; new
TestConvertScalarAnswer covers first-pass / second-pass /
success paths
- DisplayPromptResult.test.jsx: null test asserts the literal
"null" is rendered (and "Yet to run" is not)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Feat/line item executor plugin (#1899)
* [FIX] Executor Queue for Agentic extraction (#1893)
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fixing API depployment response mismatches
* Fix single-pass extraction showing only one prompt result in real-time
- Fix accumulation bug in usePromptOutput updateCoverage() where each
loop iteration spread the original promptOutputs instead of the
accumulating updatedPromptOutputs, causing only the last prompt to
render
- Improve index success toast to show document name
- Strip adapter names from index key configs for consistent hashing
- Update sdk1 uv.lock
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move summarize from sync Django plugin to executor worker for IDE index
When "Summarize" is enabled and a user indexes a new document in Prompt
Studio, the backend returned a 500 because the sync Django plugin tried
to read the extracted .txt file before extraction happened in the worker.
Fix: defer summarization to the executor worker's _handle_ide_index
(extract → summarize → index), build summarize_params in the payload,
and track the summarize index in the ide_index_complete callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address PR #1849 review comments: null guards, thread safety
- Fix null guard ordering in build_index_payload: add DefaultProfileError
check before calling validators on default_profile
- Fix null guard ordering in _fetch_single_pass_response: move check
before .llm.id access and validators (was dead code after dereference)
- Add threading.Lock to worker_celery singleton to prevent race under
concurrent gunicorn threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add documentation to ExecutionResponse DTO describing result structure
Documents the ExecutionResponse dataclass fields, especially the
result attribute's per-file dict structure (output, metadata, metrics,
error keys) as requested in PR review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup
- Strip result payload from task_status to prevent IDOR data leak
- Move null guard before validators in build_fetch_response_payload
- Roll back indexing flag on broker failure in index_document
- Use explicit IDs for spinner clearing instead of ORM serialization format
- Catch broad exceptions in summarize tracking to prevent false failures
- Guard prompt_id lookup in fetch_response with 400/404 responses
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI, tests, and add async prompt studio improvements
- Fix biome import ordering in frontend test files
- Fix 22 stale backend test assertions in test_tasks.py (patch targets,
return values, view dispatch pattern, remove TestCeleryConfig)
- Add ide_prompt_complete callback tests
- Add frontend vitest config and regression tests for null guards,
stale closures, and single-pass loading guard
- Add LLMCompat llama-index compatibility wrapper in SDK1
- Add litellm cohere embed timeout monkey-patch (v1.82.3)
- Improve S3 filesystem helper (region_name, empty credential handling)
- Add RetrieverLLM and lazy LLM creation for retrievers
- Improve worker cleanup: api_client.close() in finally blocks,
early return on setup failure in API deployment tasks
- Add workers conftest.py for test environment setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix pre-existing biome CI errors: import ordering and formatting
Auto-fix 18 pre-existing organizeImports and formatting errors across
17 frontend files that were blocking biome ci on the entire codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix ruff F821: add missing transaction import in prompt_studio_helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add input validation guards to bulk_fetch_response endpoint
Validate prompt_ids (non-empty), document_id (required), and handle
DoesNotExist for both prompts and document to return proper 400/404
instead of dispatching no-op tasks or raising unhandled 500 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* IDE Call backs
* Sonar issues fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update bun.lock to match package.json dependency ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move ExecutionContext import into TYPE_CHECKING block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix SonarQube issues: duplication, naming, nesting, unused var
- Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to
deduplicate 5 identical JSON parsing blocks in internal_views.py
- Rename `User` local variable to `user_model` (naming convention)
- Merge _emit_result/_emit_error into unified _emit_event() in
ide_callback tasks to reduce code duplication
- Extract _get_task_error() helper to deduplicate AsyncResult error
retrieval in ide_index_error and ide_prompt_error
- Remove unused `mock_ar_cls` variable in test_ide_callback.py
- Add security note documenting why @csrf_exempt is safe on internal endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace worker-ide-callback Dockerfile with worker-unified
The IDE callback worker should use the unified worker image
(worker-unified.Dockerfile) consistent with all other v2 workers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add celery_executor_agentic queue to executor worker
The executor worker only consumed from celery_executor_legacy, but the
agentic prompt studio dispatches tasks to celery_executor_agentic. This
caused agentic operations to sit in RabbitMQ with no consumer, resulting
in timeouts and stuck-in-progress states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* FIxing email enforce type
* Removing line-item from select choices
* Update workers/shared/enums/worker_enums_base.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* Update backend/workflow_manager/workflow_v2/workflow_helper.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix false success logs and silent failures in ETL destination pipelines
- Widen except clause in structure_tool_task to catch FileOperationError
and log write paths for diagnostics
- Add diagnostic logging at all silent return-None points in destination
connector so missing INFILE/METADATA paths are visible in logs
- Raise RuntimeError instead of silently skipping when no tool execution
result is available for DB/FS destinations, preventing false success
- Remove dead retry config from execute_bin task (max_retries=0)
- Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert ETL destination pipeline changes — deferring to next cut
Reverts diagnostic logging and error-raising changes in
structure_tool_task.py and destination_connector.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* [MISC] Fix blank LLM profile edit form and settings menu bold style (#1896)
- Replace form.resetFields() with form.setFieldsValue() to fix blank
edit form caused by Ant Design's initialValues being a one-time snapshot
- Remove Strict Mode-incompatible cleanup that cleared editLlmProfileId
- Clear editLlmProfileId in parent when adding new profile
- Remove unconditional bold on first settings popover menu item
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [FEAT] Port LINE_ITEM extraction to pluggable executor architecture
LINE_ITEM enforce_type prompts previously raised an unconditional error
in the new workers executor. Wire the structure-tool path to delegate to
a `line_item` executor plugin (mirroring how TABLE delegates to the
`table_extractor` plugin), so API deployments / ETL pipelines can run
LINE_ITEM extraction once the cloud `line_item_extractor` plugin is
installed.
- legacy_executor.py: replace the LINE_ITEM error site in
`_execute_single_prompt` with a delegation call and add a new
`_run_line_item_extraction()` method structurally identical to
`_run_table_extraction()`.
- test_line_item_extraction.py: 5 new tests covering plugin missing,
success, sub-context construction, failure path, and end-to-end
Celery eager-mode delegation.
- test_sanity_phase6d.py: update the existing LINE_ITEM guard test
to match the new "install the line_item_extractor plugin" hint
and patch `ExecutorRegistry.get` to simulate the missing plugin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
* Guard against undefined connector type in PostHog event lookup
ConfigureDs called type.toUpperCase() unconditionally, which threw a
TypeError when type was undefined and broke the connector save flow.
Use optional chaining and only fire the PostHog event when the key
maps to a known connector.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add worker-executor-v2 service to docker-compose under workers-v2 profile
Defines a second executor worker container that runs the unified
worker image with WORKER_TYPE=executor on port 8092. Gated behind
the workers-v2 compose profile so it is opt-in and does not affect
default deployments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Feat/line item executor plugin (#1900)
* [FIX] Executor Queue for Agentic extraction (#1893)
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic…
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fixing API depployment response mismatches
* Fix single-pass extraction showing only one prompt result in real-time
- Fix accumulation bug in usePromptOutput updateCoverage() where each
loop iteration spread the original promptOutputs instead of the
accumulating updatedPromptOutputs, causing only the last prompt to
render
- Improve index success toast to show document name
- Strip adapter names from index key configs for consistent hashing
- Update sdk1 uv.lock
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move summarize from sync Django plugin to executor worker for IDE index
When "Summarize" is enabled and a user indexes a new document in Prompt
Studio, the backend returned a 500 because the sync Django plugin tried
to read the extracted .txt file before extraction happened in the worker.
Fix: defer summarization to the executor worker's _handle_ide_index
(extract → summarize → index), build summarize_params in the payload,
and track the summarize index in the ide_index_complete callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address PR #1849 review comments: null guards, thread safety
- Fix null guard ordering in build_index_payload: add DefaultProfileError
check before calling validators on default_profile
- Fix null guard ordering in _fetch_single_pass_response: move check
before .llm.id access and validators (was dead code after dereference)
- Add threading.Lock to worker_celery singleton to prevent race under
concurrent gunicorn threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add documentation to ExecutionResponse DTO describing result structure
Documents the ExecutionResponse dataclass fields, especially the
result attribute's per-file dict structure (output, metadata, metrics,
error keys) as requested in PR review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup
- Strip result payload from task_status to prevent IDOR data leak
- Move null guard before validators in build_fetch_response_payload
- Roll back indexing flag on broker failure in index_document
- Use explicit IDs for spinner clearing instead of ORM serialization format
- Catch broad exceptions in summarize tracking to prevent false failures
- Guard prompt_id lookup in fetch_response with 400/404 responses
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI, tests, and add async prompt studio improvements
- Fix biome import ordering in frontend test files
- Fix 22 stale backend test assertions in test_tasks.py (patch targets,
return values, view dispatch pattern, remove TestCeleryConfig)
- Add ide_prompt_complete callback tests
- Add frontend vitest config and regression tests for null guards,
stale closures, and single-pass loading guard
- Add LLMCompat llama-index compatibility wrapper in SDK1
- Add litellm cohere embed timeout monkey-patch (v1.82.3)
- Improve S3 filesystem helper (region_name, empty credential handling)
- Add RetrieverLLM and lazy LLM creation for retrievers
- Improve worker cleanup: api_client.close() in finally blocks,
early return on setup failure in API deployment tasks
- Add workers conftest.py for test environment setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix pre-existing biome CI errors: import ordering and formatting
Auto-fix 18 pre-existing organizeImports and formatting errors across
17 frontend files that were blocking biome ci on the entire codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix ruff F821: add missing transaction import in prompt_studio_helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add input validation guards to bulk_fetch_response endpoint
Validate prompt_ids (non-empty), document_id (required), and handle
DoesNotExist for both prompts and document to return proper 400/404
instead of dispatching no-op tasks or raising unhandled 500 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* IDE Call backs
* Sonar issues fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update bun.lock to match package.json dependency ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move ExecutionContext import into TYPE_CHECKING block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix SonarQube issues: duplication, naming, nesting, unused var
- Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to
deduplicate 5 identical JSON parsing blocks in internal_views.py
- Rename `User` local variable to `user_model` (naming convention)
- Merge _emit_result/_emit_error into unified _emit_event() in
ide_callback tasks to reduce code duplication
- Extract _get_task_error() helper to deduplicate AsyncResult error
retrieval in ide_index_error and ide_prompt_error
- Remove unused `mock_ar_cls` variable in test_ide_callback.py
- Add security note documenting why @csrf_exempt is safe on internal endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace worker-ide-callback Dockerfile with worker-unified
The IDE callback worker should use the unified worker image
(worker-unified.Dockerfile) consistent with all other v2 workers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add celery_executor_agentic queue to executor worker
The executor worker only consumed from celery_executor_legacy, but the
agentic prompt studio dispatches tasks to celery_executor_agentic. This
caused agentic operations to sit in RabbitMQ with no consumer, resulting
in timeouts and stuck-in-progress states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* FIxing email enforce type
* Removing line-item from select choices
* Update workers/shared/enums/worker_enums_base.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* Update backend/workflow_manager/workflow_v2/workflow_helper.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix false success logs and silent failures in ETL destination pipelines
- Widen except clause in structure_tool_task to catch FileOperationError
and log write paths for diagnostics
- Add diagnostic logging at all silent return-None points in destination
connector so missing INFILE/METADATA paths are visible in logs
- Raise RuntimeError instead of silently skipping when no tool execution
result is available for DB/FS destinations, preventing false success
- Remove dead retry config from execute_bin task (max_retries=0)
- Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert ETL destination pipeline changes — deferring to next cut
Reverts diagnostic logging and error-raising changes in
structure_tool_task.py and destination_connector.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix false success logs and missing data in ETL destination pipelines
The Celery-based structure tool execution never created the COPY_TO_FOLDER
directory that FS destination connectors expect, causing "file successfully
sent" logs with no data written. DB destinations silently skipped insertion
when tool_execution_result was None, also producing false success.
- Write structured output to COPY_TO_FOLDER for both regular and agentic paths
- Widen exception handling to catch FileOperationError from fs.json_dump
- Raise RuntimeError instead of silently skipping in FS/DB destination handlers
- Add warning logs at silent return-None points in result retrieval
- Surface METADATA.json write failures with error-level logging
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix missing context_retrieval metric for single pass extraction
The cloud single_pass_extraction plugin handles retrieval internally but
does not report context_retrieval timing in its metrics. This adds a
post-execution file-read measurement that injects the metric, matching
what RetrievalService.retrieve_complete_context provides for the normal
answer_prompt path. Also forces chunk-size=0 for single pass so the OSS
fallback path uses full-context retrieval instead of vector DB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix Unstructured IO adapter PermissionError on remote storage
Rewrite UnstructuredHelper.process_document() to read file bytes via
fs.read() and wrap in BytesIO, removing the hardcoded local FileStorage
download/builtin open() pattern. The previous code assumed
input_file_path was a local filesystem path, which broke in the executor
worker container where files live in MinIO and the local staging path
isn't mounted. Now matches the pattern used by LLMWhisperer v2 and works
for both local and remote storage backends.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Defer subscription usage tracking to IDE callback workers
Move subscription usage commit from the synchronous Prompt Studio entry
points (index_document, prompt_responder) into the async IDE callback
tasks (ide_index_complete, ide_prompt_complete). With the new executor
flow, the sync entry points return before the actual execution finishes,
so the dashboard usage decorator was firing prematurely and missing real
usage data. Tracking now runs after the executor signals success via the
callback, ensuring dashboard usage metrics reflect completed runs only.
Errors from the subscription plugin are caught and logged so they never
fail the callback. Plugin is optional (OSS mode is a no-op).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix missing embedding metadata in API deployment with chunking
In the structure pipeline indexing path, _run_pipeline_index built the
INDEX executor_params without usage_kwargs. This caused EmbeddingCompat
to register its UsageHandler callback with empty kwargs, so audit DB
rows for embedding usage were stored without run_id/file_execution_id.
get_usage_by_model() in the API deployment then returned no embedding
data, leaving the embedding key absent from the response metadata.
Propagate usage_kwargs from extract_params through _run_pipeline_index
into the INDEX ExecutionContext so the embedding adapter's callback
records audit rows against the correct file_execution_id.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix email enforce type returning "NA" string and surface null in FE
The EMAIL branch in LegacyExecutor stored the literal string "NA" when
the LLM could not extract an address — first because the early-return
short-circuited before _convert_scalar_answer ran, and second because
_convert_scalar_answer never re-checked the second-pass LLM result.
The top-level NA → None sanitizer was also missing, so any "NA" that
slipped through type conversion reached the FE as a string.
Backend (workers/executor/executors/legacy_executor.py):
- _convert_scalar_answer now also returns None when the LLM
extraction itself yields "NA" (benefits both EMAIL and DATE)
- EMAIL branch in _convert_answer_by_type now delegates to
_convert_scalar_answer instead of inlining the if/else
- _sanitize_null_values re-adds the top-level scalar "NA" → None
branch as a defensive backstop, with .strip() symmetry across
dict, list, and nested helpers
Frontend (DisplayPromptResult.jsx + PromptCard.css):
- Split the null/undefined early-return: undefined still shows
"Yet to run", but null now renders an italic-grey "null" literal
so the user can distinguish "ran but produced no value" from
"never ran"
Tests:
- workers/tests/test_answer_prompt.py: TestNullSanitization NA
cases assert None instead of preserved strings; new
TestConvertScalarAnswer covers first-pass / second-pass /
success paths
- DisplayPromptResult.test.jsx: null test asserts the literal
"null" is rendered (and "Yet to run" is not)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Feat/line item executor plugin (#1899)
* [FIX] Executor Queue for Agentic extraction (#1893)
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fixing API depployment response mismatches
* Fix single-pass extraction showing only one prompt result in real-time
- Fix accumulation bug in usePromptOutput updateCoverage() where each
loop iteration spread the original promptOutputs instead of the
accumulating updatedPromptOutputs, causing only the last prompt to
render
- Improve index success toast to show document name
- Strip adapter names from index key configs for consistent hashing
- Update sdk1 uv.lock
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move summarize from sync Django plugin to executor worker for IDE index
When "Summarize" is enabled and a user indexes a new document in Prompt
Studio, the backend returned a 500 because the sync Django plugin tried
to read the extracted .txt file before extraction happened in the worker.
Fix: defer summarization to the executor worker's _handle_ide_index
(extract → summarize → index), build summarize_params in the payload,
and track the summarize index in the ide_index_complete callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Address PR #1849 review comments: null guards, thread safety
- Fix null guard ordering in build_index_payload: add DefaultProfileError
check before calling validators on default_profile
- Fix null guard ordering in _fetch_single_pass_response: move check
before .llm.id access and validators (was dead code after dereference)
- Add threading.Lock to worker_celery singleton to prevent race under
concurrent gunicorn threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add documentation to ExecutionResponse DTO describing result structure
Documents the ExecutionResponse dataclass fields, especially the
result attribute's per-file dict structure (output, metadata, metrics,
error keys) as requested in PR review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix PR review issues: IDOR, null guards, rollback, spinner, summarize, prompt lookup
- Strip result payload from task_status to prevent IDOR data leak
- Move null guard before validators in build_fetch_response_payload
- Roll back indexing flag on broker failure in index_document
- Use explicit IDs for spinner clearing instead of ORM serialization format
- Catch broad exceptions in summarize tracking to prevent false failures
- Guard prompt_id lookup in fetch_response with 400/404 responses
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI, tests, and add async prompt studio improvements
- Fix biome import ordering in frontend test files
- Fix 22 stale backend test assertions in test_tasks.py (patch targets,
return values, view dispatch pattern, remove TestCeleryConfig)
- Add ide_prompt_complete callback tests
- Add frontend vitest config and regression tests for null guards,
stale closures, and single-pass loading guard
- Add LLMCompat llama-index compatibility wrapper in SDK1
- Add litellm cohere embed timeout monkey-patch (v1.82.3)
- Improve S3 filesystem helper (region_name, empty credential handling)
- Add RetrieverLLM and lazy LLM creation for retrievers
- Improve worker cleanup: api_client.close() in finally blocks,
early return on setup failure in API deployment tasks
- Add workers conftest.py for test environment setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix pre-existing biome CI errors: import ordering and formatting
Auto-fix 18 pre-existing organizeImports and formatting errors across
17 frontend files that were blocking biome ci on the entire codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix ruff F821: add missing transaction import in prompt_studio_helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add input validation guards to bulk_fetch_response endpoint
Validate prompt_ids (non-empty), document_id (required), and handle
DoesNotExist for both prompts and document to return proper 400/404
instead of dispatching no-op tasks or raising unhandled 500 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* IDE Call backs
* Sonar issues fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix ruff errors: restore summary_profile variable, suppress TC001 in dispatcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update bun.lock to match package.json dependency ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all biome lint warnings: empty blocks, missing braces, forEach returns, unused vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move ExecutionContext import into TYPE_CHECKING block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix SonarQube issues: duplication, naming, nesting, unused var
- Extract _parse_json_body() helper and _ERR_INVALID_JSON constant to
deduplicate 5 identical JSON parsing blocks in internal_views.py
- Rename `User` local variable to `user_model` (naming convention)
- Merge _emit_result/_emit_error into unified _emit_event() in
ide_callback tasks to reduce code duplication
- Extract _get_task_error() helper to deduplicate AsyncResult error
retrieval in ide_index_error and ide_prompt_error
- Remove unused `mock_ar_cls` variable in test_ide_callback.py
- Add security note documenting why @csrf_exempt is safe on internal endpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace worker-ide-callback Dockerfile with worker-unified
The IDE callback worker should use the unified worker image
(worker-unified.Dockerfile) consistent with all other v2 workers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add celery_executor_agentic queue to executor worker
The executor worker only consumed from celery_executor_legacy, but the
agentic prompt studio dispatches tasks to celery_executor_agentic. This
caused agentic operations to sit in RabbitMQ with no consumer, resulting
in timeouts and stuck-in-progress states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* FIxing email enforce type
* Removing line-item from select choices
* Update workers/shared/enums/worker_enums_base.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* Update backend/workflow_manager/workflow_v2/workflow_helper.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix false success logs and silent failures in ETL destination pipelines
- Widen except clause in structure_tool_task to catch FileOperationError
and log write paths for diagnostics
- Add diagnostic logging at all silent return-None points in destination
connector so missing INFILE/METADATA paths are visible in logs
- Raise RuntimeError instead of silently skipping when no tool execution
result is available for DB/FS destinations, preventing false success
- Remove dead retry config from execute_bin task (max_retries=0)
- Fix duplicate EXECUTOR/IDE_CALLBACK enum entries in WorkerType
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert ETL destination pipeline changes — deferring to next cut
Reverts diagnostic logging and error-raising changes in
structure_tool_task.py and destination_connector.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* [MISC] Fix blank LLM profile edit form and settings menu bold style (#1896)
- Replace form.resetFields() with form.setFieldsValue() to fix blank
edit form caused by Ant Design's initialValues being a one-time snapshot
- Remove Strict Mode-incompatible cleanup that cleared editLlmProfileId
- Clear editLlmProfileId in parent when adding new profile
- Remove unconditional bold on first settings popover menu item
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [FEAT] Port LINE_ITEM extraction to pluggable executor architecture
LINE_ITEM enforce_type prompts previously raised an unconditional error
in the new workers executor. Wire the structure-tool path to delegate to
a `line_item` executor plugin (mirroring how TABLE delegates to the
`table_extractor` plugin), so API deployments / ETL pipelines can run
LINE_ITEM extraction once the cloud `line_item_extractor` plugin is
installed.
- legacy_executor.py: replace the LINE_ITEM error site in
`_execute_single_prompt` with a delegation call and add a new
`_run_line_item_extraction()` method structurally identical to
`_run_table_extraction()`.
- test_line_item_extraction.py: 5 new tests covering plugin missing,
success, sub-context construction, failure path, and end-to-end
Celery eager-mode delegation.
- test_sanity_phase6d.py: update the existing LINE_ITEM guard test
to match the new "install the line_item_extractor plugin" hint
and patch `ExecutorRegistry.get` to simulate the missing plugin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Ghost Jake <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
* Guard against undefined connector type in PostHog event lookup
ConfigureDs called type.toUpperCase() unconditionally, which threw a
TypeError when type was undefined and broke the connector save flow.
Use optional chaining and only fire the PostHog event when the key
maps to a known connector.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add worker-executor-v2 service to docker-compose under workers-v2 profile
Defines a second executor worker container that runs the unified
worker image with WORKER_TYPE=executor on port 8092. Gated behind
the workers-v2 compose profile so it is opt-in and does not affect
default deployments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Feat/line item executor plugin (#1900)
* [FIX] Executor Queue for Agentic extraction (#1893)
* Execution backend - revamp
* async flow
* Streaming progress to FE
* Removing multi hop in Prompt studio ide and structure tool
* UN-3234 [FIX] Add beta tag to agentic prompt studio navigation item
* Added executors for agentic prompt studio
* Added executors for agentic prompt studio
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* Removed redundant envs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed redundant envs
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pluggable apps and plugins to fit the new async prompt execution architecture
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* adding worker for callbacks
* fix: write output files in agentic extraction pipeline
Agentic extraction returned early without writing INFILE (JSON) or
METADATA.json, causing destination connectors to read the original PDF
and fail with "Expected tool output type: TXT, got: application/pdf".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests (#1850)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update docs
* UN-3266 fix: remove dead code with undefined names in fetch_response
Remove unreachable code block after the async callback return in
fetch_response that still referenced output_count_before and response
from the old synchronous implementation, causing ruff F821 errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Un 3266 fix security hotspot tmp paths (#1851)
* UN-3266 fix: replace hardcoded /tmp paths with secure temp dirs in tests
Replace hardcoded /tmp/ paths (SonarCloud S5443 security hotspots) with
pytest's tmp_path fixture or module-level tempfile.mkdtemp() constants
in all affected test files to avoid world-writable directory vulnerabilities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve ruff linting failures across multiple files
- B026: pass url positionally in worker_celery.py to avoid star-arg after keyword
- N803: rename MockAsyncResult to mock_async_result in test_tasks.py
- E501/I001: fix long line and import sort in llm_whisperer helper
- ANN401: replace Any with object|None in dispatcher.py; add noqa in test helpers
- F841: remove unused workflow_id and result assignments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-3266 fix: resolve SonarCloud bugs S2259 and S1244 in PR #1849
- S2259: guard against None after _discover_plugins() in loader.py
to satisfy static analysis on the dict[str,type]|None field type
- S1244: replace float equality checks with pytest.approx() in
test_answer_prompt.py and test_phase2h.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud code smells in PR #1849
- S5799: Merge all implicit string concatenations in log messages
(legacy_executor.py, tasks.py, dispatcher.py, orchestrator.py,
registry.py, variable_replacement.py, structure_tool_task.py)
- S1192: Extract duplicate literal to _NO_CELERY_APP_MSG constant in
dispatcher.py
- S1871: Merge identical elif/else branches in tasks.py and
test_sanity_phase6j.py
- S1186: Add comment to empty stub method in test_sanity_phase6a.py
- S1481: Remove unused local variables in test_sanity_phase6d/e/f/g/h/j
and test_phase5d.py
- S117: Rename PascalCase local variables to snake_case in
test_sanity_phase3/5/6i.py
- S5655: Broaden tool type annotation to StreamMixin in
IndexingUtils.generate_index_key and PlatformHelper.get_adapter_config
- docker:S7031: Merge consecutive RUN instructions in
worker-unified.Dockerfile
- javascript:S1128: Remove unused pollForCompletion import in
usePromptRun.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: wrap long log message in dispatcher.py to fix E501
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud S117 naming violations
Rename PascalCase local variables to snake_case to comply with S117:
- legacy_executor.py: rename tuple-unpacked _get_prompt_deps() results
(AnswerPromptService→answer_prompt_svc, RetrievalService→retrieval_svc,
VariableReplacementService→variable_replacement_svc, LLM→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls) and
update all downstream usages including _apply_type_conversion and
_handle_summarize
- test_phase1_log_streaming.py: rename Mock* local variables to
mock_* snake_case equivalents
- test_sanity_phase3.py: rename MockDispatcher→mock_dispatcher_cls
and MockShim→mock_shim_cls across all 10 test methods
- test_sanity_phase5.py: rename MockShim→mock_shim, MockX2Text→mock_x2text
in 6 test methods; MockDispatcher→mock_dispatcher_cls in dispatch test;
fix LLM_cls→llm_cls, EmbeddingCompat→embedding_compat_cls,
VectorDB→vector_db_cls in _mock_prompt_deps helper
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: resolve remaining SonarCloud code smells in PR #1849
- test_sanity_phase2/4.py, test_answer_prompt.py: rename PascalCase
local variables in _mock_prompt_deps/_mock_deps to snake_case
(RetrievalService→retrieval_svc, VariableReplacementService→
variable_replacement_svc, Index→index_cls, LLM_cls→llm_cls,
EmbeddingCompat→embedding_compat_cls, VectorDB→vector_db_cls,
AnswerPromptService→answer_prompt_svc_cls) — fixes S117
- test_sanity_phase3.py: remove unused local variable "result" — fixes S1481
- structure_tool_task.py: remove redundant json.JSONDecodeError from
except clause (subclass of ValueError) — fixes S5713
- shared/workflow/execution/service.py: replace generic Exception with
RuntimeError for structure tool failure — fixes S112
- run-worker-docker.sh: define EXECUTOR_WORKER_TYPE constant and
replace 10 literal "executor" occurrences — fixes S1192
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve SonarCloud cognitive complexity and code smell violations
- Reduce cognitive complexity in answer_prompt.py:
- Extract _build_grammar_notes, _run_webhook_postprocess helpers
- _is_safe_public_url: extracted _resolve_host_addresses helper
- handle_json: early-return pattern eliminates nesting
- construct_prompt: delegates grammar loop to _build_grammar_notes
- Reduce cognitive complexity in legacy_executor.py:
- Extract _execute_single_prompt, _run_table_extraction helpers
- Extract _run_challenge_if_enabled, _run_evaluation_if_enabled
- Extract _inject_table_settings, _finalize_pipeline_result
- Extract _convert_number_answer, _convert_scalar_answer
- Extract _sanitize_dict_values helper
- _handle_answer_prompt CC reduced from 50 to ~7
- Reduce CC in structure_tool_task.py: guard-clause refactor
- Reduce CC in backend: dto.py, deployment_helper.py,
api_deployment_views.py, prompt_studio_helper.py
- Fix S117: rename PascalCase local vars in test_answer_prompt.py
- Fix S1192: extract EXECUTOR_WORKER_TYPE constant in run-worker.sh
- Fix S1172: remove unused params from structure_tool_task.py
- Fix S5713: remove redundant JSONDecodeError in json_repair_helper.py
- Fix S112/S5727 in test_execution.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: remove unused RetrievalStrategy import from _handle_answer_prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: rename UsageHelper params to lowercase (N803)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* UN-3266 fix: resolve remaining SonarCloud issues from check run 66691002192
- Add @staticmethod to _sanitize_null_values (fixes S2325 missing self)
- Reduce _execute_single_prompt params from 25 to 11 (S107)
by grouping services as deps tuple and extracting exec params
from context.executor_params
- Add NOSONAR suppression for raise exc in test helper (S112)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-3266 fix: remove unused locals in _handle_answer_prompt (F841)
execution_id, file_hash, log_events_id, custom_data are now extracted
inside _execute_single_prompt from context.executor_params.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve Biome linting errors in frontend source files
Auto-fixed 48 lint errors across 56 files: import ordering, block
statements, unused variable prefixing, and formatting issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace dynamic import of SharePermission with static import in Workflows
Resolves vite build warning about SharePermission.jsx being both
dynamically and statically imported across the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve SonarCloud warnings in frontend components
- Remove unnecessary try-catch around PostHog event calls
- Flip negated condition in PromptOutput.handleTable for clarity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR #1849 review comments: fix null guards, dead code, and test drift
- Remove redundant inline `import uuid as _uuid` in views.py (use module-level uuid)
- URL-encode DB_USER in worker_celery.py result backend connection string
- Remove misleading task_queues=[Queue("executor")] from dispatch-only Celery app
- Remove dead `if not tool:` guards after objects.get() (already raises DoesNotExist)
- Move profile_manager/default_profile null checks before first dereference
- Reorder ProfileManager.objects.get before mark_document_indexed in tasks.py
- Handle ProfileManager.DoesNotExist as warning, not hard failure
- Wrap PostHog analytics in try/catch so failures don't block prompt execution
- Handle pending-indexing 200 response in usePromptRun.js (clear RUNNING status)
- Reset formData when metadata is missing in ConfigureDs.jsx
- Fix test_should_skip_extraction tests: function now takes 1 arg (outputs only)
- Fix agentic routing tests: mock X2Text.process, remove stale platform_helper kwarg
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing llm_usage_reason for summarize LLM usage tracking
Add PSKeys.LLM_USAGE_REASON to usage_kwargs in _handle_summarize() so
summarization costs appear under summarize_llm in API response metadata.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* UN-3266 [FIX] Fix single-pass extraction routing in LegacyExecutor
- Route _handle_structure_pipeline to _handle_single_pass_extraction when
is_single_pass=True (was always calling _handle_answer_prompt)
- Delegate _handle_single_pass_extraction to cloud plugin via ExecutorRegistry,
falling back to _handle_answer_prompt if plugin not installed
Co-Authored-By: Claude Opus 4.6 <norep…



What
celery_executor_agenticqueue to the executor worker so it can process agentic prompt studio execution tasks_sanitize_null_valuesto preserve top-level "NA" strings instead of converting them toNoneline_itemoutput type from prompt studio select choicesCELERY_QUEUES_EXECUTORenv var instead of hardcodingWhy
celery_executor_agenticqueue, but no worker was consuming from it, causing agentic prompt studio executions to hang indefinitelyNone, which changed the semantics of the extraction output and caused incorrect downstream behaviorline_itemoutput type was removed in favor oftable, and the stale entry needed cleanupHow
EXECUTOR_AGENTICtoQueueNameenum, registered it as an additional queue inWorkerRegistryfor the executor worker type, and updateddocker-compose.yamlto include bothcelery_executor_legacyandcelery_executor_agenticin the defaultCELERY_QUEUES_EXECUTORenv varworker.pyto read queues dynamically from theCELERY_QUEUES_EXECUTORenvironment variable instead of returning a hardcoded list_sanitize_null_valuesto only sanitize "NA" values inside nested lists, preserving top-level "NA" strings as-isanswer_prompt_svc.run_completiondirectly instead of the_convert_scalar_answerhelpertest_answer_prompt.pyto reflect the new NA preservation behaviorCan this PR break any existing features? If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
celery_executor_agentic). Existing legacy queue processing is unaffected since the primary queue remainscelery_executor_legacy.None. This is an intentional fix — downstream consumers that relied onNonefor "NA" answers may need to handle the string "NA" instead. Nested list "NA" values are still sanitized toNone.line_itemoutput type: Any existing prompts configured withline_itemoutput type would no longer have that option. This should be safe asline_itemhas been superseded bytable.Database Migrations
Env Config
NA
Relevant Docs
Related Issues or PRs
Dependencies Versions
Notes on Testing
workers/tests/test_answer_prompt.pyto validate the new NA preservation behaviorScreenshots
N/A
Checklist
I have read and understood the Contribution Guidelines.