Skip to content

Conversation

@tjx666
Copy link
Member

@tjx666 tjx666 commented Aug 31, 2025

💻 变更类型 | Change Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 👷 build
  • ⚡️ perf
  • ✅ test
  • 📝 docs
  • 🔨 chore

🔀 变更说明 | Description of Change

Fix compatibility issue where historical data stored complete URLs instead of keys in file services, causing expired presigned URLs and URL concatenation errors.

Key Changes:

  • Add URL detection and key extraction for both S3 and local file services
  • Extract shared utility function with comprehensive test coverage (13 new tests)
  • Maintain full backward compatibility for existing key-based data
  • Handle both http:// and https:// protocols for legacy data

Problem Solved:

  • Fixes thumbnail access failures after 2-hour presigned URL expiration
  • Prevents URL concatenation errors when full URLs are stored in database
  • Ensures consistent behavior across S3 cloud storage and Desktop local storage

📝 补充信息 | Additional Information

Closes #8994

Testing:

  • 57 total tests passing (13 new utility tests + existing file service tests)
  • Comprehensive edge case coverage including empty strings, various protocols, and malformed URLs
  • Both S3 and Desktop local environments tested

Backward Compatibility:

  • Zero breaking changes - existing key-based data continues working seamlessly
  • Only processes inputs starting with http:// or https:// for legacy compatibility
  • Normal key inputs bypass additional processing for optimal performance

Summary by Sourcery

Fix legacy full URL handling in file services and image router by extracting keys from URLs before generating or storing file paths, preserving backward compatibility and avoiding expired or malformed URLs

Bug Fixes:

  • Handle legacy full URLs in S3 and local file services by extracting keys to prevent expired presigned URL issues and URL concatenation errors

Enhancements:

  • Introduce extractKeyFromUrlOrReturnOriginal utility for shared legacy URL detection
  • Normalize single imageUrl parameter in image router by extracting keys from full URLs before database storage

Tests:

  • Add tests for legacy URL compatibility in S3 and local file service implementations and utility function

Fix compatibility issue where historical data stored complete URLs
instead of keys, causing expired presigned URLs and concatenation errors.

- Add URL detection and key extraction for both S3 and local services
- Extract shared utility function with comprehensive tests
- Maintain backward compatibility for existing key-based data

Closes #8994
@vercel
Copy link

vercel bot commented Aug 31, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
lobe-chat-database Ready Ready Preview Comment Sep 1, 2025 3:39am
lobe-chat-preview Ready Ready Preview Comment Sep 1, 2025 3:39am

@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Aug 31, 2025

Reviewer's Guide

This PR addresses legacy data compatibility by extracting S3/local file keys from stored full URLs before generating new URLs, using a shared utility, integrating it into file service implementations and the image router, and adding comprehensive tests to cover both normal and URL-based inputs without breaking existing key-based behavior.

Sequence diagram for file key extraction in getFullFileUrl (S3 and Local)

sequenceDiagram
  participant Caller
  participant FileServiceImpl
  participant extractKeyFromUrlOrReturnOriginal
  participant getKeyFromFullUrl
  participant getLocalFileUrl
  participant createPreSignedUrlForPreview

  Caller->>FileServiceImpl: getFullFileUrl(url)
  FileServiceImpl->>extractKeyFromUrlOrReturnOriginal: extractKeyFromUrlOrReturnOriginal(url, getKeyFromFullUrl)
  alt url starts with http(s)://
    extractKeyFromUrlOrReturnOriginal->>getKeyFromFullUrl: getKeyFromFullUrl(url)
    getKeyFromFullUrl-->>extractKeyFromUrlOrReturnOriginal: key
    extractKeyFromUrlOrReturnOriginal-->>FileServiceImpl: key
  else url is already a key
    extractKeyFromUrlOrReturnOriginal-->>FileServiceImpl: url
  end
  alt S3StaticFileImpl
    FileServiceImpl->>createPreSignedUrlForPreview: createPreSignedUrlForPreview(key, expiresIn)
    createPreSignedUrlForPreview-->>FileServiceImpl: presignedUrl
    FileServiceImpl-->>Caller: presignedUrl
  else DesktopLocalFileImpl
    FileServiceImpl->>getLocalFileUrl: getLocalFileUrl(key)
    getLocalFileUrl-->>FileServiceImpl: localFileUrl
    FileServiceImpl-->>Caller: localFileUrl
  end
Loading

Sequence diagram for imageRouter storing normalized image URLs

sequenceDiagram
  participant imageRouter
  participant fileService
  participant extractKeyFromUrlOrReturnOriginal
  participant getKeyFromFullUrl
  participant Database

  imageRouter->>fileService: getKeyFromFullUrl(imageUrl)
  fileService->>extractKeyFromUrlOrReturnOriginal: extractKeyFromUrlOrReturnOriginal(imageUrl, getKeyFromFullUrl)
  alt imageUrl starts with http(s)://
    extractKeyFromUrlOrReturnOriginal->>getKeyFromFullUrl: getKeyFromFullUrl(imageUrl)
    getKeyFromFullUrl-->>extractKeyFromUrlOrReturnOriginal: key
    extractKeyFromUrlOrReturnOriginal-->>fileService: key
  else imageUrl is already a key
    extractKeyFromUrlOrReturnOriginal-->>fileService: imageUrl
  end
  fileService-->>imageRouter: key
  imageRouter->>Database: store key as imageUrl
  Database-->>imageRouter: ack
Loading

Class diagram for updated file service implementations and utility

classDiagram
  class S3StaticFileImpl {
    +getFullFileUrl(url, expiresIn)
    +getKeyFromFullUrl(url)
    -createPreSignedUrlForPreview(key, expiresIn)
  }
  class DesktopLocalFileImpl {
    +getFullFileUrl(url)
    +getKeyFromFullUrl(url)
    -getLocalFileUrl(key)
  }
  class extractKeyFromUrlOrReturnOriginal {
    +extractKeyFromUrlOrReturnOriginal(url, getKeyFromFullUrl)
  }
  S3StaticFileImpl --> extractKeyFromUrlOrReturnOriginal : uses
  DesktopLocalFileImpl --> extractKeyFromUrlOrReturnOriginal : uses
Loading

File-Level Changes

Change Details Files
Introduce shared utility for legacy URL handling
  • Implement extractKeyFromUrlOrReturnOriginal to detect http/https prefixes
  • Delegate full URL key extraction to getKeyFromFullUrl when needed
  • Add unit tests covering edge cases and protocols
src/server/services/file/impls/utils.ts
src/server/services/file/impls/utils.test.ts
Integrate legacy handling into file services
  • Wrap original URL input in extractKeyFromUrlOrReturnOriginal in S3StaticFileImpl.getFullFileUrl
  • Apply same extraction logic in DesktopLocalFileImpl.getFullFileUrl
  • Preserve original key handling path for non-URL inputs
src/server/services/file/impls/s3.ts
src/server/services/file/impls/local.ts
Add compatibility tests in service implementations
  • Test S3StaticFileImpl.getFullFileUrl with expired presigned URLs and ACL variations
  • Cover http vs https full URLs and normal key inputs
  • Validate DesktopLocalFileImpl.getFullFileUrl behavior with desktop:// keys and URLs
src/server/services/file/impls/s3.test.ts
src/server/services/file/impls/local.test.ts
Normalize image URL inputs in Lambda router
  • Convert each imageUrls entry to key before DB storage
  • Add conversion for single imageUrl using getKeyFromFullUrl
  • Retain original URL on extraction failure
src/server/routers/lambda/image.ts

Assessment against linked issues

Issue Objective Addressed Explanation
#8994 Fix the bug where uploaded image thumbnails in the AI绘画 chat history become inaccessible after the presigned S3 URL expires (resulting in 403 Forbidden).
#8994 Ensure that image references stored in the database use S3 keys rather than full presigned URLs, to prevent future thumbnail access failures due to expired URLs.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Aug 31, 2025
@lobehubbot
Copy link
Member

👍 @tjx666

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献,请确保您已经遵循了我们的贡献指南,我们会尽快审查它。
如果您遇到任何问题,请随时与我们联系。

@gru-agent
Copy link
Contributor

gru-agent bot commented Aug 31, 2025

TestGru Assignment

Summary

Link CommitId Status Reason
Detail 6c627bb ✅ Finished

History Assignment

Files

File Pull Request
src/server/routers/lambda/image.ts ❌ Failed (I failed to setup the environment.)
src/server/services/file/impls/local.ts ❌ Failed (I failed to setup the environment.)
src/server/services/file/impls/utils.ts ❌ Failed (I failed to setup the environment.)
src/server/services/file/impls/s3.ts ❌ Failed (I failed to setup the environment.)

Tip

You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

@dosubot dosubot bot added the 🐛 Bug label Aug 31, 2025
@codecov
Copy link

codecov bot commented Aug 31, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.17%. Comparing base (a6b2a19) to head (871fd1e).
⚠️ Report is 16 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##             main    #9016     +/-   ##
=========================================
  Coverage   77.17%   77.17%             
=========================================
  Files         810      818      +8     
  Lines       48848    49087    +239     
  Branches     5226     6381   +1155     
=========================================
+ Hits        37696    37883    +187     
- Misses      11146    11198     +52     
  Partials        6        6             
Flag Coverage Δ
app 77.16% <100.00%> (+0.11%) ⬆️
database 96.26% <ø> (ø)
packages/electron-server-ipc 74.61% <ø> (ø)
packages/file-loaders 83.59% <ø> (ø)
packages/model-bank 100.00% <100.00%> (ø)
packages/model-runtime 74.66% <ø> (-0.42%) ⬇️
packages/prompts 100.00% <ø> (ø)
packages/utils 60.47% <ø> (ø)
packages/web-crawler 59.57% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Store 68.63% <ø> (ø)
Services 61.97% <ø> (+0.12%) ⬆️
Server 66.39% <100.00%> (+0.05%) ⬆️
Libs 42.20% <ø> (+0.09%) ⬆️
Utils 73.57% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments

### Comment 1
<location> `src/server/routers/lambda/image.ts:97` </location>
<code_context>
       }
     }
+    // 2) 处理单图 imageUrl
+    if (typeof params.imageUrl === 'string' && params.imageUrl) {
+      try {
+        const key = fileService.getKeyFromFullUrl(params.imageUrl);
+        log('Converted single imageUrl to key: %s -> %s', params.imageUrl, key);
+        configForDatabase = { ...configForDatabase, imageUrl: key };
+      } catch (error) {
+        log('Error converting imageUrl to key: %O', error);
</code_context>

<issue_to_address>
Consider using the shared utility for key extraction for consistency.

Use extractKeyFromUrlOrReturnOriginal for key extraction to align with other usages and improve maintainability.

Suggested implementation:

```typescript
        const key = extractKeyFromUrlOrReturnOriginal(params.imageUrl);
        log('Converted single imageUrl to key using shared utility: %s -> %s', params.imageUrl, key);
        configForDatabase = { ...configForDatabase, imageUrl: key };

```

If `extractKeyFromUrlOrReturnOriginal` is not already imported in this file, add the following import statement at the top of the file (adjust the import path as needed for your project structure):

```typescript
import { extractKeyFromUrlOrReturnOriginal } from 'shared/utils/fileKeyUtils';
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@tjx666 tjx666 changed the title 🐛 fix(file): handle legacy full URLs in file services 🐛 fix(ai-image): save config.imageUrl with fullUrl instead of key Aug 31, 2025
tjx666 added 2 commits August 31, 2025 20:59
Refactor the path for aiModels in the VSCode settings to simplify the structure. This change enhances the organization of model files within the project.
Add recursive validation to prevent full URLs from being stored in database:
- Validate configForDatabase before transaction to ensure only keys are stored
- Comprehensive error reporting with precise field path location
- Add 9 test cases covering valid/invalid configs and edge cases

Prevents data corruption from accidentally storing complete URLs instead of storage keys
@tjx666 tjx666 merged commit bad009a into main Sep 1, 2025
38 checks passed
@tjx666 tjx666 deleted the tj/feat/image-url branch September 1, 2025 03:59
@lobehubbot
Copy link
Member

❤️ Great PR @tjx666 ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献,感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣,请加入我们的 discord,然后私信 @arvinxx@canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发,分享和讨论全球范围内的 AI 消息。

lobehubbot pushed a commit that referenced this pull request Sep 1, 2025
### [Version&nbsp;1.120.5](v1.120.4...v1.120.5)
<sup>Released on **2025-09-01**</sup>

#### 🐛 Bug Fixes

- **ai-image**: Save config.imageUrl with fullUrl instead of key.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's fixed

* **ai-image**: Save config.imageUrl with fullUrl instead of key, closes [#9016](#9016) ([bad009a](bad009a))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
@lobehubbot
Copy link
Member

🎉 This PR is included in version 1.120.5 🎉

The release is available on:

Your semantic-release bot 📦🚀

JamieStivala pushed a commit to jaworldwideorg/OneJA-Bot that referenced this pull request Sep 2, 2025
## [Version&nbsp;1.120.0](v1.119.1...v1.120.0)
<sup>Released on **2025-09-02**</sup>

#### ♻ Code Refactoring

- **model-runtime**: Refactor model-runtime dependencies and clean code.
- **misc**: Remove base path, remove webrtc sync feature flag.

#### ✨ Features

- **misc**: Added support for Azure OpenAI Image Generation, rename Gemini 2.5 flash image to Nano Banana.

#### 🐛 Bug Fixes

- **ai-image**: Save config.imageUrl with fullUrl instead of key.
- **misc**: Update enableStreaming name.

#### 💄 Styles

- **misc**: Add upload hint for non-visual model, adjust ControlsForm component to adapt to mobile phone display, Support new provider Nebius, update i18n, update i18n.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### Code refactoring

* **model-runtime**: Refactor model-runtime dependencies and clean code, closes [lobehub#8997](https://github.com/jaworldwideorg/OneJA-Bot/issues/8997) ([9f7677d](9f7677d))
* **misc**: Remove base path, closes [lobehub#9015](https://github.com/jaworldwideorg/OneJA-Bot/issues/9015) ([2a5f8d7](2a5f8d7))
* **misc**: Remove webrtc sync feature flag, closes [lobehub#9002](https://github.com/jaworldwideorg/OneJA-Bot/issues/9002) ([0924d98](0924d98))

#### What's improved

* **misc**: Added support for Azure OpenAI Image Generation, closes [lobehub#8898](https://github.com/jaworldwideorg/OneJA-Bot/issues/8898) ([6042340](6042340))
* **misc**: Rename Gemini 2.5 flash image to Nano Banana, closes [lobehub#9004](https://github.com/jaworldwideorg/OneJA-Bot/issues/9004) ([dac5a6f](dac5a6f))

#### What's fixed

* **ai-image**: Save config.imageUrl with fullUrl instead of key, closes [lobehub#9016](https://github.com/jaworldwideorg/OneJA-Bot/issues/9016) ([bad009a](bad009a))
* **misc**: Update enableStreaming name, closes [lobehub#8995](https://github.com/jaworldwideorg/OneJA-Bot/issues/8995) ([7c7de40](7c7de40))

#### Styles

* **misc**: Add upload hint for non-visual model, closes [lobehub#7969](https://github.com/jaworldwideorg/OneJA-Bot/issues/7969) ([1224f4e](1224f4e))
* **misc**: Adjust ControlsForm component to adapt to mobile phone display, closes [lobehub#9013](https://github.com/jaworldwideorg/OneJA-Bot/issues/9013) ([c6038c0](c6038c0))
* **misc**: Support new provider Nebius, closes [lobehub#8903](https://github.com/jaworldwideorg/OneJA-Bot/issues/8903) ([c15791d](c15791d))
* **misc**: Update i18n, closes [lobehub#9033](https://github.com/jaworldwideorg/OneJA-Bot/issues/9033) ([650e552](650e552))
* **misc**: Update i18n, closes [lobehub#9005](https://github.com/jaworldwideorg/OneJA-Bot/issues/9005) ([63760f9](63760f9))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
leovvay pushed a commit to kissMyApps-tlm/kissmychat that referenced this pull request Sep 2, 2025
leovvay pushed a commit to kissMyApps-tlm/kissmychat that referenced this pull request Sep 2, 2025
### [Version&nbsp;1.120.5](lobehub/lobe-chat@v1.120.4...v1.120.5)
<sup>Released on **2025-09-01**</sup>

#### 🐛 Bug Fixes

- **ai-image**: Save config.imageUrl with fullUrl instead of key.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's fixed

* **ai-image**: Save config.imageUrl with fullUrl instead of key, closes [lobehub#9016](lobehub#9016) ([bad009a](lobehub@bad009a))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI绘画上传图片在聊天预览图失效

3 participants