diff --git a/TRANSLATION-ANDROID-PORT.md b/TRANSLATION-ANDROID-PORT.md new file mode 100644 index 00000000000..46c1887f7be --- /dev/null +++ b/TRANSLATION-ANDROID-PORT.md @@ -0,0 +1,492 @@ +# Element Android — LLM Translation System Porting Guide + +This document describes the complete translation system implemented in Element Desktop and how to port it to Element Android (or Element X Android). Give this file to an AI assistant working on the Android project. + +--- + +## Overview + +Element Desktop now has a real-time LLM translation system using any OpenAI-compatible API (Ollama, OpenAI, LM Studio, LocalAI, etc.). The goal is to replicate the same features on Android. + +### Features to implement + +| Feature | Description | +|---------|-------------| +| **Auto-translate incoming** | All received messages are translated to the user's language automatically | +| **Translate button per message** | A "Translate" button appears on each message (on long-press or as an icon) | +| **Reply quote translation** | Quoted messages in replies are also translated | +| **Selection translation** | Selected text can be translated on demand | +| **Notification translation** | Incoming notifications are translated before display | +| **Outgoing message translation** | Messages written in user's language are translated to the room's language before sending | +| **Persistent cache** | Translations are cached in memory + on disk, surviving app restarts | +| **Rate limiting** | Max 3 concurrent API requests, others queued | +| **Settings UI** | Full settings screen with API config, toggles, test button, cache management | +| **Keyboard/shortcut toggle** | Quick toggle for auto-translate | + +--- + +## 1. Configuration + +### Data model + +```kotlin +data class TranslateConfig( + val apiUrl: String = "http://localhost:11434/v1", // or user's server + val apiKey: String = "", // Bearer token, empty for local models + val model: String = "llama3", // LLM model name + val targetLanguage: String = "French", // User's language (read messages in this) + val roomLanguage: String = "", // Room's language (send messages in this, empty = disabled) + val enabled: Boolean = false, // Master toggle + val autoTranslate: Boolean = false // Auto-translate all incoming messages +) +``` + +### Storage + +Use `SharedPreferences` or Jetpack `DataStore` with keys: +- `translate_api_url` +- `translate_api_key` +- `translate_model` +- `translate_target_language` +- `translate_room_language` +- `translate_enabled` +- `translate_auto_translate` + +--- + +## 2. Translation API + +### Endpoint: Chat Completions (OpenAI-compatible) + +``` +POST {apiUrl}/chat/completions +``` + +**Headers:** +``` +Content-Type: application/json +Authorization: Bearer {apiKey} // only if apiKey is not empty +``` + +**Request body:** +```json +{ + "model": "llama3", + "messages": [ + { + "role": "system", + "content": "You are a translator. Translate the following message to {targetLanguage}. Reply ONLY with the translation, no explanations, no quotes, no extra text. If the text is already in {targetLanguage}, return it as-is. Preserve any formatting, markdown, links, and mentions exactly as they are." + }, + { + "role": "user", + "content": "{text to translate}" + } + ], + "temperature": 0.1, + "max_tokens": 2048 +} +``` + +**Response (extract `choices[0].message.content`):** +```json +{ + "choices": [ + { + "message": { + "content": "translated text here" + } + } + ] +} +``` + +### Endpoint: Test Connection + +``` +GET {apiUrl}/models +``` + +Returns available models. Use to validate URL + API key. + +### Implementation (Kotlin/Retrofit) + +```kotlin +interface TranslateApi { + @POST("chat/completions") + suspend fun translate(@Body request: ChatCompletionRequest): ChatCompletionResponse + + @GET("models") + suspend fun listModels(): ModelsResponse +} + +data class ChatCompletionRequest( + val model: String, + val messages: List, + val temperature: Double = 0.1, + val max_tokens: Int = 2048 +) + +data class Message(val role: String, val content: String) + +data class ChatCompletionResponse(val choices: List) +data class Choice(val message: Message) + +data class ModelsResponse(val data: List) +data class ModelInfo(val id: String) +``` + +--- + +## 3. Cache System (3 levels) + +### Level 1: In-memory cache (fastest, per-session) + +```kotlin +object TranslationCache { + private val memoryCache = LinkedHashMap(5000, 0.75f, true) + private const val MAX_SIZE = 5000 + + fun key(text: String, lang: String) = "$lang:$text" + + fun get(text: String, lang: String): String? = memoryCache[key(text, lang)] + + fun put(text: String, lang: String, translated: String) { + if (memoryCache.size >= MAX_SIZE) { + memoryCache.remove(memoryCache.keys.first()) + } + memoryCache[key(text, lang)] = translated + } + + fun clear() = memoryCache.clear() + fun size() = memoryCache.size +} +``` + +### Level 2: Disk cache (survives app restarts) + +Use a JSON file or Room database: + +```kotlin +// File-based (simple): +// Location: context.filesDir / "translate-cache.json" +// Format: JSON array of [key, value] pairs +// Save debounced every 2 seconds after last write +// Load on app startup + +// Or Room database (more robust): +@Entity(tableName = "translation_cache") +data class CachedTranslation( + @PrimaryKey val cacheKey: String, // "French:Hello world" + val translatedText: String, + val timestamp: Long = System.currentTimeMillis() +) +``` + +### Cache flow + +``` +translate(text, lang) → + 1. Check memory cache → hit? return immediately + 2. Check disk cache → hit? add to memory, return + 3. Call API → store in memory + schedule disk save +``` + +--- + +## 4. Rate Limiting + +Max 3 concurrent API calls, queue the rest: + +```kotlin +class TranslationRateLimiter(private val maxConcurrent: Int = 3) { + private val semaphore = Semaphore(maxConcurrent) + + suspend fun execute(block: suspend () -> T): T { + semaphore.acquire() + return try { + block() + } finally { + semaphore.release() + } + } +} +``` + +--- + +## 5. Auto-translate incoming messages + +### Where to hook + +In the message list adapter/composable, when binding a message item: + +```kotlin +// Pseudocode for RecyclerView adapter or Compose LazyColumn +fun bindMessage(event: TimelineEvent) { + val body = event.root.getClearContent()?.get("body") as? String ?: return + + if (config.enabled && config.autoTranslate) { + // Check cache first (instant) + val cached = TranslationCache.get(body, config.targetLanguage) + if (cached != null) { + showTranslated(cached, originalText = body) + return + } + // Translate async + scope.launch { + showTranslating() // show "..." indicator + val translated = translateService.translate(body, config.targetLanguage) + showTranslated(translated, originalText = body) + } + } +} +``` + +### UI for translated messages + +- Show translated text as the main content +- Add a collapsible "Original" section below (like a `
` in HTML) +- Or show original on long-press + +--- + +## 6. Translate button per message + +### Option A: Action in message long-press menu + +Add "Translate" to the existing long-press context menu on messages. This is the simplest approach. + +### Option B: Floating button (like reactions) + +Add a small translate icon button that appears near the message actions (reply, react, etc.). + +### Behavior + +1. User taps "Translate" +2. Show loading indicator on message +3. Call translation API +4. Replace message text with translation +5. Show "Original" toggle below + +--- + +## 7. Reply quote translation + +When a message contains a reply (quoted message), also translate the quoted text. + +In Element Android, replies have a `RelatesTo` field with the original event. When rendering the reply preview: + +```kotlin +fun bindReplyPreview(repliedEvent: TimelineEvent) { + val replyBody = repliedEvent.root.getClearContent()?.get("body") as? String + if (replyBody != null && config.enabled) { + val translated = TranslationCache.get(replyBody, config.targetLanguage) + ?: translateService.translate(replyBody, config.targetLanguage) + showReplyBody(translated) + } +} +``` + +--- + +## 8. Notification translation + +### Where to hook + +Intercept notification creation. In Element Android, notifications are built in a `NotificationFactory` or similar class. + +```kotlin +// Before showing the notification: +suspend fun translateNotification(title: String, body: String): Pair { + if (!config.enabled) return title to body + + val translatedTitle = translateService.translate(title, config.targetLanguage) + val translatedBody = translateService.translate(body, config.targetLanguage) + return (translatedTitle ?: title) to (translatedBody ?: body) +} + +// In notification builder: +val (title, body) = translateNotification(originalTitle, originalBody) +builder.setContentTitle(title) +builder.setContentText(body) +``` + +--- + +## 9. Outgoing message translation + +### Concept + +User writes in French → message is translated to English before being sent to the Matrix server → auto-translate shows it back in French locally. + +### Where to hook + +Intercept the message send function in the Matrix SDK: + +```kotlin +// Before sending: +suspend fun translateOutgoing(messageText: String): String { + val roomLang = config.roomLanguage + if (!config.enabled || roomLang.isBlank()) return messageText + + return translateService.translate(messageText, roomLang) ?: messageText +} + +// In the send message flow: +fun sendMessage(roomId: String, text: String) { + scope.launch { + val translatedText = translateOutgoing(text) + // Send translatedText to the room + room.sendTextMessage(translatedText) + // The auto-translate will show it back in French to the user + } +} +``` + +### Important + +- Only translate `m.text` message types +- Also translate `formatted_body` if present (HTML formatted messages) +- Don't translate media, stickers, or other message types + +--- + +## 10. Settings UI + +### Android Settings Screen + +Create a new settings fragment/composable accessible from the app's settings menu. + +**Fields:** + +| Field | Type | Description | +|-------|------|-------------| +| Enable Translation | Switch | Master toggle | +| Auto-translate all messages | Switch | Translate everything automatically | +| API URL | EditText | URL of the OpenAI-compatible API | +| Test Connection | Button | Tests the URL, shows available models | +| API Key | EditText (password) | Bearer token, optional for local models | +| Model | EditText | Model name (e.g., llama3, gpt-4o-mini) | +| Target Language | EditText | User's language (e.g., "French") | +| Room Language | EditText | Room's language for outgoing translation (e.g., "English"), empty = disabled | +| Cache info | Text | "X translations cached" | +| Clear Cache | Button | Empties all caches | + +### Test Connection button + +```kotlin +btnTest.setOnClickListener { + scope.launch { + try { + showLoading() + val result = api.listModels() + showSuccess("Connected! Models: ${result.data.joinToString { it.id }}") + } catch (e: Exception) { + showError("Failed: ${e.message}") + } + } +} +``` + +--- + +## 11. Quick toggle + +### Option A: Quick Settings Tile + +Register an Android Quick Settings tile that toggles auto-translate on/off. + +### Option B: Notification action + +Add a persistent notification with a "Toggle Translation" action button. + +### Option C: In-app shortcut + +Add a floating action button or toolbar icon in the chat screen to toggle. + +--- + +## 12. Architecture summary + +``` +┌─────────────────────────────────────────────┐ +│ Settings UI │ +│ (PreferenceFragment / Compose Screen) │ +└────────────────┬────────────────────────────┘ + │ reads/writes + ▼ +┌─────────────────────────────────────────────┐ +│ TranslateConfig │ +│ (SharedPreferences/DataStore) │ +└────────────────┬────────────────────────────┘ + │ used by + ▼ +┌─────────────────────────────────────────────┐ +│ TranslationService │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Memory │→ │ Disk │→ │ API │ │ +│ │ Cache │ │ Cache │ │ Call │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +│ ┌──────────────────────────────────────┐ │ +│ │ Rate Limiter (3 max) │ │ +│ └──────────────────────────────────────┘ │ +└────────────────┬────────────────────────────┘ + │ called by + ┌────────────┼────────────────┐ + ▼ ▼ ▼ +┌────────┐ ┌──────────┐ ┌──────────────┐ +│ Message│ │Notifica- │ │ Outgoing │ +│ List │ │tion │ │ Message │ +│Adapter │ │Builder │ │ Interceptor │ +└────────┘ └──────────┘ └──────────────┘ +``` + +--- + +## 13. Key differences from Desktop + +| Aspect | Desktop (Electron) | Android | +|--------|-------------------|---------| +| Message interception | DOM MutationObserver + injected JS | RecyclerView Adapter / Compose state | +| Outgoing interception | `fetch()` monkey-patch | SDK message send interceptor | +| Notification | `window.Notification` override | `NotificationCompat.Builder` hook | +| Cache storage | JSON file in userData | Room DB or JSON in filesDir | +| Settings | Separate BrowserWindow + HTML | PreferenceFragment or Compose | +| Rate limiting | Promise queue | Kotlin Coroutine Semaphore | +| API calls | Electron `net.fetch` | Retrofit/OkHttp | +| Toggle shortcut | Ctrl+Shift+T (keyboard) | Quick Settings tile or FAB | + +--- + +## 14. API compatibility + +This system works with any OpenAI-compatible API: + +| Provider | API URL | API Key | Notes | +|----------|---------|---------|-------| +| **Ollama** (local) | `http://localhost:11434/v1` | empty | Free, runs on device or local server | +| **OpenAI** | `https://api.openai.com/v1` | `sk-...` | Paid, best quality | +| **LM Studio** | `http://localhost:1234/v1` | empty | Free, local | +| **LocalAI** | `http://localhost:8080/v1` | empty | Free, local | +| **Groq** | `https://api.groq.com/openai/v1` | `gsk-...` | Free tier, very fast | +| **Mistral** | `https://api.mistral.ai/v1` | `...` | Paid | +| **Any OpenAI-compatible** | varies | varies | Any server implementing the chat completions API | + +For Android, the user would typically point to a remote server (not localhost) unless they're on the same network as their local LLM. + +--- + +## 15. System prompt (do not modify) + +The translation prompt is carefully crafted. Use it exactly: + +``` +You are a translator. Translate the following message to {language}. Reply ONLY with the translation, no explanations, no quotes, no extra text. If the text is already in {language}, return it as-is. Preserve any formatting, markdown, links, and mentions exactly as they are. +``` + +This ensures: +- No extra text in the response +- No wrapping in quotes +- Preserves markdown, links, @mentions +- Doesn't re-translate text already in the target language diff --git a/gradle.properties b/gradle.properties index 2433ae78cbd..ff2334305b1 100644 --- a/gradle.properties +++ b/gradle.properties @@ -8,7 +8,7 @@ # The setting is particularly useful for tweaking memory settings. # Build Time Optimizations -org.gradle.jvmargs=-Xmx4g -Xms512M -XX:MaxPermSize=2048m -XX:MaxMetaspaceSize=1g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -XX:+UseParallelGC +org.gradle.jvmargs=-Xmx4g -Xms512M -XX:MaxMetaspaceSize=1g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -XX:+UseParallelGC org.gradle.configureondemand=true org.gradle.parallel=true org.gradle.vfs.watch=true diff --git a/vector/build.gradle b/vector/build.gradle index 5f61566f44a..4c693ac31c9 100644 --- a/vector/build.gradle +++ b/vector/build.gradle @@ -140,12 +140,17 @@ dependencies { implementation "androidx.media:media:1.6.0" implementation "androidx.transition:transition:1.4.1" implementation libs.androidx.biometric + implementation "androidx.security:security-crypto:1.1.0-alpha06" api "org.threeten:threetenbp:1.4.0:no-tzdb" api "com.gabrielittner.threetenbp:lazythreetenbp:0.16.0" implementation libs.squareup.moshi kapt libs.squareup.moshiKotlin + implementation libs.squareup.retrofit + implementation libs.squareup.retrofitMoshi + implementation platform("com.squareup.okhttp3:okhttp-bom:4.11.0") + implementation 'com.squareup.okhttp3:okhttp' // Lifecycle implementation libs.androidx.lifecycleLivedata diff --git a/vector/src/main/AndroidManifest.xml b/vector/src/main/AndroidManifest.xml index 1fae87ee2ad..4cb6b4d20bc 100644 --- a/vector/src/main/AndroidManifest.xml +++ b/vector/src/main/AndroidManifest.xml @@ -53,6 +53,10 @@ + + @@ -415,6 +419,21 @@ android:foregroundServiceType="microphone" android:permission="android.permission.FOREGROUND_SERVICE_MICROPHONE" /> + + + + + + + + + + + + = arrayListOf() + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + notificationsData = arguments?.getStringArrayList(ARG_NOTIFICATIONS) ?: arrayListOf() + } + + override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View { + val context = requireContext() + val padding = (16 * context.resources.displayMetrics.density).toInt() + + val root = LinearLayout(context).apply { + orientation = LinearLayout.VERTICAL + setPadding(padding, padding, padding, padding) + } + + val title = TextView(context).apply { + text = getString(R.string.notification_summary_title) + textSize = 18f + setPadding(0, 0, 0, padding) + } + root.addView(title) + + val progressBar = ProgressBar(context).apply { + isIndeterminate = true + } + root.addView(progressBar) + + val statusText = TextView(context).apply { + text = getString(R.string.notification_summary_loading) + } + root.addView(statusText) + + val scrollView = ScrollView(context).apply { + isVisible = false + layoutParams = LinearLayout.LayoutParams( + LinearLayout.LayoutParams.MATCH_PARENT, + (250 * context.resources.displayMetrics.density).toInt() + ) + } + + val summaryText = TextView(context).apply { + textSize = 14f + setPadding(0, padding / 2, 0, padding / 2) + } + scrollView.addView(summaryText) + root.addView(scrollView) + + val buttonsLayout = LinearLayout(context).apply { + orientation = LinearLayout.HORIZONTAL + isVisible = false + } + + val dismissButton = Button(context).apply { + text = getString(R.string.notification_summary_dismiss) + setOnClickListener { + parentFragmentManager.setFragmentResult( + REQUEST_KEY, + bundleOf(RESULT_ACTION to ACTION_DISMISS) + ) + dismissAllowingStateLoss() + } + } + buttonsLayout.addView(dismissButton) + + val markReadButton = Button(context).apply { + text = getString(R.string.notification_summary_mark_read) + setOnClickListener { + parentFragmentManager.setFragmentResult( + REQUEST_KEY, + bundleOf(RESULT_ACTION to ACTION_MARK_READ) + ) + dismissAllowingStateLoss() + } + } + buttonsLayout.addView(markReadButton) + root.addView(buttonsLayout) + + // Parse notifications as pairs of room:message + val notifPairs = notificationsData.map { line -> + val parts = line.split("|", limit = 2) + if (parts.size == 2) parts[0] to parts[1] else "" to line + } + + Timber.d("NotificationSummaryBottomSheet: generating summary for ${notifPairs.size} notifications") + + lifecycleScope.launch { + try { + val result = notificationSummaryService.generateSummary(notifPairs) + progressBar.isVisible = false + if (result != null) { + statusText.isVisible = false + summaryText.text = result + scrollView.isVisible = true + buttonsLayout.isVisible = true + Timber.d("NotificationSummaryBottomSheet: summary generated successfully") + } else { + statusText.text = getString(R.string.notification_summary_failed) + Timber.d("NotificationSummaryBottomSheet: summary generation returned null") + } + } catch (e: Exception) { + Timber.e(e, "NotificationSummaryBottomSheet: summary generation failed") + progressBar.isVisible = false + statusText.text = getString(R.string.notification_summary_failed) + } + } + + return root + } + + companion object { + private const val ARG_NOTIFICATIONS = "arg_notifications" + private const val TAG = "NotificationSummaryBottomSheet" + const val REQUEST_KEY = "notification_summary_result" + const val RESULT_ACTION = "action" + const val ACTION_DISMISS = "dismiss" + const val ACTION_MARK_READ = "mark_read" + + fun show( + fragmentManager: FragmentManager, + notifications: List + ) { + // Avoid showing multiple instances + if (fragmentManager.findFragmentByTag(TAG) != null) return + + val sheet = NotificationSummaryBottomSheet().apply { + arguments = Bundle().apply { putStringArrayList(ARG_NOTIFICATIONS, ArrayList(notifications)) } + } + sheet.show(fragmentManager, TAG) + } + } +} diff --git a/vector/src/main/java/im/vector/app/features/ai/notificationsummary/NotificationSummaryService.kt b/vector/src/main/java/im/vector/app/features/ai/notificationsummary/NotificationSummaryService.kt new file mode 100644 index 00000000000..2452f892451 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/ai/notificationsummary/NotificationSummaryService.kt @@ -0,0 +1,57 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.ai.notificationsummary + +import im.vector.app.features.translation.TimelineTranslationManager +import im.vector.app.features.translation.TranslateConfig +import im.vector.app.features.translation.TranslationService +import timber.log.Timber +import javax.inject.Inject +import javax.inject.Singleton + +@Singleton +class NotificationSummaryService @Inject constructor( + private val translationService: TranslationService, + private val config: TranslateConfig, + private val timelineTranslationManager: TimelineTranslationManager +) { + /** + * Generate a summary of missed notifications using the LLM. + * For each notification, checks if a translation exists in cache and uses it if available. + * The summary is generated in French. + * + * @param notificationTexts list of pairs (roomName, messageBody) + * @return the generated summary, or null if generation failed or feature is disabled + */ + suspend fun generateSummary(notificationTexts: List>): String? { + if (!config.enabled || !config.notificationSummaryEnabled) return null + if (notificationTexts.isEmpty()) return null + + val formatted = notificationTexts.joinToString("\n") { (room, message) -> + // Check if a translation exists in cache for this message body + val translatedMessage = timelineTranslationManager.getCachedTranslation(message) + val textToUse = translatedMessage ?: message + "[$room] $textToUse" + } + + Timber.d("NotificationSummary: generating summary for ${notificationTexts.size} notifications") + + val language = config.targetLanguage + + return try { + translationService.complete( + "Summarize the following missed notifications concisely in $language. " + + "Group by room. Mention direct mentions, decisions, and pending questions.", + formatted + ) + } catch (e: Exception) { + Timber.e(e, "Notification summary generation failed") + null + } + } +} diff --git a/vector/src/main/java/im/vector/app/features/ai/reformulate/ReformulationBottomSheet.kt b/vector/src/main/java/im/vector/app/features/ai/reformulate/ReformulationBottomSheet.kt new file mode 100644 index 00000000000..6920bd86be8 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/ai/reformulate/ReformulationBottomSheet.kt @@ -0,0 +1,119 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.ai.reformulate + +import android.os.Bundle +import android.view.LayoutInflater +import android.view.View +import android.view.ViewGroup +import android.widget.LinearLayout +import android.widget.ProgressBar +import android.widget.TextView +import androidx.core.os.bundleOf +import androidx.core.view.isVisible +import androidx.fragment.app.FragmentManager +import androidx.lifecycle.lifecycleScope +import com.google.android.material.bottomsheet.BottomSheetDialogFragment +import dagger.hilt.android.AndroidEntryPoint +import im.vector.app.R +import im.vector.app.features.translation.TranslationService +import kotlinx.coroutines.launch +import javax.inject.Inject + +@AndroidEntryPoint +class ReformulationBottomSheet : BottomSheetDialogFragment() { + + @Inject lateinit var translationService: TranslationService + + private var originalText: String = "" + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + originalText = arguments?.getString(ARG_TEXT) ?: "" + } + + override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View { + val context = requireContext() + val padding = (16 * context.resources.displayMetrics.density).toInt() + + val root = LinearLayout(context).apply { + orientation = LinearLayout.VERTICAL + setPadding(padding, padding, padding, padding) + } + + val title = TextView(context).apply { + text = getString(R.string.reformulate_title) + textSize = 18f + setPadding(0, 0, 0, padding) + } + root.addView(title) + + val progressBar = ProgressBar(context).apply { + isVisible = false + isIndeterminate = true + } + + val statusText = TextView(context).apply { + isVisible = false + } + + val options = listOf( + getString(R.string.reformulate_formal) to "Rewrite the following message in a formal, professional tone. Reply ONLY with the rewritten message.", + getString(R.string.reformulate_casual) to "Rewrite the following message in a casual, friendly tone. Reply ONLY with the rewritten message.", + getString(R.string.reformulate_concise) to "Rewrite the following message to be more concise and to the point. Reply ONLY with the rewritten message.", + getString(R.string.reformulate_fix_grammar) to "Fix the grammar and spelling of the following message. Reply ONLY with the corrected message.", + ) + + options.forEach { (label, prompt) -> + val button = TextView(context).apply { + text = label + textSize = 16f + setPadding(padding, padding / 2, padding, padding / 2) + setBackgroundResource(android.R.drawable.list_selector_background) + setOnClickListener { + progressBar.isVisible = true + statusText.isVisible = true + statusText.text = getString(R.string.reformulate_loading) + lifecycleScope.launch { + val result = translationService.complete(prompt, originalText) + if (result != null) { + parentFragmentManager.setFragmentResult( + REQUEST_KEY, + bundleOf(RESULT_TEXT to result) + ) + dismissAllowingStateLoss() + } else { + statusText.text = getString(R.string.reformulate_failed) + progressBar.isVisible = false + } + } + } + } + root.addView(button) + } + + root.addView(progressBar) + root.addView(statusText) + + return root + } + + companion object { + private const val ARG_TEXT = "arg_text" + private const val TAG = "ReformulationBottomSheet" + const val REQUEST_KEY = "reformulation_result" + const val RESULT_TEXT = "text" + + fun show(fragmentManager: FragmentManager, text: String) { + val sheet = ReformulationBottomSheet().apply { + arguments = Bundle().apply { putString(ARG_TEXT, text) } + } + sheet.show(fragmentManager, TAG) + } + } +} diff --git a/vector/src/main/java/im/vector/app/features/ai/suggestedreply/SuggestedReplyService.kt b/vector/src/main/java/im/vector/app/features/ai/suggestedreply/SuggestedReplyService.kt new file mode 100644 index 00000000000..0c576035c2e --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/ai/suggestedreply/SuggestedReplyService.kt @@ -0,0 +1,57 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.ai.suggestedreply + +import im.vector.app.features.translation.TranslateConfig +import im.vector.app.features.translation.TranslationService +import org.json.JSONArray +import timber.log.Timber +import javax.inject.Inject +import javax.inject.Singleton + +@Singleton +class SuggestedReplyService @Inject constructor( + private val translationService: TranslationService, + private val config: TranslateConfig +) { + suspend fun getSuggestions(recentMessages: String): List { + if (!config.enabled || !config.suggestedRepliesEnabled) return emptyList() + + val result = translationService.complete( + "You are a helpful assistant that suggests short reply messages. " + + "Given the recent conversation, suggest exactly 3 short, natural replies the user could send. " + + "Reply ONLY with a JSON array of 3 strings, e.g. [\"reply1\", \"reply2\", \"reply3\"]. " + + "Keep each reply under 50 characters. Match the language of the conversation.", + recentMessages + ) ?: return emptyList() + + return parseSuggestions(result) + } + + private fun parseSuggestions(response: String): List { + // Try JSON array first + try { + val jsonArray = JSONArray(response.trim()) + val suggestions = mutableListOf() + for (i in 0 until jsonArray.length()) { + suggestions.add(jsonArray.getString(i)) + } + if (suggestions.isNotEmpty()) return suggestions.take(3) + } catch (e: Exception) { + Timber.d("Failed to parse suggestions as JSON, trying line-by-line") + } + + // Fallback: line-by-line parsing + val lines = response.trim().lines() + .map { it.trim().removePrefix("-").removePrefix("*").removePrefix("1.").removePrefix("2.").removePrefix("3.").trim() } + .filter { it.isNotBlank() && it.length < 100 } + .map { it.removeSurrounding("\"").removeSurrounding("'") } + .take(3) + return lines + } +} diff --git a/vector/src/main/java/im/vector/app/features/ai/suggestedreply/SuggestedReplyView.kt b/vector/src/main/java/im/vector/app/features/ai/suggestedreply/SuggestedReplyView.kt new file mode 100644 index 00000000000..32d591a803e --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/ai/suggestedreply/SuggestedReplyView.kt @@ -0,0 +1,95 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.ai.suggestedreply + +import android.content.Context +import android.util.AttributeSet +import android.view.Gravity +import android.widget.HorizontalScrollView +import android.widget.LinearLayout +import android.widget.ProgressBar +import android.widget.TextView +import androidx.core.view.isVisible +import com.google.android.material.chip.Chip + +class SuggestedReplyView @JvmOverloads constructor( + context: Context, + attrs: AttributeSet? = null, + defStyleAttr: Int = 0 +) : HorizontalScrollView(context, attrs, defStyleAttr) { + + private val chipContainer: LinearLayout + private val progressBar: ProgressBar + private var onSuggestionClick: ((String) -> Unit)? = null + + init { + isHorizontalScrollBarEnabled = false + val padding = (8 * context.resources.displayMetrics.density).toInt() + + val outerLayout = LinearLayout(context).apply { + orientation = LinearLayout.HORIZONTAL + gravity = Gravity.CENTER_VERTICAL + setPadding(padding, 0, padding, 0) + } + + progressBar = ProgressBar(context).apply { + isVisible = false + isIndeterminate = true + val size = (24 * context.resources.displayMetrics.density).toInt() + layoutParams = LinearLayout.LayoutParams(size, size).apply { + marginEnd = padding + } + } + outerLayout.addView(progressBar) + + chipContainer = LinearLayout(context).apply { + orientation = LinearLayout.HORIZONTAL + } + outerLayout.addView(chipContainer) + + addView(outerLayout) + } + + fun setOnSuggestionClickListener(listener: (String) -> Unit) { + onSuggestionClick = listener + } + + fun setSuggestions(suggestions: List) { + chipContainer.removeAllViews() + progressBar.isVisible = false + val padding = (4 * context.resources.displayMetrics.density).toInt() + suggestions.forEach { suggestion -> + val chip = Chip(context).apply { + text = suggestion + isCheckable = false + isClickable = true + layoutParams = LinearLayout.LayoutParams( + LinearLayout.LayoutParams.WRAP_CONTENT, + LinearLayout.LayoutParams.WRAP_CONTENT + ).apply { + marginEnd = padding + } + setOnClickListener { onSuggestionClick?.invoke(suggestion) } + } + chipContainer.addView(chip) + } + isVisible = suggestions.isNotEmpty() + } + + fun showLoading() { + chipContainer.removeAllViews() + progressBar.isVisible = true + isVisible = true + } + + fun clear() { + chipContainer.removeAllViews() + progressBar.isVisible = false + isVisible = false + } +} diff --git a/vector/src/main/java/im/vector/app/features/ai/summary/ConversationSummaryBottomSheet.kt b/vector/src/main/java/im/vector/app/features/ai/summary/ConversationSummaryBottomSheet.kt new file mode 100644 index 00000000000..0aedc4c68a8 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/ai/summary/ConversationSummaryBottomSheet.kt @@ -0,0 +1,136 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.ai.summary + +import android.os.Bundle +import android.view.LayoutInflater +import android.view.View +import android.view.ViewGroup +import android.widget.Button +import android.widget.LinearLayout +import android.widget.ProgressBar +import android.widget.ScrollView +import android.widget.TextView +import androidx.core.view.isVisible +import androidx.fragment.app.FragmentManager +import androidx.lifecycle.lifecycleScope +import com.google.android.material.bottomsheet.BottomSheetDialogFragment +import dagger.hilt.android.AndroidEntryPoint +import im.vector.app.R +import im.vector.app.core.utils.copyToClipboard +import im.vector.app.features.translation.TranslationService +import kotlinx.coroutines.launch +import javax.inject.Inject + +@AndroidEntryPoint +class ConversationSummaryBottomSheet : BottomSheetDialogFragment() { + + @Inject lateinit var translationService: TranslationService + + private var messagesText: String = "" + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + messagesText = arguments?.getString(ARG_MESSAGES) ?: "" + } + + override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View { + val context = requireContext() + val padding = (16 * context.resources.displayMetrics.density).toInt() + + val root = LinearLayout(context).apply { + orientation = LinearLayout.VERTICAL + setPadding(padding, padding, padding, padding) + } + + val title = TextView(context).apply { + text = getString(R.string.summary_title) + textSize = 18f + setPadding(0, 0, 0, padding) + } + root.addView(title) + + val progressBar = ProgressBar(context).apply { + isIndeterminate = true + } + root.addView(progressBar) + + val statusText = TextView(context).apply { + text = getString(R.string.summary_loading) + } + root.addView(statusText) + + val scrollView = ScrollView(context).apply { + isVisible = false + layoutParams = LinearLayout.LayoutParams( + LinearLayout.LayoutParams.MATCH_PARENT, + (300 * context.resources.displayMetrics.density).toInt() + ) + } + + val summaryText = TextView(context).apply { + textSize = 14f + setPadding(0, padding / 2, 0, padding / 2) + } + scrollView.addView(summaryText) + root.addView(scrollView) + + val buttonsLayout = LinearLayout(context).apply { + orientation = LinearLayout.HORIZONTAL + isVisible = false + } + + val copyButton = Button(context).apply { + text = getString(R.string.summary_copy) + setOnClickListener { + copyToClipboard(requireContext(), summaryText.text.toString()) + } + } + buttonsLayout.addView(copyButton) + + val closeButton = Button(context).apply { + text = getString(R.string.summary_close) + setOnClickListener { dismissAllowingStateLoss() } + } + buttonsLayout.addView(closeButton) + root.addView(buttonsLayout) + + lifecycleScope.launch { + val result = translationService.complete( + "You are a conversation summarizer. Summarize the following conversation concisely in the user's language. " + + "Focus on key topics, decisions, and action items. Be brief but comprehensive.", + messagesText + ) + progressBar.isVisible = false + if (result != null) { + statusText.isVisible = false + summaryText.text = result + scrollView.isVisible = true + buttonsLayout.isVisible = true + } else { + statusText.text = getString(R.string.summary_failed) + } + } + + return root + } + + companion object { + private const val ARG_MESSAGES = "arg_messages" + private const val TAG = "ConversationSummaryBottomSheet" + + fun show(fragmentManager: FragmentManager, messagesText: String) { + // Truncate to avoid TransactionTooLargeException from Bundle size limit + val safeText = messagesText.take(500_000) + val sheet = ConversationSummaryBottomSheet().apply { + arguments = Bundle().apply { putString(ARG_MESSAGES, safeText) } + } + sheet.show(fragmentManager, TAG) + } + } +} diff --git a/vector/src/main/java/im/vector/app/features/appfunctions/AppFunctionModels.kt b/vector/src/main/java/im/vector/app/features/appfunctions/AppFunctionModels.kt new file mode 100644 index 00000000000..2a5d3b63435 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/appfunctions/AppFunctionModels.kt @@ -0,0 +1,46 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.appfunctions + +import com.squareup.moshi.JsonClass + +@JsonClass(generateAdapter = true) +data class MessageResult( + val roomId: String, + val roomName: String, + val senderId: String, + val senderName: String, + val body: String, + val timestamp: Long, + val eventId: String +) + +@JsonClass(generateAdapter = true) +data class RoomInfo( + val roomId: String, + val displayName: String, + val lastMessage: String?, + val lastMessageSender: String?, + val unreadCount: Int, + val isDirect: Boolean +) + +@JsonClass(generateAdapter = true) +data class UnreadSummary( + val totalUnread: Int, + val totalMentions: Int, + val rooms: List +) + +@JsonClass(generateAdapter = true) +data class RoomUnread( + val roomId: String, + val roomName: String, + val unreadCount: Int, + val hasMentions: Boolean +) diff --git a/vector/src/main/java/im/vector/app/features/appfunctions/ElementAppFunctionService.kt b/vector/src/main/java/im/vector/app/features/appfunctions/ElementAppFunctionService.kt new file mode 100644 index 00000000000..a6bb756351b --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/appfunctions/ElementAppFunctionService.kt @@ -0,0 +1,428 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.appfunctions + +import android.app.Service +import android.content.Intent +import android.os.Bundle +import android.os.IBinder +import android.os.Build +import android.os.ResultReceiver +import com.squareup.moshi.Moshi +import com.squareup.moshi.Types +import dagger.hilt.android.AndroidEntryPoint +import im.vector.app.core.di.ActiveSessionHolder +import im.vector.app.features.translation.TranslationService +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.cancel +import kotlinx.coroutines.launch +import kotlinx.coroutines.withContext +import org.matrix.android.sdk.api.session.Session +import org.matrix.android.sdk.api.session.events.model.EventType +import org.matrix.android.sdk.api.session.events.model.toModel +import org.matrix.android.sdk.api.session.getRoom +import org.matrix.android.sdk.api.session.getRoomSummary +import org.matrix.android.sdk.api.session.room.model.Membership +import org.matrix.android.sdk.api.session.room.model.message.MessageContent +import org.matrix.android.sdk.api.session.room.roomSummaryQueryParams +import org.matrix.android.sdk.api.session.room.timeline.TimelineSettings +import org.matrix.android.sdk.api.session.room.timeline.getLastMessageContent +import timber.log.Timber +import javax.inject.Inject + +/** + * ElementAppFunctionService — exposes Element Android messaging capabilities + * for inter-app orchestration by AI agents (Off Grid Mobile AI, etc.). + * + * Supports both Android 16 AppFunctions (future) and Intent-based API (current). + * + * Intent Actions: + * - im.vector.app.ACTION_SEARCH_MESSAGES (extras: query, maxResults) + * - im.vector.app.ACTION_READ_MESSAGES (extras: roomId, count) + * - im.vector.app.ACTION_SEND_MESSAGE (extras: roomId, message) + * - im.vector.app.ACTION_LIST_ROOMS (no extras) + * - im.vector.app.ACTION_SUMMARIZE_ROOM (extras: roomId, messageCount) + * - im.vector.app.ACTION_UNREAD_SUMMARY (no extras) + * + * All results are returned as JSON strings in the "result" extra of the result Bundle + * via the ResultReceiver provided in the "resultReceiver" extra. + */ +@AndroidEntryPoint +class ElementAppFunctionService : Service() { + + companion object { + private const val TAG = "APPFUNCTIONS" + + const val ACTION_SEARCH_MESSAGES = "im.vector.app.ACTION_SEARCH_MESSAGES" + const val ACTION_READ_MESSAGES = "im.vector.app.ACTION_READ_MESSAGES" + const val ACTION_SEND_MESSAGE = "im.vector.app.ACTION_SEND_MESSAGE" + const val ACTION_LIST_ROOMS = "im.vector.app.ACTION_LIST_ROOMS" + const val ACTION_SUMMARIZE_ROOM = "im.vector.app.ACTION_SUMMARIZE_ROOM" + const val ACTION_UNREAD_SUMMARY = "im.vector.app.ACTION_UNREAD_SUMMARY" + + const val EXTRA_QUERY = "query" + const val EXTRA_MAX_RESULTS = "maxResults" + const val EXTRA_ROOM_ID = "roomId" + const val EXTRA_COUNT = "count" + const val EXTRA_MESSAGE = "message" + const val EXTRA_MESSAGE_COUNT = "messageCount" + const val EXTRA_RESULT_RECEIVER = "resultReceiver" + + const val RESULT_CODE_SUCCESS = 0 + const val RESULT_CODE_ERROR = 1 + const val RESULT_KEY = "result" + const val ERROR_KEY = "error" + } + + @Inject lateinit var activeSessionHolder: ActiveSessionHolder + @Inject lateinit var translationService: TranslationService + + private val serviceScope = CoroutineScope(SupervisorJob() + Dispatchers.Main) + private val moshi: Moshi by lazy { Moshi.Builder().build() } + + override fun onBind(intent: Intent?): IBinder? { + // No binding for now; use startService with ResultReceiver + return null + } + + override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { + Timber.tag(TAG).d("onStartCommand action=${intent?.action}") + val resultReceiver = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) { + intent?.getParcelableExtra(EXTRA_RESULT_RECEIVER, ResultReceiver::class.java) + } else { + @Suppress("DEPRECATION") + intent?.getParcelableExtra(EXTRA_RESULT_RECEIVER) + } + + if (intent == null) { + sendError(resultReceiver, "No intent provided") + stopSelfIfIdle(startId) + return START_NOT_STICKY + } + + serviceScope.launch { + try { + val session = activeSessionHolder.getSafeActiveSession() + if (session == null) { + sendError(resultReceiver, "No active session. User is not logged in.") + stopSelfIfIdle(startId) + return@launch + } + + val result = when (intent.action) { + ACTION_SEARCH_MESSAGES -> { + val query = intent.getStringExtra(EXTRA_QUERY) ?: "" + val maxResults = intent.getIntExtra(EXTRA_MAX_RESULTS, 10) + searchMessages(session, query, maxResults) + } + ACTION_READ_MESSAGES -> { + val roomId = intent.getStringExtra(EXTRA_ROOM_ID) ?: "" + val count = intent.getIntExtra(EXTRA_COUNT, 20) + readMessages(session, roomId, count) + } + ACTION_SEND_MESSAGE -> { + val roomId = intent.getStringExtra(EXTRA_ROOM_ID) ?: "" + val message = intent.getStringExtra(EXTRA_MESSAGE) ?: "" + sendMessage(session, roomId, message) + } + ACTION_LIST_ROOMS -> { + listRooms(session) + } + ACTION_SUMMARIZE_ROOM -> { + val roomId = intent.getStringExtra(EXTRA_ROOM_ID) ?: "" + val messageCount = intent.getIntExtra(EXTRA_MESSAGE_COUNT, 50) + summarizeRoom(session, roomId, messageCount) + } + ACTION_UNREAD_SUMMARY -> { + getUnreadSummary(session) + } + else -> { + """{"error":"Unknown action: ${intent.action}"}""" + } + } + + sendSuccess(resultReceiver, result) + } catch (e: Exception) { + Timber.tag(TAG).e(e, "Error processing action ${intent.action}") + sendError(resultReceiver, e.message ?: "Unknown error") + } finally { + stopSelfIfIdle(startId) + } + } + + return START_NOT_STICKY + } + + override fun onDestroy() { + super.onDestroy() + serviceScope.cancel() + } + + // ─── Search Messages ────────────────────────────────────────────── + + private suspend fun searchMessages(session: Session, query: String, maxResults: Int): String { + if (query.isBlank()) return "[]" + + Timber.tag(TAG).d("searchMessages query='$query' maxResults=$maxResults") + + val results = mutableListOf() + + // Search across all joined rooms + val roomSummaries = session.roomService().getRoomSummaries( + roomSummaryQueryParams { + memberships = listOf(Membership.JOIN) + } + ) + + for (summary in roomSummaries) { + if (results.size >= maxResults) break + try { + val searchResult = withContext(Dispatchers.IO) { + session.searchService().search( + searchTerm = query, + roomId = summary.roomId, + nextBatch = null, + orderByRecent = true, + limit = maxResults - results.size, + beforeLimit = 0, + afterLimit = 0, + includeProfile = true + ) + } + searchResult.results?.forEach { eventAndSender -> + val event = eventAndSender.event + val body = event.getClearContent()?.toModel()?.body ?: return@forEach + results.add(MessageResult( + roomId = summary.roomId, + roomName = summary.displayName, + senderId = event.senderId ?: "", + senderName = eventAndSender.sender?.displayName ?: event.senderId ?: "", + body = body, + timestamp = event.originServerTs ?: 0L, + eventId = event.eventId ?: "" + )) + } + } catch (e: Exception) { + Timber.tag(TAG).w(e, "Search failed in room ${summary.roomId}") + // Continue to next room + } + } + + val adapter = moshi.adapter>( + Types.newParameterizedType(List::class.java, MessageResult::class.java) + ) + return adapter.toJson(results.take(maxResults)) + } + + // ─── Read Messages ──────────────────────────────────────────────── + + private suspend fun readMessages(session: Session, roomId: String, count: Int): String { + if (roomId.isBlank()) return """{"error":"roomId is required"}""" + + Timber.tag(TAG).d("readMessages roomId='$roomId' count=$count") + + val room = session.getRoom(roomId) ?: return """{"error":"Room not found: $roomId"}""" + val summary = session.getRoomSummary(roomId) + + val results = mutableListOf() + + // Create a timeline, get snapshot, dispose + val timeline = room.timelineService().createTimeline( + eventId = null, + settings = TimelineSettings(initialSize = count, buildReadReceipts = false) + ) + try { + withContext(Dispatchers.IO) { + timeline.start() + // Poll until the timeline snapshot is non-empty or we hit the max wait time. + // The timeline needs time to load events from the local database after start(). + val maxWaitMs = 3000L + val pollIntervalMs = 200L + var waited = 0L + while (waited < maxWaitMs) { + if (timeline.getSnapshot().isNotEmpty()) break + kotlinx.coroutines.delay(pollIntervalMs) + waited += pollIntervalMs + } + } + + val snapshot = timeline.getSnapshot() + + for (event in snapshot) { + val clearType = event.root.getClearType() + if (clearType != EventType.MESSAGE) continue + + val content = event.getLastMessageContent() ?: continue + val body = content.body + + results.add(MessageResult( + roomId = roomId, + roomName = summary?.displayName ?: roomId, + senderId = event.senderInfo.userId, + senderName = event.senderInfo.disambiguatedDisplayName, + body = body, + timestamp = event.root.originServerTs ?: 0L, + eventId = event.eventId + )) + + if (results.size >= count) break + } + } finally { + timeline.dispose() + } + + val adapter = moshi.adapter>( + Types.newParameterizedType(List::class.java, MessageResult::class.java) + ) + return adapter.toJson(results) + } + + // ─── Send Message ───────────────────────────────────────────────── + + private suspend fun sendMessage(session: Session, roomId: String, message: String): String { + if (roomId.isBlank()) return """{"success":false,"error":"roomId is required"}""" + if (message.isBlank()) return """{"success":false,"error":"message is required"}""" + + Timber.tag(TAG).d("sendMessage roomId='$roomId' message length=${message.length}") + + val room = session.getRoom(roomId) ?: return """{"success":false,"error":"Room not found: $roomId"}""" + + return try { + room.sendService().sendTextMessage(message) + """{"success":true}""" + } catch (e: Exception) { + Timber.tag(TAG).e(e, "Failed to send message to $roomId") + """{"success":false,"error":"${e.message?.replace("\"", "\\\"") ?: "Unknown error"}"}""" + } + } + + // ─── List Rooms ─────────────────────────────────────────────────── + + private fun listRooms(session: Session): String { + Timber.tag(TAG).d("listRooms") + + val summaries = session.roomService().getRoomSummaries( + roomSummaryQueryParams { + memberships = listOf(Membership.JOIN) + } + ) + + val rooms = summaries.map { summary -> + val lastEvent = summary.latestPreviewableEvent + val lastBody = lastEvent?.getLastMessageContent()?.body + val lastSender = lastEvent?.senderInfo?.disambiguatedDisplayName + + RoomInfo( + roomId = summary.roomId, + displayName = summary.displayName, + lastMessage = lastBody, + lastMessageSender = lastSender, + unreadCount = summary.notificationCount, + isDirect = summary.isDirect + ) + } + + val adapter = moshi.adapter>( + Types.newParameterizedType(List::class.java, RoomInfo::class.java) + ) + return adapter.toJson(rooms) + } + + // ─── Summarize Room ─────────────────────────────────────────────── + + private suspend fun summarizeRoom(session: Session, roomId: String, messageCount: Int): String { + if (roomId.isBlank()) return """{"error":"roomId is required"}""" + + Timber.tag(TAG).d("summarizeRoom roomId='$roomId' messageCount=$messageCount") + + // First, read the messages + val messagesJson = readMessages(session, roomId, messageCount) + val adapter = moshi.adapter>( + Types.newParameterizedType(List::class.java, MessageResult::class.java) + ) + val messages = adapter.fromJson(messagesJson) ?: emptyList() + + if (messages.isEmpty()) { + return """{"summary":"No messages found in this room."}""" + } + + // Build conversation text for the AI + val conversation = messages.reversed().joinToString("\n") { msg -> + "[${msg.senderName}]: ${msg.body}" + } + + val summary = translationService.complete( + systemPrompt = "You are a helpful assistant. Summarize the following conversation concisely. " + + "Focus on the key topics discussed, decisions made, and action items. " + + "Reply in the same language as the majority of the messages. " + + "Keep the summary under 200 words.", + userMessage = conversation + ) + + return if (summary != null) { + val resultMap = mapOf("summary" to summary, "messageCount" to messages.size) + moshi.adapter>( + Types.newParameterizedType(Map::class.java, String::class.java, Any::class.java) + ).toJson(resultMap) + } else { + """{"error":"AI summarization failed. Check TranslationService configuration.","messageCount":${messages.size}}""" + } + } + + // ─── Unread Summary ─────────────────────────────────────────────── + + private fun getUnreadSummary(session: Session): String { + Timber.tag(TAG).d("getUnreadSummary") + + val summaries = session.roomService().getRoomSummaries( + roomSummaryQueryParams { + memberships = listOf(Membership.JOIN) + } + ) + + val roomUnreads = summaries + .filter { it.notificationCount > 0 || it.highlightCount > 0 } + .map { summary -> + RoomUnread( + roomId = summary.roomId, + roomName = summary.displayName, + unreadCount = summary.notificationCount, + hasMentions = summary.highlightCount > 0 + ) + } + + val unreadSummary = UnreadSummary( + totalUnread = roomUnreads.sumOf { it.unreadCount }, + totalMentions = summaries.sumOf { it.highlightCount }, + rooms = roomUnreads + ) + + return moshi.adapter(UnreadSummary::class.java).toJson(unreadSummary) + } + + // ─── Helpers ────────────────────────────────────────────────────── + + private fun sendSuccess(receiver: ResultReceiver?, result: String) { + receiver?.send(RESULT_CODE_SUCCESS, Bundle().apply { + putString(RESULT_KEY, result) + }) + } + + private fun sendError(receiver: ResultReceiver?, error: String) { + Timber.tag(TAG).e("Error: $error") + receiver?.send(RESULT_CODE_ERROR, Bundle().apply { + putString(ERROR_KEY, error) + }) + } + + private fun stopSelfIfIdle(startId: Int) { + stopSelf(startId) + } +} diff --git a/vector/src/main/java/im/vector/app/features/appfunctions/ElementAppFunctions.kt b/vector/src/main/java/im/vector/app/features/appfunctions/ElementAppFunctions.kt new file mode 100644 index 00000000000..dbf5459c427 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/appfunctions/ElementAppFunctions.kt @@ -0,0 +1,110 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.appfunctions + +import android.content.Context +import android.content.Intent +import android.os.Bundle +import android.os.ResultReceiver +import kotlinx.coroutines.suspendCancellableCoroutine +import kotlinx.coroutines.withTimeout +import timber.log.Timber +import kotlin.coroutines.resume + +/** + * Element Android AppFunctions — high-level API for inter-app AI orchestration. + * + * This class provides a convenient Kotlin API that wraps the Intent-based + * ElementAppFunctionService. It can be used directly within the app, or + * will be annotated with @AppFunction once the Jetpack library is stable. + * + * Functions exposed: + * - searchMessages: Search messages across all rooms + * - readMessages: Read the last N messages from a room + * - sendMessage: Send a text message to a room + * - listRooms: List all joined rooms + * - summarizeRoom: Summarize a room conversation using AI + * - getUnreadSummary: Get unread counts across all rooms + */ +class ElementAppFunctions(private val context: Context) { + + companion object { + private const val TAG = "APPFUNCTIONS" + private const val SERVICE_TIMEOUT_MS = 30_000L + } + + /** Search messages across all rooms matching the given query. */ + suspend fun searchMessages(query: String, maxResults: Int = 10): String { + return callService(ElementAppFunctionService.ACTION_SEARCH_MESSAGES) { + putExtra(ElementAppFunctionService.EXTRA_QUERY, query) + putExtra(ElementAppFunctionService.EXTRA_MAX_RESULTS, maxResults) + } + } + + /** Read the last N messages from a specific room. */ + suspend fun readMessages(roomId: String, count: Int = 20): String { + return callService(ElementAppFunctionService.ACTION_READ_MESSAGES) { + putExtra(ElementAppFunctionService.EXTRA_ROOM_ID, roomId) + putExtra(ElementAppFunctionService.EXTRA_COUNT, count) + } + } + + /** Send a text message to a specific room. */ + suspend fun sendMessage(roomId: String, message: String): String { + return callService(ElementAppFunctionService.ACTION_SEND_MESSAGE) { + putExtra(ElementAppFunctionService.EXTRA_ROOM_ID, roomId) + putExtra(ElementAppFunctionService.EXTRA_MESSAGE, message) + } + } + + /** List all joined rooms with their display name, last message, and unread count. */ + suspend fun listRooms(): String { + return callService(ElementAppFunctionService.ACTION_LIST_ROOMS) {} + } + + /** Summarize the last N messages in a room using AI. */ + suspend fun summarizeRoom(roomId: String, messageCount: Int = 50): String { + return callService(ElementAppFunctionService.ACTION_SUMMARIZE_ROOM) { + putExtra(ElementAppFunctionService.EXTRA_ROOM_ID, roomId) + putExtra(ElementAppFunctionService.EXTRA_MESSAGE_COUNT, messageCount) + } + } + + /** Get unread message count and mentions across all rooms. */ + suspend fun getUnreadSummary(): String { + return callService(ElementAppFunctionService.ACTION_UNREAD_SUMMARY) {} + } + + private suspend fun callService(action: String, configure: Intent.() -> Unit): String { + return withTimeout(SERVICE_TIMEOUT_MS) { + suspendCancellableCoroutine { continuation -> + val intent = Intent(context, ElementAppFunctionService::class.java).apply { + this.action = action + configure() + putExtra(ElementAppFunctionService.EXTRA_RESULT_RECEIVER, object : ResultReceiver(null) { + override fun onReceiveResult(resultCode: Int, resultData: Bundle?) { + val result = if (resultCode == ElementAppFunctionService.RESULT_CODE_SUCCESS) { + resultData?.getString(ElementAppFunctionService.RESULT_KEY) ?: """{"error":"No result"}""" + } else { + val error = resultData?.getString(ElementAppFunctionService.ERROR_KEY) ?: "Unknown error" + """{"error":"$error"}""" + } + continuation.resume(result) + } + }) + } + try { + context.startService(intent) + } catch (e: Exception) { + Timber.tag(TAG).e(e, "Failed to start ElementAppFunctionService") + continuation.resume("""{"error":"Failed to start service: ${e.message}"}""") + } + } + } + } +} diff --git a/vector/src/main/java/im/vector/app/features/home/HomeActivity.kt b/vector/src/main/java/im/vector/app/features/home/HomeActivity.kt index bea9d1303ba..9e94b9d933d 100644 --- a/vector/src/main/java/im/vector/app/features/home/HomeActivity.kt +++ b/vector/src/main/java/im/vector/app/features/home/HomeActivity.kt @@ -131,8 +131,11 @@ class HomeActivity : @Inject lateinit var unifiedPushHelper: UnifiedPushHelper @Inject lateinit var nightlyProxy: NightlyProxy @Inject lateinit var notificationPermissionManager: NotificationPermissionManager + @Inject lateinit var translateConfig: im.vector.app.features.translation.TranslateConfig + @Inject lateinit var notificationSummaryService: im.vector.app.features.ai.notificationsummary.NotificationSummaryService private var isNewAppLayoutEnabled: Boolean = false // delete once old app layout is removed + private var lastPauseTimestamp: Long = 0L private val createSpaceResultLauncher = registerStartForActivityResult { activityResult -> if (activityResult.resultCode == Activity.RESULT_OK) { @@ -580,9 +583,17 @@ class HomeActivity : super.onDestroy() } + override fun onPause() { + super.onPause() + lastPauseTimestamp = System.currentTimeMillis() + } + override fun onResume() { super.onResume() + // Check if user was away and show notification summary + checkNotificationSummary() + if (vectorUncaughtExceptionHandler.didAppCrash()) { vectorUncaughtExceptionHandler.clearAppCrashStatus() @@ -719,6 +730,35 @@ class HomeActivity : // nop } + private fun checkNotificationSummary() { + if (!translateConfig.enabled || !translateConfig.notificationSummaryEnabled) return + if (lastPauseTimestamp == 0L) return + val awayDurationMs = System.currentTimeMillis() - lastPauseTimestamp + if (awayDurationMs < 5 * 60 * 1000L) return // less than 5 minutes + + lastPauseTimestamp = 0L // reset to avoid showing again + + // Get queued message notifications + val messageEvents = notificationDrawerManager.getQueuedMessageEvents() + if (messageEvents.isEmpty()) return + + Timber.d("NotificationSummary: ${messageEvents.size} unread notifications after ${awayDurationMs / 1000}s away") + + // Format as "roomName|messageBody" strings for the bottom sheet + val notificationStrings = messageEvents.mapNotNull { event -> + val roomName = event.roomName ?: return@mapNotNull null + val body = event.body ?: return@mapNotNull null + "$roomName|$body" + } + if (notificationStrings.isEmpty()) return + + // Show the bottom sheet which will generate the summary asynchronously + im.vector.app.features.ai.notificationsummary.NotificationSummaryBottomSheet.show( + fragmentManager = supportFragmentManager, + notifications = notificationStrings + ) + } + companion object { fun newIntent( context: Context, diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/TimelineFragment.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/TimelineFragment.kt index 01be29d84ff..a83692eb6f4 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/TimelineFragment.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/TimelineFragment.kt @@ -37,6 +37,7 @@ import androidx.core.view.isInvisible import androidx.core.view.isVisible import androidx.fragment.app.setFragmentResultListener import androidx.lifecycle.lifecycleScope +import kotlinx.coroutines.launch import androidx.lifecycle.withResumed import androidx.recyclerview.widget.ItemTouchHelper import androidx.recyclerview.widget.LinearLayoutManager @@ -131,6 +132,10 @@ import im.vector.app.features.home.room.detail.timeline.TimelineEventController import im.vector.app.features.home.room.detail.timeline.action.EventSharedAction import im.vector.app.features.home.room.detail.timeline.action.MessageActionsBottomSheet import im.vector.app.features.home.room.detail.timeline.action.MessageSharedActionViewModel +import im.vector.app.features.ai.summary.ConversationSummaryBottomSheet +import im.vector.app.features.translation.TranslateConfig +import im.vector.app.features.translation.TranslationService +import im.vector.app.features.translation.TimelineTranslationManager import im.vector.app.features.home.room.detail.timeline.edithistory.ViewEditHistoryBottomSheet import im.vector.app.features.home.room.detail.timeline.helper.AudioMessagePlaybackTracker import im.vector.app.features.home.room.detail.timeline.helper.MatrixItemColorProvider @@ -242,6 +247,9 @@ class TimelineFragment : @Inject lateinit var vectorFeatures: VectorFeatures @Inject lateinit var galleryOrCameraDialogHelperFactory: GalleryOrCameraDialogHelperFactory @Inject lateinit var permalinkFactory: PermalinkFactory + @Inject lateinit var translateConfig: TranslateConfig + @Inject lateinit var translationService: TranslationService + @Inject lateinit var timelineTranslationManager: TimelineTranslationManager companion object { const val MAX_TYPING_MESSAGE_USERS_COUNT = 4 @@ -255,6 +263,7 @@ class TimelineFragment : private val messageComposerViewModel: MessageComposerViewModel by fragmentViewModel() private val debouncer = Debouncer(createUIHandler()) private val itemVisibilityTracker = EpoxyVisibilityTracker() + private var lastAutoTranslateState: Boolean = false private lateinit var scrollOnNewMessageCallback: ScrollOnNewMessageCallback private lateinit var scrollOnHighlightedEventCallback: ScrollOnHighlightedEventCallback @@ -682,6 +691,7 @@ class TimelineFragment : override fun onDestroyView() { lazyLoadedViews.unBind() + timelineTranslationManager.setListener(null) timelineEventController.callback = null timelineEventController.removeModelBuildListener(modelBuildListener) currentCallsViewPresenter.unBind() @@ -872,6 +882,10 @@ class TimelineFragment : handleSearchAction() true } + R.id.menu_summarize -> { + handleSummarizeAction() + true + } R.id.dev_tools -> { navigator.openDevTools(requireContext(), timelineArgs.roomId) true @@ -967,6 +981,13 @@ class TimelineFragment : notificationDrawerManager.setCurrentThread(timelineArgs.threadTimelineArgs?.rootThreadEventId) roomDetailPendingActionStore.data?.let { handlePendingAction(it) } roomDetailPendingActionStore.data = null + + // Detect auto-translate setting changes and refresh timeline + val currentAutoTranslate = translateConfig.autoTranslate && translateConfig.enabled + if (currentAutoTranslate != lastAutoTranslateState) { + lastAutoTranslateState = currentAutoTranslate + timelineEventController.invalidateAllCache() + } } private fun handlePendingAction(roomDetailPendingAction: RoomDetailPendingAction) { @@ -1028,6 +1049,13 @@ class TimelineFragment : timelineEventController.callback = this timelineEventController.timeline = timelineViewModel.timeline + // Set up translation listener to refresh timeline when translations arrive + timelineTranslationManager.setListener(object : TimelineTranslationManager.TranslationListener { + override fun onTranslationReady(eventId: String) { + timelineEventController.invalidateCacheForEvent(eventId) + } + }) + layoutManager = object : LinearLayoutManager(requireContext(), RecyclerView.VERTICAL, true) { override fun onLayoutCompleted(state: RecyclerView.State) { super.onLayoutCompleted(state) @@ -1891,11 +1919,42 @@ class TimelineFragment : is EventSharedAction.EndPoll -> { askConfirmationToEndPoll(action.eventId) } + is EventSharedAction.Translate -> { + handleTranslateAction(action) + } is EventSharedAction.ReportContent -> Unit /* Not clickable */ EventSharedAction.Separator -> Unit /* Not clickable */ } } + private fun handleSummarizeAction() { + val messages = timelineEventController.getRecentTextMessages(50) + if (messages.isEmpty()) { + showSnackWithMessage(getString(R.string.no_text_messages_found)) + return + } + val formatted = messages.joinToString("\n") { (sender, timestamp, body) -> + val time = timestamp?.let { + java.text.SimpleDateFormat("HH:mm", java.util.Locale.getDefault()).format(java.util.Date(it)) + } ?: "" + "[$sender] ($time): $body" + } + ConversationSummaryBottomSheet.show(childFragmentManager, formatted) + } + + private fun handleTranslateAction(action: EventSharedAction.Translate) { + showSnackWithMessage(getString(R.string.translation_translating)) + lifecycleScope.launch { + val translated = translationService.translate(action.content, translateConfig.targetLanguage) + if (translated != null) { + copyToClipboard(requireContext(), translated, false) + showSnackWithMessage(translated) + } else { + showSnackWithMessage(getString(R.string.translation_failed)) + } + } + } + private fun openEmojiReactionPicker(eventId: String) { emojiActivityResultLauncher.launch(EmojiReactionPickerActivity.intent(requireContext(), eventId)) } diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerFragment.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerFragment.kt index 894c3844fd0..3cb6f4a0de0 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerFragment.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerFragment.kt @@ -15,8 +15,11 @@ import android.net.Uri import android.os.Build import android.os.Bundle import android.text.Spannable +import android.view.ActionMode import android.view.KeyEvent import android.view.LayoutInflater +import android.view.Menu +import android.view.MenuItem import android.view.View import android.view.ViewGroup import android.view.inputmethod.EditorInfo @@ -84,6 +87,10 @@ import im.vector.app.features.location.LocationSharingMode import im.vector.app.features.poll.PollMode import im.vector.app.features.settings.VectorPreferences import im.vector.app.features.share.SharedData +import im.vector.app.features.ai.reformulate.ReformulationBottomSheet +import im.vector.app.features.ai.suggestedreply.SuggestedReplyService +import im.vector.app.features.ai.suggestedreply.SuggestedReplyView +import im.vector.app.features.translation.TranslateConfig import im.vector.app.features.voice.VoiceFailure import im.vector.lib.strings.CommonStrings import kotlinx.coroutines.flow.debounce @@ -116,6 +123,8 @@ class MessageComposerFragment : VectorBaseFragment(), A @Inject lateinit var buildMeta: BuildMeta @Inject lateinit var session: Session @Inject lateinit var errorTracker: ErrorTracker + @Inject lateinit var translateConfig: TranslateConfig + @Inject lateinit var suggestedReplyService: SuggestedReplyService private val permalinkService: PermalinkService get() = session.permalinkService() @@ -136,6 +145,8 @@ class MessageComposerFragment : VectorBaseFragment(), A get() = vectorPreferences.showEmojiKeyboard() private var lockSendButton = false + private var originalTextBeforeReformulation: String? = null + private var suggestedReplyView: SuggestedReplyView? = null private lateinit var attachmentsHelper: AttachmentsHelper private lateinit var attachmentTypeSelector: AttachmentTypeSelectorView @@ -170,6 +181,9 @@ class MessageComposerFragment : VectorBaseFragment(), A setupBottomSheet() setupComposer() setupEmojiButton() + setupReformulationButton() + setupReformulationResultListener() + setupSuggestedReplies() views.composerLayout.isGone = vectorPreferences.isRichTextEditorEnabled() views.richTextComposerLayout.isVisible = vectorPreferences.isRichTextEditorEnabled() @@ -541,6 +555,101 @@ class MessageComposerFragment : VectorBaseFragment(), A } } + private fun setupReformulationButton() { + if (!translateConfig.enabled || !translateConfig.reformulationEnabled) return + + // Add "Reformuler" option to the text selection / long-press context menu of the composer EditText + val composerEditText = composer.editText + + val reformulationCallback = object : ActionMode.Callback { + override fun onCreateActionMode(mode: ActionMode, menu: Menu): Boolean { + if (composerEditText.text?.isNotBlank() == true) { + menu.add(Menu.NONE, R.id.menu_reformulate, 10, R.string.reformulate_title) + } + return true + } + + override fun onPrepareActionMode(mode: ActionMode, menu: Menu): Boolean = false + + override fun onActionItemClicked(mode: ActionMode, item: MenuItem): Boolean { + return if (item.itemId == R.id.menu_reformulate) { + val text = composerEditText.text?.toString() ?: "" + if (text.isNotBlank()) { + originalTextBeforeReformulation = text + ReformulationBottomSheet.show(childFragmentManager, text) + } + mode.finish() + true + } else { + false + } + } + + override fun onDestroyActionMode(mode: ActionMode) {} + } + + composerEditText.customSelectionActionModeCallback = reformulationCallback + composerEditText.customInsertionActionModeCallback = reformulationCallback + } + + private fun setupReformulationResultListener() { + childFragmentManager.setFragmentResultListener( + ReformulationBottomSheet.REQUEST_KEY, + viewLifecycleOwner + ) { _, bundle -> + val text = bundle.getString(ReformulationBottomSheet.RESULT_TEXT) + if (text != null) { + composer.setTextIfDifferent(text) + } + } + } + + @Suppress("UNUSED_VARIABLE") + private fun setupSuggestedReplies() { + if (!translateConfig.enabled || !translateConfig.suggestedRepliesEnabled) return + // Create SuggestedReplyView and add it above the composer + val composerView = composer as? View ?: return + val composerParent = composerView.parent as? ViewGroup ?: return + + suggestedReplyView = SuggestedReplyView(requireContext()).apply { + isVisible = false + layoutParams = ViewGroup.LayoutParams( + ViewGroup.LayoutParams.MATCH_PARENT, + ViewGroup.LayoutParams.WRAP_CONTENT + ) + setOnSuggestionClickListener { suggestion -> + composer.setTextIfDifferent(suggestion) + clear() + } + } + + val index = composerParent.indexOfChild(composerView) + composerParent.addView(suggestedReplyView, index) + + // Hide suggestions when user starts typing + composer.editText.textChanges() + .skipInitialValue() + .onEach { + if (it.isNotEmpty()) { + suggestedReplyView?.clear() + } + } + .launchIn(viewLifecycleOwner.lifecycleScope) + + // Fetch suggestions when room opens + fetchSuggestedReplies() + } + + private fun fetchSuggestedReplies() { + if (!translateConfig.enabled || !translateConfig.suggestedRepliesEnabled) return + if (suggestedReplyView == null) return + + // Suggested replies will be triggered from the TimelineFragment which has access + // to the timeline controller and can pass recent messages. + // For now, the SuggestedReplyView is set up and ready to receive suggestions. + Timber.d("Suggested replies view ready, waiting for messages from timeline") + } + private fun onCannotRecord() { // Update the UI, cancel the animation messageComposerViewModel.handle(MessageComposerAction.OnVoiceRecordingUiStateChanged(VoiceMessageRecorderView.RecordingUiState.Idle)) diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerViewModel.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerViewModel.kt index 4e645f93aeb..1476369879e 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerViewModel.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/composer/MessageComposerViewModel.kt @@ -75,6 +75,7 @@ import org.matrix.android.sdk.api.session.space.CreateSpaceParams import org.matrix.android.sdk.api.util.Optional import org.matrix.android.sdk.flow.flow import org.matrix.android.sdk.flow.unwrap +import im.vector.app.features.translation.TranslationService import timber.log.Timber class MessageComposerViewModel @AssistedInject constructor( @@ -89,6 +90,7 @@ class MessageComposerViewModel @AssistedInject constructor( private val voiceBroadcastHelper: VoiceBroadcastHelper, private val clock: Clock, private val getVoiceBroadcastStateEventLiveUseCase: GetVoiceBroadcastStateEventLiveUseCase, + private val translationService: TranslationService, ) : VectorViewModel(initialState) { private val room = session.getRoom(initialState.roomId) @@ -239,24 +241,35 @@ class MessageComposerViewModel @AssistedInject constructor( isInThreadTimeline = state.isInThreadTimeline() )) { is ParsedCommand.ErrorNotACommand -> { - // Send the text message to the room - if (state.rootThreadEventId != null) { - room.relationService().replyInThread( - rootThreadEventId = state.rootThreadEventId, - replyInThreadText = action.text, - formattedText = action.formattedText, - autoMarkdown = action.autoMarkdown - ) - } else { - if (action.formattedText != null) { - room.sendService().sendFormattedTextMessage(action.text.toString(), action.formattedText) - } else { - room.sendService().sendTextMessage(action.text, autoMarkdown = action.autoMarkdown) + // Send the text message to the room, with optional outgoing translation + viewModelScope.launch { + try { + val translatedText = translationService.translateOutgoing(action.text.toString()) + val translatedFormatted = if (action.formattedText != null) { + translationService.translateOutgoing(action.formattedText) + } else null + if (state.rootThreadEventId != null) { + room.relationService().replyInThread( + rootThreadEventId = state.rootThreadEventId, + replyInThreadText = translatedText, + formattedText = translatedFormatted, + autoMarkdown = action.autoMarkdown + ) + } else { + if (translatedFormatted != null) { + room.sendService().sendFormattedTextMessage(translatedText, translatedFormatted) + } else { + room.sendService().sendTextMessage(translatedText, autoMarkdown = action.autoMarkdown) + } + } + _viewEvents.post(MessageComposerViewEvents.MessageSent) + popDraft(room) + } catch (e: Exception) { + Timber.e(e, "Failed to translate/send message") + _viewEvents.post(MessageComposerViewEvents.MessageSent) + popDraft(room) } } - - _viewEvents.post(MessageComposerViewEvents.MessageSent) - popDraft(room) } is ParsedCommand.ErrorSyntax -> { _viewEvents.post(MessageComposerViewEvents.SlashCommandError(parsedCommand.command)) diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/TimelineEventController.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/TimelineEventController.kt index 722e30b8fdf..06227324808 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/TimelineEventController.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/TimelineEventController.kt @@ -18,6 +18,7 @@ import com.airbnb.epoxy.EpoxyModel import com.airbnb.epoxy.VisibilityState import im.vector.app.core.date.DateFormatKind import im.vector.app.core.date.VectorDateFormatter +import im.vector.app.core.extensions.getVectorLastMessageContent import im.vector.app.core.epoxy.LoadingItem_ import im.vector.app.core.extensions.localDateTime import im.vector.app.core.extensions.nextOrNull @@ -69,6 +70,8 @@ import org.matrix.android.sdk.api.session.room.model.message.MessageVideoContent import org.matrix.android.sdk.api.session.room.timeline.Timeline import org.matrix.android.sdk.api.session.room.timeline.TimelineEvent import timber.log.Timber +import im.vector.app.core.utils.Debouncer +import im.vector.app.core.utils.createUIHandler import javax.inject.Inject import kotlin.math.min import kotlin.system.measureTimeMillis @@ -293,6 +296,72 @@ class TimelineEventController @Inject constructor( interceptorHelper.intercept(models, partialState.unreadState, timeline, callback) } + private val translationDebouncer = Debouncer(Handler(Looper.getMainLooper())) + private val pendingTranslationInvalidations = mutableSetOf() + + /** + * Invalidate the cached model for a specific event, forcing it to be rebuilt. + * Uses debouncing to batch multiple invalidations into a single requestModelBuild(). + */ + fun invalidateCacheForEvent(eventId: String) { + synchronized(pendingTranslationInvalidations) { + pendingTranslationInvalidations.add(eventId) + } + translationDebouncer.debounce("translation_invalidation", 200) { + synchronized(modelCache) { + val toInvalidate: Set + synchronized(pendingTranslationInvalidations) { + toInvalidate = pendingTranslationInvalidations.toSet() + pendingTranslationInvalidations.clear() + } + var invalidated = false + toInvalidate.forEach { id -> + val position = currentSnapshot.indexOfFirst { it.root.eventId == id } + if (position != -1 && position < modelCache.size) { + modelCache[position] = null + invalidated = true + } + } + if (invalidated) { + requestModelBuild() + } + } + } + } + + /** + * Invalidate all cached models, forcing a full rebuild. + * Used when auto-translate setting changes. + */ + fun invalidateAllCache() { + synchronized(modelCache) { + for (i in modelCache.indices) { + modelCache[i] = null + } + requestModelBuild() + } + } + + /** + * Get recent text messages from the current snapshot for summarization. + */ + fun getRecentTextMessages(count: Int): List> { + val result = mutableListOf>() + synchronized(modelCache) { + for (event in currentSnapshot) { + if (result.size >= count) break + val content = event.getVectorLastMessageContent() + if (content is org.matrix.android.sdk.api.session.room.model.message.MessageTextContent) { + val senderName = event.senderInfo.disambiguatedDisplayName + val timestamp = event.root.originServerTs + val body = content.body + result.add(Triple(senderName, timestamp, body)) + } + } + } + return result.reversed() // chronological order + } + fun update(viewState: RoomDetailViewState) { val newPartialState = PartialState(viewState) if (newPartialState != partialState) { diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/EventSharedAction.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/EventSharedAction.kt index b8ca29ee1ed..addd5940823 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/EventSharedAction.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/EventSharedAction.kt @@ -119,4 +119,7 @@ sealed class EventSharedAction( data class EndPoll(val eventId: String) : EventSharedAction(CommonStrings.poll_end_action, R.drawable.ic_check_on) + + data class Translate(val eventId: String, val content: String) : + EventSharedAction(R.string.message_action_translate, R.drawable.ic_translate) } diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/MessageActionsViewModel.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/MessageActionsViewModel.kt index 2280a268425..6444d25acf1 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/MessageActionsViewModel.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/action/MessageActionsViewModel.kt @@ -25,6 +25,7 @@ import im.vector.app.features.html.PillsPostProcessor import im.vector.app.features.html.VectorHtmlCompressor import im.vector.app.features.reactions.data.EmojiDataSource import im.vector.app.features.settings.VectorPreferences +import im.vector.app.features.translation.TranslateConfig import im.vector.lib.strings.CommonStrings import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.flatMapLatest @@ -70,6 +71,7 @@ class MessageActionsViewModel @AssistedInject constructor( private val stringProvider: StringProvider, private val pillsPostProcessorFactory: PillsPostProcessor.Factory, private val vectorPreferences: VectorPreferences, + private val translateConfig: TranslateConfig, private val checkIfCanReplyEventUseCase: CheckIfCanReplyEventUseCase, private val checkIfCanRedactEventUseCase: CheckIfCanRedactEventUseCase, ) : VectorViewModel(initialState) { @@ -353,6 +355,10 @@ class MessageActionsViewModel @AssistedInject constructor( add(EventSharedAction.Copy(messageContent!!.body)) } + if (translateConfig.enabled && canCopy(msgType) && messageContent != null) { + add(EventSharedAction.Translate(eventId, messageContent.body)) + } + if (timelineEvent.canReact() && actionPermissions.canReact) { add(EventSharedAction.AddReaction(eventId)) } diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/factory/MessageItemFactory.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/factory/MessageItemFactory.kt index 6415e06d3e0..1f7dc3fd21c 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/factory/MessageItemFactory.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/factory/MessageItemFactory.kt @@ -16,6 +16,7 @@ import android.text.style.ClickableSpan import android.text.style.ForegroundColorSpan import android.view.View import dagger.Lazy +import timber.log.Timber import im.vector.app.R import im.vector.app.core.epoxy.ClickListener import im.vector.app.core.epoxy.VectorEpoxyModel @@ -69,6 +70,7 @@ import im.vector.app.features.location.toLocationData import im.vector.app.features.media.ImageContentRenderer import im.vector.app.features.media.VideoContentRenderer import im.vector.app.features.settings.VectorPreferences +import im.vector.app.features.translation.TimelineTranslationManager import im.vector.app.features.voice.AudioWaveformView import im.vector.app.features.voicebroadcast.isVoiceBroadcast import im.vector.app.features.voicebroadcast.model.MessageVoiceBroadcastInfoContent @@ -138,6 +140,7 @@ class MessageItemFactory @Inject constructor( private val pollItemViewStateFactory: PollItemViewStateFactory, private val voiceBroadcastItemFactory: VoiceBroadcastItemFactory, private val processBodyOfReplyToEventUseCase: ProcessBodyOfReplyToEventUseCase, + private val timelineTranslationManager: TimelineTranslationManager, ) { // TODO inject this properly? @@ -600,8 +603,69 @@ class MessageItemFactory @Inject constructor( callback: TimelineEventController.Callback?, attributes: AbsMessageItem.Attributes, ): VectorEpoxyModel<*>? { + // Try to get auto-translated text + val translatedBody = timelineTranslationManager.getTranslatedText(informationData.eventId, messageContent.body) + val matrixFormattedBody = messageContent.matrixFormattedBody - return if (matrixFormattedBody != null) { + return if (translatedBody != null) { + // Use translated text but preserve reply formatting if present + val replyToContent = messageContent.relatesTo?.inReplyTo + if (replyToContent != null && matrixFormattedBody != null) { + // Keep the reply block from the original formatted body and translate the quoted text too + val endMxReply = matrixFormattedBody.indexOf("") + val endBlockQuote = matrixFormattedBody.lastIndexOf("") + if (endBlockQuote != -1) { + var replyBlock = matrixFormattedBody.substring(0, endBlockQuote + "".length) + + // Try to translate the quoted text: + // 1. First try the parent event's cached translation + // 2. Fall back to the reply quote translation cached for the current event + val parentEventId = replyToContent.eventId + var quoteTranslation: String? = null + if (parentEventId != null) { + val parentEvent = session.roomService().getRoom(roomId)?.getTimelineEvent(parentEventId) + val parentBody = parentEvent?.getVectorLastMessageContent()?.body + if (parentBody != null) { + quoteTranslation = timelineTranslationManager.getCachedTranslation(parentBody) + } + } + // Fall back to reply quote translation extracted from the current message's body + if (quoteTranslation == null) { + quoteTranslation = timelineTranslationManager.getTranslatedReplyQuote(informationData.eventId) + } + Timber.d("TRANSLATION_DEBUG buildItem eventId=${informationData.eventId} parentEventId=$parentEventId quoteTranslation=$quoteTranslation replyBlock=${replyBlock.take(300)}") + if (quoteTranslation != null) { + // Try
first, then
, then
— Matrix HTML varies + val brPatterns = listOf("
", "
", "
") + var lastBrIndex = -1 + var brLen = 0 + for (br in brPatterns) { + val idx = replyBlock.lastIndexOf(br) + if (idx > lastBrIndex) { + lastBrIndex = idx + brLen = br.length + } + } + if (lastBrIndex != -1) { + val beforeQuoteText = replyBlock.substring(0, lastBrIndex + brLen) + replyBlock = "$beforeQuoteText$quoteTranslation" + Timber.d("TRANSLATION_DEBUG replaced quote HTML OK for ${informationData.eventId}") + } else { + Timber.d("TRANSLATION_DEBUG NO
found in replyBlock for ${informationData.eventId}") + } + } + + // Close the mx-reply wrapper if present, without extra
to avoid spacing issues + val mxReplyClose = if (endMxReply != -1) "" else "" + val combinedHtml = "$replyBlock$mxReplyClose$translatedBody" + buildFormattedTextItem(combinedHtml, informationData, highlight, callback, attributes, replyToContent, originalBody = messageContent.body) + } else { + buildMessageTextItem(translatedBody, false, informationData, highlight, callback, attributes, originalBody = messageContent.body) + } + } else { + buildMessageTextItem(translatedBody, false, informationData, highlight, callback, attributes, originalBody = messageContent.body) + } + } else if (matrixFormattedBody != null) { val replyToContent = messageContent.relatesTo?.inReplyTo buildFormattedTextItem(matrixFormattedBody, informationData, highlight, callback, attributes, replyToContent) } else { @@ -616,6 +680,7 @@ class MessageItemFactory @Inject constructor( callback: TimelineEventController.Callback?, attributes: AbsMessageItem.Attributes, replyToContent: ReplyToContent?, + originalBody: CharSequence? = null, ): MessageTextItem? { val processedBody = replyToContent ?.let { processBodyOfReplyToEventUseCase.execute(roomId, matrixFormattedBody, it) } @@ -629,6 +694,7 @@ class MessageItemFactory @Inject constructor( highlight, callback, attributes, + originalBody = originalBody, ) } @@ -639,6 +705,7 @@ class MessageItemFactory @Inject constructor( highlight: Boolean, callback: TimelineEventController.Callback?, attributes: AbsMessageItem.Attributes, + originalBody: CharSequence? = null, ): MessageTextItem? { val renderedBody = textRenderer.render(body) val bindingOptions = spanUtils.getBindingOptions(renderedBody) @@ -652,6 +719,7 @@ class MessageItemFactory @Inject constructor( linkifiedBody }.toEpoxyCharSequence() ) + .originalMessage(originalBody?.toEpoxyCharSequence()) .useBigFont(linkifiedBody.length <= MAX_NUMBER_OF_EMOJI_FOR_BIG_FONT * 2 && containsOnlyEmojis(linkifiedBody.toString())) .bindingOptions(bindingOptions) .markwonPlugins(htmlRenderer.get().plugins) diff --git a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/item/MessageTextItem.kt b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/item/MessageTextItem.kt index 7ea59a47ecd..0b4fbed373d 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/item/MessageTextItem.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/detail/timeline/item/MessageTextItem.kt @@ -10,6 +10,7 @@ package im.vector.app.features.home.room.detail.timeline.item import android.text.Spanned import android.text.method.MovementMethod import android.view.ViewStub +import android.widget.TextView import androidx.appcompat.widget.AppCompatTextView import androidx.core.text.PrecomputedTextCompat import androidx.core.view.isVisible @@ -63,6 +64,9 @@ abstract class MessageTextItem : AbsMessageItem() { @EpoxyAttribute var useRichTextEditorStyle: Boolean = false + @EpoxyAttribute + var originalMessage: EpoxyCharSequence? = null + private val previewUrlViewUpdater = PreviewUrlViewUpdater() override fun bind(holder: Holder) { @@ -105,6 +109,30 @@ abstract class MessageTextItem : AbsMessageItem() { messageView.onLongClickIgnoringLinks(attributes.itemLongClickListener) messageView.setTextWithEmojiSupport(message?.charSequence, bindingOptions) markwonPlugins?.forEach { plugin -> plugin.afterSetText(messageView) } + + // Translation toggle + val toggleView = holder.translationToggle + val originalTextView = holder.originalTextView + if (originalMessage != null) { + toggleView.isVisible = true + toggleView.text = holder.view.context.getString(R.string.translation_toggle_show_original) + originalTextView.isVisible = false + originalTextView.text = originalMessage?.charSequence + toggleView.setOnClickListener { + val isShowing = originalTextView.isVisible + if (isShowing) { + originalTextView.isVisible = false + toggleView.text = holder.view.context.getString(R.string.translation_toggle_show_original) + } else { + originalTextView.isVisible = true + toggleView.text = holder.view.context.getString(R.string.translation_toggle_hide_original) + } + } + } else { + toggleView.isVisible = false + originalTextView.isVisible = false + toggleView.setOnClickListener(null) + } } private fun AppCompatTextView.setTextWithEmojiSupport(message: CharSequence?, bindingOptions: BindingOptions?) { @@ -128,6 +156,8 @@ abstract class MessageTextItem : AbsMessageItem() { class Holder : AbsMessageItem.Holder(STUB_ID) { val previewUrlView by bind(R.id.messageUrlPreview) + val translationToggle by bind(R.id.messageTranslationToggle) + val originalTextView by bind(R.id.messageOriginalText) private val richMessageStub by bind(R.id.richMessageTextViewStub) private val plainMessageStub by bind(R.id.plainMessageTextViewStub) var richMessageView: EditorStyledTextView? = null diff --git a/vector/src/main/java/im/vector/app/features/home/room/list/RoomSummaryItemFactory.kt b/vector/src/main/java/im/vector/app/features/home/room/list/RoomSummaryItemFactory.kt index 57cd955cb85..41a5395c0e9 100644 --- a/vector/src/main/java/im/vector/app/features/home/room/list/RoomSummaryItemFactory.kt +++ b/vector/src/main/java/im/vector/app/features/home/room/list/RoomSummaryItemFactory.kt @@ -14,23 +14,29 @@ import im.vector.app.core.date.DateFormatKind import im.vector.app.core.date.VectorDateFormatter import im.vector.app.core.epoxy.VectorEpoxyModel import im.vector.app.core.error.ErrorFormatter +import im.vector.app.core.resources.ColorProvider import im.vector.app.core.resources.StringProvider import im.vector.app.features.home.AvatarRenderer import im.vector.app.features.home.RoomListDisplayMode import im.vector.app.features.home.room.detail.timeline.format.DisplayableEventFormatter import im.vector.app.features.home.room.list.usecase.GetLatestPreviewableEventUseCase import im.vector.app.features.home.room.typing.TypingHelper +import im.vector.app.features.translation.TimelineTranslationManager +import me.gujun.android.span.span import im.vector.app.features.voicebroadcast.isLive import im.vector.app.features.voicebroadcast.model.asVoiceBroadcastEvent import im.vector.lib.core.utils.epoxy.charsequence.toEpoxyCharSequence import im.vector.lib.strings.CommonPlurals import im.vector.lib.strings.CommonStrings import org.matrix.android.sdk.api.extensions.orFalse +import org.matrix.android.sdk.api.session.events.model.toModel import org.matrix.android.sdk.api.session.room.members.ChangeMembershipState import org.matrix.android.sdk.api.session.room.model.Membership import org.matrix.android.sdk.api.session.room.model.RoomSummary import org.matrix.android.sdk.api.session.room.model.SpaceChildInfo +import org.matrix.android.sdk.api.session.room.model.message.MessageContent import org.matrix.android.sdk.api.util.toMatrixItem +import timber.log.Timber import javax.inject.Inject class RoomSummaryItemFactory @Inject constructor( @@ -40,7 +46,9 @@ class RoomSummaryItemFactory @Inject constructor( private val typingHelper: TypingHelper, private val avatarRenderer: AvatarRenderer, private val errorFormatter: ErrorFormatter, + private val colorProvider: ColorProvider, private val getLatestPreviewableEventUseCase: GetLatestPreviewableEventUseCase, + private val timelineTranslationManager: TimelineTranslationManager, ) { fun create( @@ -130,6 +138,28 @@ class RoomSummaryItemFactory @Inject constructor( if (latestEvent != null) { latestFormattedEvent = displayableEventFormatter.format(latestEvent, roomSummary.isDirect, roomSummary.isDirect.not()) latestEventTime = dateFormatter.format(latestEvent.root.originServerTs, DateFormatKind.ROOM_LIST) + + // Check if we can use a cached translation for the preview + val clearContent = latestEvent.root.getClearContent() + val msgContent = clearContent?.toModel() + val body = msgContent?.body + Timber.d("TRANSLATION_DEBUG Room list: room=${roomSummary.displayName} clearContent=${clearContent != null} msgContent=${msgContent != null} body=${body?.take(50)} autoTranslate=${timelineTranslationManager.getCachedTranslation(body ?: "") != null}") + if (body != null) { + val translatedPreview = timelineTranslationManager.getCachedTranslation(body) + if (translatedPreview != null) { + Timber.d("TRANSLATION_DEBUG Room list preview translated for room ${roomSummary.displayName}: $translatedPreview") + latestFormattedEvent = if (roomSummary.isDirect.not()) { + // For non-DM rooms, prepend the sender name with proper styling like DisplayableEventFormatter.simpleFormat() + val senderName = latestEvent.senderInfo.disambiguatedDisplayName + span { + text = senderName + textColor = colorProvider.getColorFromAttribute(im.vector.lib.ui.styles.R.attr.vctr_content_primary) + }.append(": ").append(translatedPreview) + } else { + translatedPreview + } + } + } } val typingMessage = typingHelper.getTypingMessage(roomSummary.typingUsers) diff --git a/vector/src/main/java/im/vector/app/features/notifications/NotifiableEventResolver.kt b/vector/src/main/java/im/vector/app/features/notifications/NotifiableEventResolver.kt index dde6fa69380..d566ce427c8 100644 --- a/vector/src/main/java/im/vector/app/features/notifications/NotifiableEventResolver.kt +++ b/vector/src/main/java/im/vector/app/features/notifications/NotifiableEventResolver.kt @@ -14,6 +14,8 @@ import im.vector.app.core.resources.StringProvider import im.vector.app.features.displayname.getBestName import im.vector.app.features.home.room.detail.timeline.format.DisplayableEventFormatter import im.vector.app.features.home.room.detail.timeline.format.NoticeEventFormatter +import im.vector.app.features.translation.TranslateConfig +import im.vector.app.features.translation.TranslationCache import im.vector.lib.core.utils.timer.Clock import im.vector.lib.strings.CommonStrings import org.matrix.android.sdk.api.extensions.orFalse @@ -57,8 +59,19 @@ class NotifiableEventResolver @Inject constructor( private val displayableEventFormatter: DisplayableEventFormatter, private val clock: Clock, private val buildMeta: BuildMeta, + private val translateConfig: TranslateConfig, + private val translationCache: TranslationCache, ) { + /** + * Only use cached translations for notifications to avoid blocking delivery with network calls. + * If no cached translation exists, the original text is returned. + */ + private fun translateBodyIfEnabled(body: String): String { + if (!translateConfig.enabled || !translateConfig.autoTranslate) return body + return translationCache.get(body, translateConfig.targetLanguage) ?: body + } + suspend fun resolveEvent(event: Event, session: Session, isNoisy: Boolean): NotifiableEvent? { val roomID = event.roomId ?: return null val eventId = event.eventId ?: return null @@ -130,7 +143,7 @@ class NotifiableEventResolver @Inject constructor( return if (room == null) { Timber.e("## Unable to resolve room for eventId [$event]") // Ok room is not known in store, but we can still display something - val body = displayableEventFormatter.format(event, isDm = false, appendAuthor = false) + val body = translateBodyIfEnabled(displayableEventFormatter.format(event, isDm = false, appendAuthor = false).toString()) val roomName = stringProvider.getString(CommonStrings.notification_unknown_room_name) val senderDisplayName = event.senderInfo.disambiguatedDisplayName @@ -142,7 +155,7 @@ class NotifiableEventResolver @Inject constructor( noisy = isNoisy, senderName = senderDisplayName, senderId = event.root.senderId, - body = body.toString(), + body = body, imageUriString = event.fetchImageIfPresent(session)?.toString(), roomId = event.root.roomId!!, threadId = event.root.getRootThreadEventId(), @@ -156,7 +169,7 @@ class NotifiableEventResolver @Inject constructor( event.root.getClearContent()?.toModel()?.isUserMentioned(session.myUserId) == true when { isIncomingElementCall || event.root.supportsNotification() -> { - val body = displayableEventFormatter.format(event, isDm = room.roomSummary()?.isDirect.orFalse(), appendAuthor = false).toString() + val body = translateBodyIfEnabled(displayableEventFormatter.format(event, isDm = room.roomSummary()?.isDirect.orFalse(), appendAuthor = false).toString()) val roomName = room.roomSummary()?.displayName ?: "" val senderDisplayName = event.senderInfo.disambiguatedDisplayName diff --git a/vector/src/main/java/im/vector/app/features/notifications/NotificationDrawerManager.kt b/vector/src/main/java/im/vector/app/features/notifications/NotificationDrawerManager.kt index 52a233b014a..10e19541124 100644 --- a/vector/src/main/java/im/vector/app/features/notifications/NotificationDrawerManager.kt +++ b/vector/src/main/java/im/vector/app/features/notifications/NotificationDrawerManager.kt @@ -226,6 +226,20 @@ class NotificationDrawerManager @Inject constructor( return resolvedEvent.shouldIgnoreMessageEventInRoom(currentRoomId, currentThreadId) } + /** + * Returns a snapshot of the current queued message notification events. + * Used by the notification summary feature to generate an AI summary. + */ + fun getQueuedMessageEvents(): List { + val result = mutableListOf() + notificationState.queuedEvents { queuedEvents -> + queuedEvents.rawEvents().filterIsInstance().forEach { + result.add(it) + } + } + return result + } + companion object { const val SUMMARY_NOTIFICATION_ID = 0 const val ROOM_MESSAGES_NOTIFICATION_ID = 1 diff --git a/vector/src/main/java/im/vector/app/features/translation/TimelineTranslationManager.kt b/vector/src/main/java/im/vector/app/features/translation/TimelineTranslationManager.kt new file mode 100644 index 00000000000..0414aac81e0 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/TimelineTranslationManager.kt @@ -0,0 +1,250 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import android.text.Html +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.cancel +import kotlinx.coroutines.launch +import java.lang.ref.WeakReference +import timber.log.Timber +import javax.inject.Inject +import javax.inject.Singleton + +/** + * Manages async translations for timeline messages. + * When auto-translate is on, messages are translated in the background + * and a listener is notified to refresh the timeline. + */ +@Singleton +class TimelineTranslationManager @Inject constructor( + private val translationService: TranslationService, + private val translateConfig: TranslateConfig, + private val translationCache: TranslationCache +) { + private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO) + private val pendingTranslations = mutableSetOf() + private val pendingLock = Any() + private var listenerRef: WeakReference? = null + + interface TranslationListener { + fun onTranslationReady(eventId: String) + } + + fun setListener(listener: TranslationListener?) { + listenerRef = listener?.let { WeakReference(it) } + } + + /** + * Cancel pending coroutines. Good practice even though singletons live for app lifetime. + */ + fun destroy() { + scope.cancel() + } + + /** + * Strip the reply fallback from a Matrix plain-text body. + * Reply fallback lines start with "> " and are followed by a blank line. + */ + private fun stripReplyFallback(body: String): String { + val lines = body.lines() + var i = 0 + while (i < lines.size && lines[i].startsWith("> ")) { + i++ + } + // Skip the blank line after the reply fallback + if (i > 0 && i < lines.size && lines[i].isBlank()) { + i++ + } + return if (i > 0) lines.drop(i).joinToString("\n") else body + } + + /** + * Extract the quoted text from a reply fallback. + * Reply fallback lines start with "> ". The first line is usually "> <@user:server> " + * and subsequent lines contain the actual quoted text. + * Returns the quoted text (without "> " prefix) or null if no reply fallback found. + */ + private fun extractReplyQuote(body: String): String? { + val lines = body.lines() + if (lines.isEmpty() || !lines[0].startsWith("> ")) return null + val quoteLines = mutableListOf() + for (line in lines) { + if (!line.startsWith("> ")) break + quoteLines.add(line.removePrefix("> ")) + } + if (quoteLines.isEmpty()) return null + // The first line often contains the user mention like "<@user:server> quoted text" + // The quoted text may be on the SAME line after the mention, or on subsequent lines. + val firstLine = quoteLines[0] + val processedFirst = if (firstLine.startsWith("<@") || firstLine.startsWith("* <@")) { + // Extract text after the mention on the same line: "<@user:server> quoted text" → "quoted text" + val afterMention = firstLine.replace(Regex("^\\*?\\s*<@[^>]+>\\s*"), "") + afterMention.ifBlank { null } + } else { + firstLine + } + val textLines = mutableListOf() + if (processedFirst != null) textLines.add(processedFirst) + if (quoteLines.size > 1) textLines.addAll(quoteLines.drop(1)) + val quoteText = textLines.joinToString("\n").trim() + return if (quoteText.isNotBlank()) quoteText else null + } + + private fun replyQuoteCacheKey(eventId: String) = "reply_quote:$eventId" + + /** + * Replace Matrix user IDs like @user:server.org with just "user". + */ + private fun stripMatrixIds(text: String): String { + return text.replace(Regex("@([^:]+):[^\\s]+")) { matchResult -> + matchResult.groupValues[1] + } + } + + /** + * Strip HTML tags and decode HTML entities from text. + */ + private fun stripHtml(text: String): String { + @Suppress("DEPRECATION") + return Html.fromHtml(text).toString().trim() + } + + /** + * Clean text before sending to the LLM for translation. + */ + fun cleanTextForTranslation(body: String): String { + var cleaned = stripReplyFallback(body) + cleaned = stripMatrixIds(cleaned) + // If text still contains HTML tags, strip them + if (cleaned.contains(Regex("<[^>]+>"))) { + cleaned = stripHtml(cleaned) + } + // Decode any remaining HTML entities + if (cleaned.contains("&") && cleaned.contains(";")) { + @Suppress("DEPRECATION") + cleaned = Html.fromHtml(cleaned).toString().trim() + } + return cleaned.trim() + } + + /** + * Returns the cached translation if available, otherwise triggers async translation. + * @return translated text or null if not yet available + */ + fun getTranslatedText(eventId: String, body: String): String? { + if (!translateConfig.enabled || !translateConfig.autoTranslate) return null + val targetLang = translateConfig.targetLanguage + + val cleanedBody = cleanTextForTranslation(body) + if (cleanedBody.isBlank()) return null + + // Extract reply quote (needed for both cached and non-cached paths) + val replyQuote = extractReplyQuote(body) + val cleanedQuote = if (replyQuote != null) { + var cq = stripMatrixIds(replyQuote) + if (cq.contains(Regex("<[^>]+>"))) cq = stripHtml(cq) + if (cq.contains("&") && cq.contains(";")) { + @Suppress("DEPRECATION") + cq = Html.fromHtml(cq).toString().trim() + } + cq.takeIf { it.isNotBlank() } + } else null + + // Check cache using cleaned body + val cached = translationCache.get(cleanedBody, targetLang) + if (cached != null) { + // Body is cached, but we still need to ensure the reply quote is translated + if (cleanedQuote != null) { + val quoteCacheKey = replyQuoteCacheKey(eventId) + if (translationCache.get(quoteCacheKey, targetLang) == null) { + // Reply quote not yet translated — do it async + scope.launch { + try { + val translatedQuote = translationService.translate(cleanedQuote, targetLang) + if (translatedQuote != null) { + translationCache.put(quoteCacheKey, targetLang, translatedQuote) + Timber.d("TRANSLATION_DEBUG Reply quote CACHED (late) for event $eventId: $translatedQuote") + kotlinx.coroutines.withContext(Dispatchers.Main) { + listenerRef?.get()?.onTranslationReady(eventId) + } + } + } catch (e: Exception) { + Timber.e(e, "Reply quote translation failed for event $eventId") + } + } + } + } + return cached + } + + // Trigger async translation if not already pending + synchronized(pendingLock) { + if (pendingTranslations.contains(eventId)) return null + pendingTranslations.add(eventId) + } + + scope.launch { + try { + val translated = translationService.translate(cleanedBody, targetLang) + + // Also translate the reply quote if present and not already cached + if (cleanedQuote != null) { + val quoteCacheKey = replyQuoteCacheKey(eventId) + if (translationCache.get(quoteCacheKey, targetLang) == null) { + try { + val translatedQuote = translationService.translate(cleanedQuote, targetLang) + if (translatedQuote != null) { + translationCache.put(quoteCacheKey, targetLang, translatedQuote) + Timber.d("TRANSLATION_DEBUG Reply quote CACHED for event $eventId: $translatedQuote") + } + } catch (e: Exception) { + Timber.e(e, "Reply quote translation failed for event $eventId") + } + } + } + + if (translated != null) { + Timber.d("Translation ready for event $eventId") + // Notify on main thread + kotlinx.coroutines.withContext(Dispatchers.Main) { + listenerRef?.get()?.onTranslationReady(eventId) + } + } + } catch (e: Exception) { + Timber.e(e, "Async translation failed for event $eventId") + } finally { + synchronized(pendingLock) { + pendingTranslations.remove(eventId) + } + } + } + return null + } + + /** + * Check if a translation is available in cache (synchronous, no async trigger). + */ + fun getCachedTranslation(body: String): String? { + if (!translateConfig.enabled || !translateConfig.autoTranslate) return null + val cleanedBody = cleanTextForTranslation(body) + if (cleanedBody.isBlank()) return null + return translationCache.get(cleanedBody, translateConfig.targetLanguage) + } + + /** + * Get the translated reply quote for a given event, if available in cache. + * This is the quote text that was translated alongside the main body in getTranslatedText(). + */ + fun getTranslatedReplyQuote(eventId: String): String? { + if (!translateConfig.enabled || !translateConfig.autoTranslate) return null + return translationCache.get(replyQuoteCacheKey(eventId), translateConfig.targetLanguage) + } +} diff --git a/vector/src/main/java/im/vector/app/features/translation/TranslateApi.kt b/vector/src/main/java/im/vector/app/features/translation/TranslateApi.kt new file mode 100644 index 00000000000..1044e58b3cc --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/TranslateApi.kt @@ -0,0 +1,56 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import com.squareup.moshi.Json +import com.squareup.moshi.JsonClass +import retrofit2.http.Body +import retrofit2.http.GET +import retrofit2.http.POST + +interface TranslateApi { + @POST("chat/completions") + suspend fun translate(@Body request: ChatCompletionRequest): ChatCompletionResponse + + @GET("models") + suspend fun listModels(): ModelsResponse +} + +@JsonClass(generateAdapter = true) +data class ChatCompletionRequest( + @Json(name = "model") val model: String, + @Json(name = "messages") val messages: List, + @Json(name = "temperature") val temperature: Double = 0.1, + @Json(name = "max_tokens") val maxTokens: Int = 2048 +) + +@JsonClass(generateAdapter = true) +data class ChatMessage( + @Json(name = "role") val role: String, + @Json(name = "content") val content: String +) + +@JsonClass(generateAdapter = true) +data class ChatCompletionResponse( + @Json(name = "choices") val choices: List +) + +@JsonClass(generateAdapter = true) +data class ChatChoice( + @Json(name = "message") val message: ChatMessage +) + +@JsonClass(generateAdapter = true) +data class ModelsResponse( + @Json(name = "data") val data: List +) + +@JsonClass(generateAdapter = true) +data class ModelInfo( + @Json(name = "id") val id: String +) diff --git a/vector/src/main/java/im/vector/app/features/translation/TranslateConfig.kt b/vector/src/main/java/im/vector/app/features/translation/TranslateConfig.kt new file mode 100644 index 00000000000..fec210592b2 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/TranslateConfig.kt @@ -0,0 +1,105 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import android.content.Context +import android.content.SharedPreferences +import androidx.core.content.edit +import androidx.security.crypto.EncryptedSharedPreferences +import androidx.security.crypto.MasterKey +import im.vector.app.core.di.DefaultPreferences +import timber.log.Timber +import javax.inject.Inject +import javax.inject.Singleton + +@Singleton +class TranslateConfig @Inject constructor( + @DefaultPreferences private val prefs: SharedPreferences, + private val context: Context +) { + companion object { + private const val KEY_API_URL = "translate_api_url" + private const val KEY_API_KEY = "translate_api_key" + private const val KEY_MODEL = "translate_model" + private const val KEY_TARGET_LANGUAGE = "translate_target_language" + private const val KEY_ROOM_LANGUAGE = "translate_room_language" + private const val KEY_ENABLED = "translate_enabled" + private const val KEY_AUTO_TRANSLATE = "translate_auto_translate" + private const val KEY_REFORMULATION_ENABLED = "translate_reformulation_enabled" + private const val KEY_SUMMARY_ENABLED = "translate_summary_enabled" + private const val KEY_SUGGESTED_REPLIES_ENABLED = "translate_suggested_replies_enabled" + private const val KEY_NOTIFICATION_SUMMARY_ENABLED = "translate_notification_summary_enabled" + } + + /** + * Secure SharedPreferences for storing sensitive data like API keys. + * Falls back to regular SharedPreferences if EncryptedSharedPreferences fails + * (can happen on some devices with Keystore issues). + */ + private val securePrefs: SharedPreferences by lazy { + try { + val masterKey = MasterKey.Builder(context) + .setKeyScheme(MasterKey.KeyScheme.AES256_GCM) + .build() + EncryptedSharedPreferences.create( + context, + "translate_secure_prefs", + masterKey, + EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV, + EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM + ) + } catch (e: Exception) { + Timber.e(e, "Failed to create EncryptedSharedPreferences, falling back to regular prefs") + prefs + } + } + + var apiUrl: String + get() = prefs.getString(KEY_API_URL, "http://localhost:11434/v1") ?: "http://localhost:11434/v1" + set(value) = prefs.edit { putString(KEY_API_URL, value) } + + var apiKey: String + get() = securePrefs.getString(KEY_API_KEY, "") ?: "" + set(value) = securePrefs.edit { putString(KEY_API_KEY, value) } + + var model: String + get() = prefs.getString(KEY_MODEL, "llama3") ?: "llama3" + set(value) = prefs.edit { putString(KEY_MODEL, value) } + + var targetLanguage: String + get() = prefs.getString(KEY_TARGET_LANGUAGE, "French") ?: "French" + set(value) = prefs.edit { putString(KEY_TARGET_LANGUAGE, value) } + + var roomLanguage: String + get() = prefs.getString(KEY_ROOM_LANGUAGE, "") ?: "" + set(value) = prefs.edit { putString(KEY_ROOM_LANGUAGE, value) } + + var enabled: Boolean + get() = prefs.getBoolean(KEY_ENABLED, false) + set(value) = prefs.edit { putBoolean(KEY_ENABLED, value) } + + var autoTranslate: Boolean + get() = prefs.getBoolean(KEY_AUTO_TRANSLATE, false) + set(value) = prefs.edit { putBoolean(KEY_AUTO_TRANSLATE, value) } + + var reformulationEnabled: Boolean + get() = prefs.getBoolean(KEY_REFORMULATION_ENABLED, true) + set(value) = prefs.edit { putBoolean(KEY_REFORMULATION_ENABLED, value) } + + var summaryEnabled: Boolean + get() = prefs.getBoolean(KEY_SUMMARY_ENABLED, true) + set(value) = prefs.edit { putBoolean(KEY_SUMMARY_ENABLED, value) } + + var suggestedRepliesEnabled: Boolean + get() = prefs.getBoolean(KEY_SUGGESTED_REPLIES_ENABLED, true) + set(value) = prefs.edit { putBoolean(KEY_SUGGESTED_REPLIES_ENABLED, value) } + + var notificationSummaryEnabled: Boolean + get() = prefs.getBoolean(KEY_NOTIFICATION_SUMMARY_ENABLED, true) + set(value) = prefs.edit { putBoolean(KEY_NOTIFICATION_SUMMARY_ENABLED, value) } +} diff --git a/vector/src/main/java/im/vector/app/features/translation/TranslationCache.kt b/vector/src/main/java/im/vector/app/features/translation/TranslationCache.kt new file mode 100644 index 00000000000..ee2dd8db7cd --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/TranslationCache.kt @@ -0,0 +1,133 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import android.content.Context +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.Job +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.cancel +import kotlinx.coroutines.delay +import kotlinx.coroutines.launch +import org.json.JSONArray +import org.json.JSONObject +import timber.log.Timber +import java.io.File +import javax.inject.Inject +import javax.inject.Singleton + +// TODO: Translation cache stores translations in plain text on disk. +// For E2EE rooms, ideally we should skip disk caching or encrypt the cache file. +// This requires knowing whether a room is encrypted at cache-put time, which adds complexity. +// For now, be aware that translated message content is persisted unencrypted. +@Singleton +class TranslationCache @Inject constructor( + private val context: Context +) { + private val memoryCache = LinkedHashMap(5000, 0.75f, true) + private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO) + private var saveDiskJob: Job? = null + + companion object { + private const val MAX_SIZE = 5000 + private const val CACHE_FILE = "translate-cache.json" + private const val SAVE_DEBOUNCE_MS = 2000L + } + + private fun key(text: String, lang: String) = "$lang:$text" + + fun get(text: String, lang: String): String? { + return synchronized(memoryCache) { memoryCache[key(text, lang)] } + } + + fun put(text: String, lang: String, translated: String) { + synchronized(memoryCache) { + if (memoryCache.size >= MAX_SIZE) { + val firstKey = memoryCache.keys.firstOrNull() + if (firstKey != null) { + memoryCache.remove(firstKey) + } + } + memoryCache[key(text, lang)] = translated + } + scheduleDiskSave() + } + + fun clear() { + synchronized(memoryCache) { memoryCache.clear() } + scope.launch { + try { + getCacheFile().delete() + } catch (e: Exception) { + Timber.e(e, "Failed to delete cache file") + } + } + } + + fun size(): Int = synchronized(memoryCache) { memoryCache.size } + + /** + * Cancel pending coroutines. Good practice even though singletons live for app lifetime. + */ + fun destroy() { + scope.cancel() + } + + fun loadFromDisk() { + scope.launch { + try { + val file = getCacheFile() + if (!file.exists()) return@launch + val json = file.readText() + val array = JSONArray(json) + synchronized(memoryCache) { + for (i in 0 until array.length()) { + val obj = array.getJSONObject(i) + memoryCache[obj.getString("k")] = obj.getString("v") + } + } + Timber.d("Loaded ${array.length()} translations from disk cache") + } catch (e: Exception) { + Timber.e(e, "Failed to load translation cache from disk") + } + } + } + + private fun scheduleDiskSave() { + saveDiskJob?.cancel() + saveDiskJob = scope.launch { + delay(SAVE_DEBOUNCE_MS) + saveToDisk() + } + } + + private fun saveToDisk() { + try { + val entries = synchronized(memoryCache) { + memoryCache.map { (k, v) -> k to v } + } + val array = JSONArray() + entries.forEach { (k, v) -> + array.put(JSONObject().apply { + put("k", k) + put("v", v) + }) + } + // Atomic write: write to temp file first, then rename + val tempFile = File(context.filesDir, "$CACHE_FILE.tmp") + tempFile.writeText(array.toString()) + tempFile.renameTo(getCacheFile()) + Timber.d("Saved ${entries.size} translations to disk cache") + } catch (e: Exception) { + Timber.e(e, "Failed to save translation cache to disk") + } + } + + private fun getCacheFile(): File = File(context.filesDir, CACHE_FILE) +} diff --git a/vector/src/main/java/im/vector/app/features/translation/TranslationRateLimiter.kt b/vector/src/main/java/im/vector/app/features/translation/TranslationRateLimiter.kt new file mode 100644 index 00000000000..c81d29d0e93 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/TranslationRateLimiter.kt @@ -0,0 +1,26 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import kotlinx.coroutines.sync.Semaphore +import javax.inject.Inject +import javax.inject.Singleton + +@Singleton +class TranslationRateLimiter @Inject constructor() { + private val semaphore = Semaphore(3) + + suspend fun execute(block: suspend () -> T): T { + semaphore.acquire() + return try { + block() + } finally { + semaphore.release() + } + } +} diff --git a/vector/src/main/java/im/vector/app/features/translation/TranslationService.kt b/vector/src/main/java/im/vector/app/features/translation/TranslationService.kt new file mode 100644 index 00000000000..68a673f5741 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/TranslationService.kt @@ -0,0 +1,177 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import com.squareup.moshi.Moshi +import okhttp3.Interceptor +import okhttp3.OkHttpClient +import retrofit2.Retrofit +import retrofit2.converter.moshi.MoshiConverterFactory +import timber.log.Timber +import java.util.concurrent.TimeUnit +import javax.inject.Inject +import javax.inject.Singleton + +@Singleton +class TranslationService @Inject constructor( + private val config: TranslateConfig, + private val cache: TranslationCache, + private val rateLimiter: TranslationRateLimiter +) { + companion object { + /** Maximum number of characters to send to the API. */ + private const val MAX_INPUT_LENGTH = 4000 + } + + private var currentApiUrl: String? = null + private var currentApiKey: String? = null + private var api: TranslateApi? = null + private var currentClient: OkHttpClient? = null + private val apiLock = Any() + + private val moshi: Moshi by lazy { + Moshi.Builder().build() + } + + init { + cache.loadFromDisk() + } + + private fun getApi(): TranslateApi { + synchronized(apiLock) { + val url = config.apiUrl.trimEnd('/') + val key = config.apiKey + + // Enforce HTTPS for non-localhost URLs + if (!isLocalhost(url) && !url.startsWith("https://")) { + throw IllegalArgumentException("Non-localhost API URLs must use HTTPS") + } + + if (api == null || currentApiUrl != url || currentApiKey != key) { + // Shutdown previous client to avoid resource leaks + currentClient?.dispatcher?.executorService?.shutdown() + currentClient?.connectionPool?.evictAll() + + currentApiUrl = url + currentApiKey = key + val client = OkHttpClient.Builder() + .connectTimeout(30, TimeUnit.SECONDS) + .readTimeout(60, TimeUnit.SECONDS) + .writeTimeout(30, TimeUnit.SECONDS) + .apply { + if (key.isNotBlank()) { + addInterceptor(Interceptor { chain -> + val request = chain.request().newBuilder() + .addHeader("Authorization", "Bearer $key") + .build() + chain.proceed(request) + }) + } + } + .build() + currentClient = client + + val baseUrl = if (url.endsWith("/")) url else "$url/" + api = Retrofit.Builder() + .baseUrl(baseUrl) + .client(client) + .addConverterFactory(MoshiConverterFactory.create(moshi)) + .build() + .create(TranslateApi::class.java) + } + return api!! + } + } + + private fun isLocalhost(url: String): Boolean { + return url.contains("localhost") || url.contains("127.0.0.1") || url.contains("0.0.0.0") + } + + suspend fun translate(text: String, targetLanguage: String): String? { + if (text.isBlank()) return text + + // Check cache first + cache.get(text, targetLanguage)?.let { return it } + + // Truncate input to limit + val truncatedText = text.take(MAX_INPUT_LENGTH) + + return try { + rateLimiter.execute { + val request = ChatCompletionRequest( + model = config.model, + messages = listOf( + ChatMessage( + role = "system", + content = "You are a translator. Translate the following message to $targetLanguage. " + + "Reply ONLY with the translation, no explanations, no quotes, no extra text. " + + "If the text is already in $targetLanguage, return it as-is. " + + "Do NOT include any HTML tags, XML tags, or HTML entities in your response. " + + "Return only plain text. No

,
, ", &, etc." + ), + ChatMessage( + role = "user", + content = truncatedText + ) + ) + ) + val response = getApi().translate(request) + val translated = response.choices.firstOrNull()?.message?.content?.trim() + if (translated != null) { + cache.put(text, targetLanguage, translated) + } + translated + } + } catch (e: Exception) { + Timber.e(e, "Translation failed") + null + } + } + + suspend fun translateOutgoing(text: String): String { + val roomLang = config.roomLanguage + if (!config.enabled || roomLang.isBlank()) return text + return translate(text, roomLang) ?: text + } + + suspend fun testConnection(): Result> { + return try { + val response = getApi().listModels() + Result.success(response.data.map { it.id }) + } catch (e: Exception) { + Timber.e(e, "Test connection failed") + Result.failure(e) + } + } + + suspend fun complete(systemPrompt: String, userMessage: String): String? { + // Truncate input to limit + val truncatedMessage = userMessage.take(MAX_INPUT_LENGTH) + + return try { + rateLimiter.execute { + val request = ChatCompletionRequest( + model = config.model, + messages = listOf( + ChatMessage(role = "system", content = systemPrompt), + ChatMessage(role = "user", content = truncatedMessage) + ) + ) + val response = getApi().translate(request) + response.choices.firstOrNull()?.message?.content?.trim() + } + } catch (e: Exception) { + Timber.e(e, "AI completion failed") + null + } + } + + fun getCacheSize(): Int = cache.size() + + fun clearCache() = cache.clear() +} diff --git a/vector/src/main/java/im/vector/app/features/translation/VectorSettingsTranslationFragment.kt b/vector/src/main/java/im/vector/app/features/translation/VectorSettingsTranslationFragment.kt new file mode 100644 index 00000000000..02892673826 --- /dev/null +++ b/vector/src/main/java/im/vector/app/features/translation/VectorSettingsTranslationFragment.kt @@ -0,0 +1,172 @@ +/* + * Copyright 2024 New Vector Ltd. + * + * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial + * Please see LICENSE files in the repository root for full details. + */ + +package im.vector.app.features.translation + +import android.os.Bundle +import android.widget.Toast +import androidx.lifecycle.lifecycleScope +import androidx.preference.EditTextPreference +import androidx.preference.Preference +import androidx.preference.SwitchPreference +import dagger.hilt.android.AndroidEntryPoint +import im.vector.app.R +import im.vector.app.features.settings.VectorSettingsBaseFragment +import kotlinx.coroutines.launch +import javax.inject.Inject + +@AndroidEntryPoint +class VectorSettingsTranslationFragment : VectorSettingsBaseFragment() { + + override var titleRes: Int = R.string.settings_translation + override val preferenceXmlRes = R.xml.vector_settings_translation + + @Inject lateinit var translateConfig: TranslateConfig + @Inject lateinit var translationService: TranslationService + + private val enabledPref by lazy { findPreference("translate_enabled")!! } + private val autoTranslatePref by lazy { findPreference("translate_auto_translate")!! } + private val apiUrlPref by lazy { findPreference("translate_api_url")!! } + private val apiKeyPref by lazy { findPreference("translate_api_key")!! } + private val modelPref by lazy { findPreference("translate_model")!! } + private val targetLanguagePref by lazy { findPreference("translate_target_language")!! } + private val roomLanguagePref by lazy { findPreference("translate_room_language")!! } + private val testConnectionPref by lazy { findPreference("translate_test_connection")!! } + private val cacheInfoPref by lazy { findPreference("translate_cache_info")!! } + private val clearCachePref by lazy { findPreference("translate_clear_cache")!! } + private val reformulationPref by lazy { findPreference("translate_reformulation_enabled")!! } + private val summaryPref by lazy { findPreference("translate_summary_enabled")!! } + private val suggestedRepliesPref by lazy { findPreference("translate_suggested_replies_enabled")!! } + private val notificationSummaryPref by lazy { findPreference("translate_notification_summary_enabled")!! } + + override fun bindPref() { + // Initialize values + enabledPref.isChecked = translateConfig.enabled + autoTranslatePref.isChecked = translateConfig.autoTranslate + apiUrlPref.text = translateConfig.apiUrl + apiUrlPref.summary = translateConfig.apiUrl + apiKeyPref.text = translateConfig.apiKey + apiKeyPref.summary = if (translateConfig.apiKey.isBlank()) "(empty)" else "****" + modelPref.text = translateConfig.model + modelPref.summary = translateConfig.model + targetLanguagePref.text = translateConfig.targetLanguage + targetLanguagePref.summary = translateConfig.targetLanguage + roomLanguagePref.text = translateConfig.roomLanguage + roomLanguagePref.summary = translateConfig.roomLanguage.ifBlank { "(disabled)" } + updateCacheInfo() + + // Listeners + enabledPref.setOnPreferenceChangeListener { _, newValue -> + translateConfig.enabled = newValue as Boolean + true + } + + autoTranslatePref.setOnPreferenceChangeListener { _, newValue -> + translateConfig.autoTranslate = newValue as Boolean + true + } + + apiUrlPref.setOnPreferenceChangeListener { _, newValue -> + val url = newValue as String + if (!url.startsWith("http://") && !url.startsWith("https://")) { + Toast.makeText(requireContext(), "URL must start with http:// or https://", Toast.LENGTH_SHORT).show() + return@setOnPreferenceChangeListener false + } + val isLocalhost = url.contains("localhost") || url.contains("127.0.0.1") || url.contains("0.0.0.0") + if (!isLocalhost && !url.startsWith("https://")) { + Toast.makeText(requireContext(), "Warning: Non-localhost URLs should use HTTPS. Message content will be sent to this server.", Toast.LENGTH_LONG).show() + } + translateConfig.apiUrl = url + apiUrlPref.summary = url + true + } + + apiKeyPref.setOnPreferenceChangeListener { _, newValue -> + val key = newValue as String + translateConfig.apiKey = key + apiKeyPref.summary = if (key.isBlank()) "(empty)" else "****" + true + } + + modelPref.setOnPreferenceChangeListener { _, newValue -> + val model = newValue as String + translateConfig.model = model + modelPref.summary = model + true + } + + targetLanguagePref.setOnPreferenceChangeListener { _, newValue -> + val lang = newValue as String + translateConfig.targetLanguage = lang + targetLanguagePref.summary = lang + true + } + + roomLanguagePref.setOnPreferenceChangeListener { _, newValue -> + val lang = newValue as String + translateConfig.roomLanguage = lang + roomLanguagePref.summary = lang.ifBlank { "(disabled)" } + true + } + + testConnectionPref.setOnPreferenceClickListener { + testConnection() + true + } + + clearCachePref.setOnPreferenceClickListener { + translationService.clearCache() + updateCacheInfo() + Toast.makeText(requireContext(), R.string.translation_cache_cleared, Toast.LENGTH_SHORT).show() + true + } + + // AI Feature toggles + reformulationPref.isChecked = translateConfig.reformulationEnabled + reformulationPref.setOnPreferenceChangeListener { _, newValue -> + translateConfig.reformulationEnabled = newValue as Boolean + true + } + + summaryPref.isChecked = translateConfig.summaryEnabled + summaryPref.setOnPreferenceChangeListener { _, newValue -> + translateConfig.summaryEnabled = newValue as Boolean + true + } + + suggestedRepliesPref.isChecked = translateConfig.suggestedRepliesEnabled + suggestedRepliesPref.setOnPreferenceChangeListener { _, newValue -> + translateConfig.suggestedRepliesEnabled = newValue as Boolean + true + } + + notificationSummaryPref.isChecked = translateConfig.notificationSummaryEnabled + notificationSummaryPref.setOnPreferenceChangeListener { _, newValue -> + translateConfig.notificationSummaryEnabled = newValue as Boolean + true + } + } + + private fun updateCacheInfo() { + cacheInfoPref.summary = getString(R.string.translation_cache_count, translationService.getCacheSize()) + } + + private fun testConnection() { + testConnectionPref.summary = getString(R.string.translation_testing) + lifecycleScope.launch { + val result = translationService.testConnection() + result.fold( + onSuccess = { models -> + testConnectionPref.summary = getString(R.string.translation_test_success, models.joinToString(", ")) + }, + onFailure = { error -> + testConnectionPref.summary = getString(R.string.translation_test_failed, error.message ?: "Unknown error") + } + ) + } + } +} diff --git a/vector/src/main/res/drawable/ic_reformulate.xml b/vector/src/main/res/drawable/ic_reformulate.xml new file mode 100644 index 00000000000..f18395c27d0 --- /dev/null +++ b/vector/src/main/res/drawable/ic_reformulate.xml @@ -0,0 +1,9 @@ + + + diff --git a/vector/src/main/res/drawable/ic_settings_root_translation.xml b/vector/src/main/res/drawable/ic_settings_root_translation.xml new file mode 100644 index 00000000000..e05cfbef020 --- /dev/null +++ b/vector/src/main/res/drawable/ic_settings_root_translation.xml @@ -0,0 +1,9 @@ + + + diff --git a/vector/src/main/res/drawable/ic_summary.xml b/vector/src/main/res/drawable/ic_summary.xml new file mode 100644 index 00000000000..0fad46ffc4c --- /dev/null +++ b/vector/src/main/res/drawable/ic_summary.xml @@ -0,0 +1,9 @@ + + + diff --git a/vector/src/main/res/drawable/ic_translate.xml b/vector/src/main/res/drawable/ic_translate.xml new file mode 100644 index 00000000000..e05cfbef020 --- /dev/null +++ b/vector/src/main/res/drawable/ic_translate.xml @@ -0,0 +1,9 @@ + + + diff --git a/vector/src/main/res/layout/item_timeline_event_text_message_stub.xml b/vector/src/main/res/layout/item_timeline_event_text_message_stub.xml index 32785a41aff..47f58c6706b 100644 --- a/vector/src/main/res/layout/item_timeline_event_text_message_stub.xml +++ b/vector/src/main/res/layout/item_timeline_event_text_message_stub.xml @@ -19,6 +19,28 @@ android:layout_height="wrap_content" android:layout="@layout/item_timeline_event_text_message_rich_stub" /> + + + + + + + diff --git a/vector/src/main/res/values/strings_translation.xml b/vector/src/main/res/values/strings_translation.xml new file mode 100644 index 00000000000..3644d50d6cf --- /dev/null +++ b/vector/src/main/res/values/strings_translation.xml @@ -0,0 +1,85 @@ + + + + + Traduction & IA + + Traduction + Activer la traduction + Traduction automatique + Traduire automatiquement tous les messages entrants + + Configuration API + URL de l\'API + Cl\u00e9 API + Bearer token (optionnel pour les mod\u00e8les locaux) + Mod\u00e8le + Tester la connexion + Appuyer pour tester la connexion API + Test en cours\u2026 + Connect\u00e9 ! Mod\u00e8les : %1$s + \u00c9chec : %1$s + + Langues + Langue cible + Langue vers laquelle traduire les messages entrants + Langue du salon (sortant) + Traduire les messages sortants vers cette langue (vide = d\u00e9sactiv\u00e9) + + Cache + Info cache + %1$d traductions en cache + Vider le cache de traduction + Cache de traduction vid\u00e9 + + + Traduire + Traduction en cours\u2026 + Original + Afficher l\'original + Afficher la traduction + Voir l\'original ▾ + Masquer l\'original ▴ + \u00c9chec de la traduction + + + Fonctionnalit\u00e9s IA + Reformulation + Reformuler les messages avant envoi + R\u00e9sum\u00e9 de conversation + R\u00e9sumer les conversations + R\u00e9ponses sugg\u00e9r\u00e9es + Sugg\u00e9rer des r\u00e9ponses rapides + R\u00e9sum\u00e9 des notifications + R\u00e9sumer les notifications manqu\u00e9es + + + Reformuler + Formel + D\u00e9contract\u00e9 + Concis + Corriger la grammaire + Reformulation en cours\u2026 + \u00c9chec de la reformulation + Annuler + + Aucun message texte trouv\u00e9 + + + R\u00e9sum\u00e9 de la conversation + G\u00e9n\u00e9ration du r\u00e9sum\u00e9\u2026 + \u00c9chec de la g\u00e9n\u00e9ration du r\u00e9sum\u00e9 + Copier + Fermer + R\u00e9sumer + + + Chargement des suggestions\u2026 + + + R\u00e9sum\u00e9 des notifications + G\u00e9n\u00e9ration du r\u00e9sum\u00e9\u2026 + \u00c9chec du r\u00e9sum\u00e9 + Fermer + Tout marquer comme lu + diff --git a/vector/src/main/res/xml/vector_settings_root.xml b/vector/src/main/res/xml/vector_settings_root.xml index 59548d10cb3..003501a7e7c 100644 --- a/vector/src/main/res/xml/vector_settings_root.xml +++ b/vector/src/main/res/xml/vector_settings_root.xml @@ -38,6 +38,12 @@ app:fragment="im.vector.app.features.settings.labs.VectorSettingsLabsFragment" app:isPreferenceVisible="@bool/settings_root_labs_visible" /> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +