Stabilize assistant tool execution, mail access, and chat performance

[SebastianBoehler]

Context

The Flutter assistant path has several reliability and performance issues that are now visible across cloud/API usage, local LiteRT/native usage, mail tooling, and chat rendering. This issue intentionally covers the broader backlog, not only the quick wins, so the work can be split into focused PRs while keeping the end-to-end stability goal visible.

Confirmed issues

Tool execution and model loop reliability

• Cloud tool use is currently limited to one tool round. After first-round tool calls are executed, the client sends one follow-up request and returns that response without checking for additional tool calls.
• Tool execution failures currently abort the whole assistant turn instead of returning a failed tool result that lets the model continue with partial context.
• Local tool execution uses a bracketed prompt protocol such as [TOOL:name:args] instead of a structured/native tool-call path. If the active LiteRT target supports structured tool calls, we should align with that API; if not, document the bracketed protocol as deliberate prompt-level emulation.
• The local provider reads memory through a captured snapshot, while cloud tools can read fresh memory from disk. This can produce inconsistent answers after memory changes during a turn.

Mail access and data transfer

• Every mail repository operation creates a new authenticated IMAP client, then closes it. In the mail UI, initial load performs mailbox listing and summary fetch in parallel, so it can open multiple TLS/IMAP sessions.
• Mail summaries and detail views fetch BODY.PEEK[], which downloads the full message body and can include attachments even when only headers/previews are needed.
• Mail view message-opening has no local loading/error state, so failures in opening a message are not handled inline.
• There is no meaningful mail cache across tool calls or UI refreshes.

HTTP client lifecycle

• Cloud/API message sends instantiate a new http.Client via CloudAgentClient per prompt path, and the client is never closed. This risks leaking sockets/resources and prevents connection reuse.

Rendering and cache behavior

• Chat/tool trace updates mutate top-level agent state, which rebuilds the scaffold and selected view.
• MessageList reruns compactTraceMessages over the full message list on every build.
• This is especially risky if token streaming is added or already exists on another branch, because each token tick would rebuild more UI than necessary.
• Campus data is only cached as a mounted view future; switching away/recreating the view fetches again. There is no TTL/invalidation strategy.

Proposed implementation plan

Split into small PRs:

1. Tool loop robustness

   • Add bounded multi-round cloud tool execution.
   • Convert tool exceptions into tool-result messages/traces instead of aborting the whole turn.
   • Keep failed tool calls visible in traces.
   • Make local and cloud memory reads consistent.

2. Client/session lifecycle

   • Reuse or explicitly close CloudAgentClient/http.Client.
   • Reuse an authenticated mail session within a mail view/tool turn where practical.
   • Avoid parallel duplicate IMAP logins for initial mail UI load.

3. Mail transfer efficiency

   • Fetch headers/envelope and a bounded preview for mail summaries.
   • Fetch body structure or selected text parts for message detail.
   • Avoid downloading attachments unless explicitly requested.
   • Add inline loading/error state for opening a message.

4. Caching

   • Add explicit TTL caches for mail summaries/details and Campus menus.
   • Include manual refresh/invalidation behavior.
   • Avoid caching credentials or full attachment payloads.

5. Rendering performance

   • Narrow chat updates so tool traces/message changes do not rebuild unrelated views.
   • Memoize or incrementally maintain compacted trace messages.
   • Verify behavior under streamed response updates if streaming is added/enabled.

6. Tool-call format follow-up

   • Check current LiteRT/target SDK support for structured tool calling.
   • Either migrate local tool calls to the supported native format or document why bracketed prompt calls remain necessary.

Acceptance criteria

• Cloud assistant supports at least several bounded tool rounds and stops cleanly at a configured max.
• A failing tool call does not abort the whole assistant turn; the model receives a failure result and can answer with available context.
• Local and cloud providers use consistent memory freshness semantics.
• Cloud/API client resources are reused or closed deterministically.
• Mail UI/tool calls avoid unnecessary repeated TLS + IMAP login/logout cycles within a single logical operation.
• Mail summary fetches do not download full messages or attachments.
• Opening a mail message shows loading and recoverable error states.
• Mail and Campus data have explicit cache/refresh behavior.
• Chat updates avoid rebuilding unrelated views, and trace compaction is not recomputed over the full history on every minor update.
• Tests cover multi-round tools, tool failure recovery, mail preview fetch behavior, message-open error handling, and provider memory consistency.

Notes and constraints

• Keep university credentials local to the client/device. Do not introduce hosted credential routing.
• Avoid silently hiding tool failures; expose them as trace/status data and as model-visible tool outputs.
• This issue can be implemented as multiple PRs; it should not become one large risky refactor.

Additional prompt and model-runtime issues

These concerns should be tracked with the same stability/performance backlog rather than treated as separate quick wins:

• The app sends the entire assembled system prompt with every user turn. This increases token cost/latency for cloud providers and prompt-processing cost for local models.
• System prompt/context assembly appears layered in more than one place, including Dart prompt construction and Android native prompt handling. This risks duplicated context or memory injection, especially around prompt_context.dart and MainActivity.kt/native prompt setup.
• Sampling parameters are hardcoded in local model runtime paths. For example, tool syntax and JSON-like outputs should use deterministic settings, but current defaults can leave temperature/top-p style values too loose for reproducible tool calls.
• LiteRT model generation lacks timeout guards, so a stuck model generation can hang the assistant turn instead of failing with a recoverable error.
• Default model entries do not have complete predefined download URLs and do not include integrity checks such as expected hash/size.
• lfm2.5 is listed as a default option even though there is currently no openly available LiteRT checkpoint for it, making the default catalog misleading.
• The assistant prompt instructs the model to call at least one available StudyOS tool before answering on every turn, even when no tool is needed. This adds latency and can produce unnecessary tool use.
• Tool output is not persisted across turns, so expensive tool results can be lost immediately and recomputed or omitted in later context.

Additional acceptance criteria

• Prompt/context assembly has a single clear ownership boundary. System prompt, memory, world state, timetable, and native model instructions are not duplicated across Dart and platform layers.
• Repeated full-system-prompt sending is reduced where possible, or explicitly justified for providers that require stateless requests.
• Sampling/runtime parameters are configurable by mode. Tool/JSON reproduction paths use deterministic settings; free-form answer paths can use more creative settings.
• LiteRT generation has bounded timeouts and returns actionable recoverable errors on timeout/hang.
• Default downloadable models include valid URLs only when a compatible checkpoint is available, and downloads verify integrity before use.
• Model catalog entries that are not currently downloadable as LiteRT checkpoints are hidden, marked unavailable, or moved out of defaults.
• Tool-use instructions allow direct answers when tools are unnecessary.
• Tool outputs that should remain useful across turns are persisted or summarized into durable memory/session context with size limits and privacy constraints.