Skip to content

perf(agent): stop accumulating generator-tool preliminaryResults #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LukasParke wants to merge 1 commit into
base: main
Choose a base branch
from
Open

perf(agent): stop accumulating generator-tool preliminaryResults #110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/drop-preliminary-results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@openrouter/agent': patch
---

Stop accumulating generator-tool `preliminaryResults` arrays. Yields are still broadcast live; the terminal `tool.result` event no longer copies every yield.
Comment on lines +1 to +5

@devin-ai-integration devin-ai-integration Bot Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Release note marks a consumer-visible data removal as a bug-fix-level change

The release note for this change is labelled as a bug-fix-level bump ('@openrouter/agent': patch at .changeset/drop-preliminary-results.md:2) even though the completed-tool event and generator execution result stop carrying the list of intermediate updates entirely, so anyone reading that list now silently gets nothing.
Impact: Consumers upgrading on what looks like a safe fix-level release can lose data they depend on without any warning in the changelog.

Why this conflicts with the repository's changeset bump guidance

.agents/skills/changeset-versioning/SKILL.md defines bump types: patch for "Bug fixes, type fixes, defensive coding improvements", major for "Breaking API changes". The runtime no longer populates ToolExecutionResult.preliminaryResults (packages/agent/src/lib/tool-executor.ts:383-388) nor preliminaryResults on the tool.result / tool_result events (packages/agent/src/lib/model-result.ts:970-991), while the public types at packages/agent/src/lib/tool-types.ts:1214 and packages/agent/src/lib/tool-types.ts:1368-1369 still advertise the field as available. Existing consumers reading these fields compile fine but observe undefined at runtime — a behavioral breaking change on the public event surface, not a patch-level fix.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

29 changes: 6 additions & 23 deletions packages/agent/src/lib/model-result.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -972,17 +972,13 @@ export class ModelResult<
toolName: string,
source: 'client' | 'mcp',
result: InferToolOutputsUnion<TTools>,
preliminaryResults?: InferToolEventsUnion<TTools>[],
): void {
this.toolEventBroadcaster?.push({
type: 'tool_result' as const,
toolCallId,
toolName,
source,
result,
...(preliminaryResults?.length && {
preliminaryResults,
}),
});
this.turnBroadcaster?.push({
type: 'tool.result' as const,
Expand All @@ -991,9 +987,6 @@ export class ModelResult<
source,
result,
timestamp: Date.now(),
...(preliminaryResults?.length && {
preliminaryResults,
}),
} as CorrelatedResponseStreamEvent<TTools>);
}

Expand Down Expand Up @@ -3246,7 +3239,6 @@ export class ModelResult<
controller: AbortController;
timeoutMs: number | undefined;
runBinding: RunBinding;
preliminaryResultsForCall: InferToolEventsUnion<TTools>[];
}
| {
type: 'execution';
Expand All @@ -3256,7 +3248,6 @@ export class ModelResult<
result: unknown;
error?: Error;
};
preliminaryResultsForCall: InferToolEventsUnion<TTools>[];
}
> {
// Universal task-tool dispatch: ONE static tool ("task") handles every
Expand Down Expand Up @@ -3288,14 +3279,14 @@ export class ModelResult<
return hookDenied;
}

const preliminaryResultsForCall: InferToolEventsUnion<TTools>[] = [];

const hasBroadcaster = this.toolEventBroadcaster || this.turnBroadcaster;
const onPreliminaryResult = hasBroadcaster
? (callId: string, resultValue: unknown) => {
const typedResult = resultValue as InferToolEventsUnion<TTools>;
preliminaryResultsForCall.push(typedResult);
this.broadcastPreliminaryResult(callId, String(toolCall.name), typedResult);
this.broadcastPreliminaryResult(
callId,
String(toolCall.name),
resultValue as InferToolEventsUnion<TTools>,
);
}
: undefined;

Expand Down Expand Up @@ -3364,7 +3355,7 @@ export class ModelResult<
);

if (executed === 'timeout') {
return this.buildToolTimeoutOutcome(toolCall, tool, timeoutMs, preliminaryResultsForCall);
return this.buildToolTimeoutOutcome(toolCall, tool, timeoutMs);
}

if (executed.type === 'parse_error') {
Expand Down Expand Up @@ -3403,7 +3394,6 @@ export class ModelResult<
controller,
timeoutMs,
runBinding,
preliminaryResultsForCall,
};
}

Expand All @@ -3412,7 +3402,6 @@ export class ModelResult<
toolCall: executed.effectiveToolCall,
tool,
result,
preliminaryResultsForCall,
};
} finally {
releaseGates();
Expand All @@ -3432,7 +3421,6 @@ export class ModelResult<
toolCall: ParsedToolCall<Tool>,
tool: Tool,
timeoutMs: number | undefined,
preliminaryResultsForCall: InferToolEventsUnion<TTools>[],
): {
type: 'execution';
toolCall: ParsedToolCall<Tool>;
Expand All @@ -3441,7 +3429,6 @@ export class ModelResult<
result: unknown;
error?: Error;
};
preliminaryResultsForCall: InferToolEventsUnion<TTools>[];
} {
const message = `Tool "${toolCall.name}" timed out after ${timeoutMs}ms`;
this.broadcastToolResult(
Expand All @@ -3462,7 +3449,6 @@ export class ModelResult<
code: 'tool_timeout',
}),
},
preliminaryResultsForCall,
};
}

Expand Down Expand Up @@ -3661,7 +3647,6 @@ export class ModelResult<
String(value.toolCall.name),
isMcpTool(value.tool) ? 'mcp' : 'client',
toolResult,
value.preliminaryResultsForCall.length > 0 ? value.preliminaryResultsForCall : undefined,
);

const outputForModel = await this.computeToolOutputForModel(value);
Expand Down Expand Up @@ -4136,7 +4121,6 @@ export class ModelResult<
result: unknown;
error?: Error;
};
preliminaryResultsForCall: InferToolEventsUnion<TTools>[];
}> {
const taskTool = buildTaskToolStub();
const answer = (result: unknown, error?: Error) => {
Expand All @@ -4160,7 +4144,6 @@ export class ModelResult<
: {
result,
},
preliminaryResultsForCall: [] as InferToolEventsUnion<TTools>[],
};
};

Expand Down
3 changes: 0 additions & 3 deletions packages/agent/src/lib/tool-executor.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,6 @@ export async function executeGeneratorTool(
extras,
);

const preliminaryResults: unknown[] = [];
let finalResult: unknown;
let hasFinalResult = false;
let lastEmittedValue: unknown;
Expand All @@ -359,7 +358,6 @@ export async function executeGeneratorTool(
hasFinalResult = true;
} else {
const validatedPreliminary = validateToolOutput(tool.function.eventSchema, event);
preliminaryResults.push(validatedPreliminary);
if (onPreliminaryResult) {
onPreliminaryResult(toolCall.id, validatedPreliminary);
}
Expand Down Expand Up @@ -387,7 +385,6 @@ export async function executeGeneratorTool(
toolName: toolCall.name,
source,
result: finalResult,
preliminaryResults,
};
} catch (error) {
return {
Expand Down
6 changes: 1 addition & 5 deletions packages/agent/src/lib/tool-orchestrator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,15 +186,13 @@ export function toolResultsToMap(results: ToolExecutionResult<Tool>[]): Map<
string,
{
result: unknown;
preliminaryResults?: unknown[];
}
> {
const map = new Map();

for (const result of results) {
map.set(result.toolCallId, {
result: result.result,
preliminaryResults: result.preliminaryResults,
});
}

Expand All @@ -211,9 +209,7 @@ export function summarizeToolExecutions(results: ToolExecutionResult<Tool>[]): s
if (result.error) {
lines.push(`❌ ${result.toolName} (${result.toolCallId}): ERROR - ${result.error.message}`);
} else {
const prelimCount = result.preliminaryResults?.length ?? 0;
const prelimInfo = prelimCount > 0 ? ` (${prelimCount} preliminary results)` : '';
lines.push(`✅ ${result.toolName} (${result.toolCallId}): SUCCESS${prelimInfo}`);
lines.push(`✅ ${result.toolName} (${result.toolCallId}): SUCCESS`);
}
}

Expand Down
3 changes: 2 additions & 1 deletion packages/agent/src/lib/tool-types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1213,7 +1213,7 @@ export interface ToolExecutionResult<T extends Tool> {
: unknown; // Final result (sent to model)
preliminaryResults?: T extends ToolWithGenerator<$ZodObject<$ZodShape>, infer E>
? zodInfer<E>[]
: undefined; // All yielded values from generator
: undefined; // Kept for type compatibility; no longer populated
error?: Error;
}

Expand Down Expand Up @@ -1365,6 +1365,7 @@ export type ToolResultEvent<
source: 'client' | 'mcp';
result: TResult;
timestamp: number;
/** Kept for type compatibility; the runtime no longer copies every yield here. */
preliminaryResults?: TPreliminaryResults[];
};

Expand Down
14 changes: 3 additions & 11 deletions packages/agent/tests/unit/tool-executor-return.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,7 @@ describe('executeGeneratorTool - return value capture', () => {
const result = await executeGeneratorTool(generatorTool, toolCall, mockContext);

expect(result.error).toBeUndefined();
expect(result.preliminaryResults).toHaveLength(2);
expect(result.preliminaryResults).toEqual([
{
status: 'working',
},
{
status: 'almost done',
},
]);
expect(result.preliminaryResults).toBeUndefined();
// The return value should be captured as the final result
expect(result.result).toEqual({
result: 'Done: test',
Expand Down Expand Up @@ -91,7 +83,7 @@ describe('executeGeneratorTool - return value capture', () => {
const result = await executeGeneratorTool(generatorTool, toolCall, mockContext);

expect(result.error).toBeUndefined();
expect(result.preliminaryResults).toHaveLength(0);
expect(result.preliminaryResults).toBeUndefined();
expect(result.result).toEqual({
result: 'Direct: test',
});
Expand Down Expand Up @@ -134,7 +126,7 @@ describe('executeGeneratorTool - return value capture', () => {
const result = await executeGeneratorTool(generatorTool, toolCall, mockContext);

expect(result.error).toBeUndefined();
expect(result.preliminaryResults).toHaveLength(2);
expect(result.preliminaryResults).toBeUndefined();
// Return value should take precedence
expect(result.result).toEqual({
finalValue: 'Final: test',
Expand Down
9 changes: 1 addition & 8 deletions packages/agent/tests/unit/tool-name-events.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,15 +269,8 @@ describe('toolName on runtime tool events', () => {
result: {
done: true,
},
preliminaryResults: [
{
stage: 'one',
},
{
stage: 'two',
},
],
});
expect(finals[0]?.preliminaryResults).toBeUndefined();

const legacyPrelims = legacyEvents.filter(
(
Expand Down
19 changes: 2 additions & 17 deletions packages/agent/tests/unit/tool-orchestrator.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,12 +437,8 @@ describe('executeToolLoop', () => {
result: {
done: true,
},
preliminaryResults: [
{
pct: 50,
},
],
});
expect(result.toolExecutionResults[0]?.preliminaryResults).toBeUndefined();
});

it('applies nextTurnParams input changes to the conversation', async () => {
Expand Down Expand Up @@ -551,23 +547,16 @@ describe('toolResultsToMap', () => {
makeResult({
toolCallId: 'b',
result: 2,
preliminaryResults: [
'p',
],
}),
];

const map = toolResultsToMap(results);

expect(map.get('a')).toEqual({
result: 1,
preliminaryResults: undefined,
});
expect(map.get('b')).toEqual({
result: 2,
preliminaryResults: [
'p',
],
});
expect(map.size).toBe(2);
});
Expand All @@ -587,10 +576,6 @@ describe('summarizeToolExecutions', () => {
makeResult({
toolName: 'chatty',
toolCallId: 'b',
preliminaryResults: [
1,
2,
],
}),
makeResult({
toolName: 'bad',
Expand All @@ -603,7 +588,7 @@ describe('summarizeToolExecutions', () => {
expect(summary).toBe(
[
'✅ good (a): SUCCESS',
'✅ chatty (b): SUCCESS (2 preliminary results)',
'✅ chatty (b): SUCCESS',
'❌ bad (c): ERROR - nope',
].join('\n'),
);
Expand Down
Loading