Adding MCP problem type basic sampler infrastructure by guidorc · Pull Request #1628 · WebFuzzing/EvoMaster

guidorc · 2026-07-04T16:18:59Z

Summary

Adding the basic search infrastructure for blackbox fuzzing of Model Context Protocol (MCP) servers. The approach is: connect to the target server, discover its capabilities via the standard MCP handshake, automatically construct a mutable input representation (gene tree) for each tool and resource from their JSON Schema definitions, then use a search algorithm to evolve a sequences of calls that maximise coverage; reaching each capability and obtaining a non-error response

Components

MCP client (HttpMcpClient, McpClient, McpDTOs)

An HTTP-based client that implements the MCP JSON-RPC protocol: initialization handshake, tool invocation (tools/call), static resource reads (resources/read), and capability discovery (tools/list, resources/list, resources/templates/list).

Action model (McpAction, McpToolCallAction, McpResourceReadAction)

Two concrete action types mirror the two MCP interaction patterns: calling a named tool with structured arguments, and reading a resource by URI (including template URIs with mutable parameters).

Individual (McpIndividual)

A test case represented as an ordered sequence of McpActions, integrating with EvoMaster's existing EnterpriseIndividual and mutation infrastructure.

Sampler (McpSampler)

On startup, connects to the target MCP server, performs the handshake, and discovers all tools and resources. Each tool's JSON Schema input definition is converted into an ObjectGene tree used by the search engine to mutate arguments. Pre-built single-action individuals ensure every capability is exercised at least once before random search begins.

Fitness function (McpBlackBoxFitness)

Evaluates individuals by executing their actions sequentially and scoring coverage targets based on observable protocol signals: 1.0 for a successful tool call or resource read, 0.5 for a server-reported error (partial credit keeps the search gradient useful). Exceptions break the action sequence early.

McpBlackBoxModule

Wires all the above into EvoMaster's dependency injection container and registers the problem type in the main entry point.

Follow Up

McpTestCaseWriter — the test output pipeline currently uses NoTestCaseWriter, so generated test files have empty bodies. A dedicated writer that emits executable MCP
client calls is needed.
Format-aware gene building — string fields carrying semantic formats (date-time, email, uuid) are built as plain StringGene. The sampler needs a three-layer cascade
(explicit JSON Schema format → type switch → name-based heuristics) to generate valid values for these fields, matching the approach used by the REST builder.
SortingHelper / TestCaseNamingStrategyFactory — McpIndividual is not yet registered in the test sorting and naming strategies, causing a handled warning during output.

@PostConstruct

Introduces a new problem domain for blackbox testing of MCP (Model Context Protocol) servers, following the same architecture as the existing GraphQL and RPC problem domains. New components under core/src/main/kotlin/.../problem/mcp/: - McpAction (abstract base), McpToolCallAction (tools/call), McpResourceReadAction (resources/read), McpInputParam, McpUriParam - McpIndividual: chromosome wrapping sequences of MCP actions - McpCallResult: per-action execution record storing isError flag - client/McpClient (interface), client/HttpMcpClient (JSON-RPC 2.0 over HTTP with pagination), client/McpDTOs - service/McpSampler: @PostConstruct discovery via tools/list, resources/list, resources/templates/list; gene tree building from JSON Schema; random and smart (ad-hoc) sampling - service/McpFitness (abstract base), service/McpBlackBoxFitness: per-action scoring using isError flag and successful reads as signals - service/McpBlackBoxModule: Guice bindings mirroring GraphQLBlackBoxModule Unit tests cover action construction, individual copy/mutation, and HttpMcpClient JSON-RPC parsing via WireMock stubs.

- Add ProblemType.MCP (experimental) to EMConfig enum - Add bbTargetUrl validation for MCP black-box mode - Route ProblemType.MCP to McpBlackBoxModule in Main.init() - Add getAlgorithmKeyMcp() supporting RANDOM, MIO, MOSA, WTS, SMARTS - Dispatch MCP algorithm key in Main.run() - Bind NoTestCaseWriter in McpBlackBoxModule EvoMaster can now be invoked with: --blackBox true --problemType MCP --bbTargetUrl <url>

@PostConstruct

Connection failures during @PostConstruct were surfacing as an uncaught InvocationTargetException labelled as an EvoMaster bug. Wrap the listTools() call so a failed connection to the MCP server produces a clean, user-readable SutProblemException instead.

Servers may return 400/404/405 for MCP methods they don't support (e.g., resources/list on a tools-only server). Previously this caused an IOException from HttpURLConnection.inputStream on non-2xx responses, crashing the sampler during initialization. Extracted openConnection() helper and made post() return null on 400/404/405, which callers treat as an empty/error result.

* Setting up MCP Problem Type Skeleton * remove db related size from MCP individual * Fix for resource read action id Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * feedback * adjust resource read action parameters --------- Co-authored-by: guido-rodriguez_sfemu <guido.rodriguez@salesforce.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Pgarrett · 2026-07-18T23:22:18Z

+    /**
+     * MCP protocol version negotiated during the initialize handshake.
+     */
+    const val PROTOCOL_VERSION = "2025-11-25"


Just as a question, could this be a parameter for when fuzzing an MCP API? Or are protocol versions not backward compatible?

Pgarrett · 2026-07-18T23:23:23Z

+            mapOf(
+                "protocolVersion" to "2024-11-05",
+                "capabilities" to emptyMap<String, Any>(),
+                "clientInfo" to mapOf("name" to "EvoMaster", "version" to "1.0.0")


Unaddressed comment @guidorc

Pgarrett · 2026-07-18T23:28:34Z

+            ?: return McpToolResult(isError = true)
+        val result = response["result"] as? Map<String, Any?> ?: return McpToolResult(isError = true)
+        val rawContent = result["content"] as? List<Map<String, Any?>> ?: emptyList()
+        val content = rawContent.map { c ->


Favour readability, this could be extracted to a getMcpContent method

Pgarrett · 2026-07-18T23:29:18Z

+        val rawContent = result["content"] as? List<Map<String, Any?>> ?: emptyList()
+        val content = rawContent.map { c ->
+            val type = c["type"] as? String
+                ?: throw IllegalStateException("tools/call response content item missing required 'type' field")


Can we add more information on which tool call is actually causing this error? If we're fuzzing to test an API, it's best if we provide the most information to our users. Applicable to the other exceptions being thrown

Pgarrett · 2026-07-18T23:39:54Z

+import javax.ws.rs.client.Entity
+import javax.ws.rs.core.MediaType
+import javax.ws.rs.core.Response
+import kotlin.String


Are all this imports needed?

Pgarrett · 2026-07-18T23:44:29Z

+        val result = response["result"] as? Map<String, Any?> ?: return McpResourceResult()
+        val rawContents = result["contents"] as? List<Map<String, Any?>> ?: emptyList()
+        val contents = rawContents.map { c ->
+            McpContent(


How come this McpContent can be created without type and we throw an exception for the one in callTool? It's not a good practice to use the same object to represent different things. Either this should have a type assigned, or it should be a different object

Pgarrett · 2026-07-18T23:46:16Z

+/** Content item within a tool or resource response. */
 data class McpContent(
-    val type: String,
+    val type: String? = null,


Everything is nullable? Why? I fear we might hide issues due to this flexibility

Pgarrett · 2026-07-18T23:47:43Z

+        }
+
+        // Discover tools
+        val tools = mcpClient.listTools()


Maybe extract this and subsequent discoveries to their own methods. That way we have a more concise and declarative initialize method

Pgarrett · 2026-07-18T23:48:20Z

+            return ind
+        }
+
+        val n = randomness.nextInt(1, getMaxTestSizeDuringSampler())


What's n? Choose a better name

guido-rodriguez_sfemu and others added 15 commits May 28, 2026 22:11

Add missing header

badf32d

Add MCP initialization handshake

cf1c967

Merge branch 'master' into feature/mcp-blackbox-sampler

af0cdd2

merge skeleton setup

999c3e0

Fix for MCP uriTemplate

5aea547

store MCP-Session-Id on inititalize

38ef921

Merge branch 'master' into feature/mcp-blackbox-sampler

b4a4384

Adding JavaDocs

2b0e604

remove supress unchecked cast annotations

11df7b1

Merge branch 'master' into feature/mcp-blackbox-sampler

ef80412

guidorc changed the base branch from master to external-pr-mcp July 4, 2026 16:37

Merge branch 'external-pr-mcp' into feature/mcp-blackbox-sampler

e2820c6

guidorc changed the base branch from external-pr-mcp to master July 4, 2026 16:41

Merge branch 'master' into feature/mcp-blackbox-sampler

5cf695e

guidorc requested a review from Pgarrett July 7, 2026 00:57

Merge branch 'master' into feature/mcp-blackbox-sampler

e210728

Pgarrett requested changes Jul 12, 2026

View reviewed changes

guidorc changed the title ~~Adding MCP problem type basic fuzzing infrastructure~~ Adding MCP problem type basic sampler infrastructure Jul 13, 2026

guido-rodriguez_sfemu and others added 4 commits July 12, 2026 23:02

Adopt Jersey client

d240e23

Extract constants into shared file

c501ad1

use JsonNode data type for tools input schema

ed9906d

use MediaType constants

d4c0c34

guidorc force-pushed the feature/mcp-blackbox-sampler branch from c7cc742 to d4c0c34 Compare July 13, 2026 03:05

guidorc added 2 commits July 13, 2026 20:13

remove redundant HTTP status check

4e6b786

refactor MCP list response navigation

14b9b8d

guidorc added 2 commits July 13, 2026 21:50

Validate tool result includes type

76948f5

general feedback

9517441

guidorc requested a review from Pgarrett July 14, 2026 02:36

Pgarrett reviewed Jul 18, 2026

View reviewed changes

Pgarrett requested changes Jul 18, 2026

View reviewed changes

Uh oh!

Conversation

guidorc commented Jul 4, 2026

Summary

Components

MCP client (HttpMcpClient, McpClient, McpDTOs)

Action model (McpAction, McpToolCallAction, McpResourceReadAction)

Individual (McpIndividual)

Sampler (McpSampler)

Fitness function (McpBlackBoxFitness)

McpBlackBoxModule

Follow Up

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants