Skip to content

docs: First cut at gRPC Gotchas blog - #2150

Open
kflynn wants to merge 4 commits into
mainfrom
flynn/grpc-gotchas
Open

docs: First cut at gRPC Gotchas blog#2150
kflynn wants to merge 4 commits into
mainfrom
flynn/grpc-gotchas

Conversation

@kflynn

@kflynn kflynn commented Aug 11, 2026

Copy link
Copy Markdown
Member

Signed-off-by: Flynn flynn@buoyant.io

kflynn added 3 commits August 11, 2026 18:48
Signed-off-by: Flynn <flynn@buoyant.io>
Signed-off-by: Flynn <flynn@buoyant.io>
Signed-off-by: Flynn <flynn@buoyant.io>
@kflynn kflynn changed the title docs: First cut at gRPC gotchas, needs better close for sure docs: First cut at gRPC Gotchas blog Aug 12, 2026

@schristoff schristoff left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should change the header image ofc

Comment thread linkerd.io/content/blog/2026/0812-gprc-gotchas/index.md Outdated
Comment thread linkerd.io/content/blog/2026/0812-gprc-gotchas/index.md Outdated
Comment thread linkerd.io/content/blog/2026/0812-gprc-gotchas/index.md Outdated
Comment thread linkerd.io/content/blog/2026/0812-gprc-gotchas/index.md Outdated

Over the wire, gRPC is layered over HTTP/2, so every gRPC call is an HTTP/2 stream with `HEADERS` frames carrying metadata, `DATA` frames carrying the inputs and outputs, and trailers (`HEADERS` frames after all the `DATA` frames) carrying the final status of the gRPC call.

![gPRC is layered within HTTP/2](<grpc-diagram.png>)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to speak for Catherine who is on PTO
Will you "Linkerd/Buoyant-ized" that diagram? (ugh, hate that I'm asking this)


This need to know the details of the connection isn't only a gRPC thing, of course. In general, anything more subtle than just passing bytes back and forth with no analysis requires you to know what the protocol is. This includes not just golden metrics and reliability, but even basic features like per-request routing -- after all, you can't route a single request if you don't know where the request starts and ends in the data stream.

There are fundamentally only two ways to know what protocol is in play: you can to look at the bytes in transit and figure out what protocol it is (_protocol detection_), or you can be told up front what protocol it is (_protocol declaration_). Linkerd takes the attitude that [detection is friendlier than declaration], so it leans heavily on protocol detection.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diagram probably here would be cute

Comment thread linkerd.io/content/blog/2026/0812-gprc-gotchas/index.md Outdated
Comment thread linkerd.io/content/blog/2026/0812-gprc-gotchas/index.md Outdated

So if you're using protocol declaration - which we're sure many of you are - first, **pay attention to the standards** when you set up your protocols! A wrong value can produce a very confusing error.

Second, though, is a debugging tip: Linkerd usually gets all of this stuff _right_, enough so that you should consider it something of a red flag to see opaque behavior -- if Linkerd seems to be ignoring Routes that it's marked as `Accepted`, don't burn a lot of time before looking to incorrect protocol declarations.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strongly consider rewording
"Linkerd prides itself in following standards as closely as possible, so you should consider it a red flag to see opaque behavior. If Linkerd sseems to be ignoring routes marked as accepted, first look at incorrect protocol declarations"


Second, though, is a debugging tip: Linkerd usually gets all of this stuff _right_, enough so that you should consider it something of a red flag to see opaque behavior -- if Linkerd seems to be ignoring Routes that it's marked as `Accepted`, don't burn a lot of time before looking to incorrect protocol declarations.

And go easy on the test authors, right? They're doing good work.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/ "Also, a huge thank you to test authors. You're doing great work. This is no means a call out. Maybe consider helping them by contributing (link)"

Signed-off-by: Flynn <flynn@buoyant.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants