feat: add support for Firestore Pipeline - #292
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces Firestore Pipeline operations to the google_cloud_firestore package, enabling server-side projections, expressions, aggregates, and vector search, complete with comprehensive E2E and unit tests. The review feedback highlights several key improvement opportunities: reverting a breaking change to the DistanceMeasure enum by converting values to lowercase locally within the pipeline execution, optimizing performance by extracting a frequently compiled regular expression into a file-level constant, and adding as well as exporting missing top-level comparison helpers (lessThan and greaterThan) to ensure API completeness.
Coverage Report✅ Coverage 73.12% meets 40% threshold Total Coverage: 73.12% Package Breakdown
Minimum threshold: 40% |
There was a problem hiding this comment.
PipelineFunctions accepts Object? but only Expression.field() actually works without a fieldOrExpression-style check — untested since every call site already used field(). We should either commit to requiring Expression.field() explicitly or handle the coercion like Node's fieldOrExpression.
|
|
||
| /// STARTS_WITH string function. | ||
| static PipelineBooleanExpression startsWith(Object? value, Object? prefix) { | ||
| return _bool('starts_with', [value, prefix]); |
There was a problem hiding this comment.
Since value is typed as Object?, we should check at runtime whether it was passed as a String and wrap it with field(value) so it's encoded as a field reference.
Without that check, the string is encoded as a literal value instead of a field reference — so the expression compares against the fixed text 'title', not the value of the title field.
Example (Bug) — this looks like it filters on the title field, but instead compares the literal text "title" against "Harry":
await firestore
.pipeline()
.collection('books')
.where(PipelineFunctions.startsWith('title', 'Harry'))
.execute();This problem also exists in other methods across PipelineFunctions e.g. equal, lessThan, arrayContains, mapGet, and others
| } | ||
|
|
||
| /// STARTS_WITH string function. | ||
| static PipelineBooleanExpression startsWith(Object? value, Object? prefix) { |
There was a problem hiding this comment.
we should rename value to fieldName which is more descriptive and matches node.
| static PipelineBooleanExpression equal(Object? left, Object? right) { | ||
| return _bool('equal', [left, right]); | ||
| } |
There was a problem hiding this comment.
same issue here, we should check if left is String and wrap with field(left).
-
In node,
fieldOrExpressionhandles convertingleftto field if a string was passed:
https://github.com/googleapis/google-cloud-node/blob/main/handwritten/firestore/dev/src/pipelines/expression.ts#L5061-L5068 -
pipeline node tests (validates bug):
https://github.com/googleapis/google-cloud-node/blob/main/handwritten/firestore/dev/system-test/pipeline.ts#L1128-L1144
Arguments in a "field or expression" position kept String values as string
literals, so `PipelineFunctions.startsWith('title', 'Harry')` compared the
literal text "title" against "Harry" instead of reading the `title` field.
Add `_fieldOrExpression`, mirroring the Node SDK's `fieldOrExpression`, and
apply it across the function catalog. Value positions still keep Strings as
literals, and document paths stay values as they do in Node.
Also:
- Add `PipelineSource.createFrom()` to convert a `Query` or `VectorQuery`
into an equivalent Pipeline, translating filters, projections, implicit
orderings, cursors, limit/limitToLast and offset.
- Add expressions released since the initial port: `coalesce`, `length`,
`reverse`, `concat`, `getField`, `geoDistance`, `documentMatches` and
`score`, plus `logicalMinimum`/`logicalMaximum`.
- Align `PipelineExpression.length()` and `.concat()` with the Node SDK's
generic `length`/`concat` backend functions, and expose the
string-specific `charLength()`/`stringConcat()` alongside them.
- Rename field-position parameters to `fieldName` to match Node.
- Hide `greaterThan`/`lessThan` from the Firestore import in two suites that
want the `matcher` versions.
…dings Four Pipeline stages diverged from the backend contract, verified against the canonical Node SDK stage definitions in dev/src/pipelines/stage.ts: - `unnest` sent only the array expression, so there was no way to name the emitted element. It now sends `[expr, field(alias)]`, taking the alias from the selectable, and encodes `index_field` as a field reference rather than a string. - `replace_with` omitted the required mode argument; it now sends `[map, 'full_replace']`. - `sample` passed the rate as a `documents`/`percentage` option; it now sends `[rate, mode]` with mode `documents` or `percent`. - `distinct` sent a positional list of expressions; it now sends a single map keyed by alias, reusing the same projection map as `select` and `aggregate`. `unnest` and `sample` are breaking signature changes. Also make `PipelineFunctions.minimum`/`maximum` aggregate-only, matching Node, now that `logicalMinimum`/`logicalMaximum` cover the element-wise form. Adds golden proto tests asserting each stage's arguments and options, so this class of wire-format drift is caught without an Enterprise database.
The README covered 4 of the ~16 Pipeline stages and the example directory had no pipeline content at all. - Add `example/pipeline_example.dart` covering every stage: source/filter/ sort/project/limit, aggregates with and without grouping, `unnest`, `replaceWith` with `addFields`/`removeFields`, `distinct`, `sample`, `union`, `findNearest`, and `createFrom`. Each example seeds and cleans up its own documents, and carries `[START]`/`[END]` region markers for docs ingestion. - Expand the README: every source and stage with a runnable snippet, a function reference table mapping Dart helpers to backend function names, the field-argument vs value-argument rule, execution options, the `PipelineSnapshot`/`PipelineResult` surface, and a Query migration guide. - Note the Enterprise-edition requirement and the failure behavior on `Pipeline.execute()` and `Firestore.pipeline()`. - Add `example/README.md` so both examples are discoverable, and so pub.dev's Example tab shows representative code.
Add support for Firestore Pipeline
How to use
Aggregates
Aggregate stages use aliased aggregate expressions:
Expressions
Use
Expression.field,Expression.constant, andExpression.variabletobuild expressions. Most helpers are also available as fluent methods: