Skip to content

feat: add support for Firestore Pipeline - #292

Open
Lyokone wants to merge 8 commits into
mainfrom
feat/pipeline
Open

feat: add support for Firestore Pipeline#292
Lyokone wants to merge 8 commits into
mainfrom
feat/pipeline

Conversation

@Lyokone

@Lyokone Lyokone commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Add support for Firestore Pipeline

How to use

final snapshot = await firestore
    .pipeline()
    .collection('books')
    .where(Expression.field('active').equalValue(true))
    .sort([Expression.field('price').ascending()])
    .select([
      Expression.field('title'),
      Expression.field('price'),
      Expression.field('title').toUpperCase().as('upperTitle'),
      Expression.field('tags').arrayLength().as('tagCount'),
    ])
    .limit(10)
    .execute();
for (final result in snapshot.results) {
  print(result.data());
}

Aggregates

Aggregate stages use aliased aggregate expressions:

final snapshot = await firestore
    .pipeline()
    .collection('books')
    .where(Expression.field('active').equalValue(true))
    .aggregate([
      Expression.field('price').sum().as('totalPrice'),
      Expression.field('rating').average().as('averageRating'),
      PipelineFunctions.count().as('bookCount'),
    ])
    .execute();
final data = snapshot.results.single.data();
print(data);

Expressions

Use Expression.field, Expression.constant, and Expression.variable to
build expressions. Most helpers are also available as fluent methods:

final expression = Expression.field('createdAt')
    .timestampSubtract('day', 7)
    .timestampToUnixSeconds()
    .as('createdSeconds');

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces Firestore Pipeline operations to the google_cloud_firestore package, enabling server-side projections, expressions, aggregates, and vector search, complete with comprehensive E2E and unit tests. The review feedback highlights several key improvement opportunities: reverting a breaking change to the DistanceMeasure enum by converting values to lowercase locally within the pipeline execution, optimizing performance by extracting a frequently compiled regular expression into a file-level constant, and adding as well as exporting missing top-level comparison helpers (lessThan and greaterThan) to ensure API completeness.

Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart Outdated
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart
Comment thread packages/google_cloud_firestore/lib/google_cloud_firestore.dart
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

Coverage Report

✅ Coverage 73.12% meets 40% threshold

Total Coverage: 73.12%
Lines Covered: 5849/7999

Package Breakdown

Package Coverage
firebase_admin_sdk 73.02%
google_cloud_firestore 73.21%

Minimum threshold: 40%

@demolaf
demolaf self-requested a review June 30, 2026 15:03
Comment thread packages/google_cloud_firestore/lib/src/pipeline.dart

@demolaf demolaf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PipelineFunctions accepts Object? but only Expression.field() actually works without a fieldOrExpression-style check — untested since every call site already used field(). We should either commit to requiring Expression.field() explicitly or handle the coercion like Node's fieldOrExpression.


/// STARTS_WITH string function.
static PipelineBooleanExpression startsWith(Object? value, Object? prefix) {
return _bool('starts_with', [value, prefix]);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since value is typed as Object?, we should check at runtime whether it was passed as a String and wrap it with field(value) so it's encoded as a field reference.

Without that check, the string is encoded as a literal value instead of a field reference — so the expression compares against the fixed text 'title', not the value of the title field.

Example (Bug) — this looks like it filters on the title field, but instead compares the literal text "title" against "Harry":

await firestore
    .pipeline()
    .collection('books')
    .where(PipelineFunctions.startsWith('title', 'Harry'))
    .execute();

This problem also exists in other methods across PipelineFunctions e.g. equal, lessThan, arrayContains, mapGet, and others

}

/// STARTS_WITH string function.
static PipelineBooleanExpression startsWith(Object? value, Object? prefix) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should rename value to fieldName which is more descriptive and matches node.

Comment on lines +460 to +462
static PipelineBooleanExpression equal(Object? left, Object? right) {
return _bool('equal', [left, right]);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same issue here, we should check if left is String and wrap with field(left).

Lyokone added 5 commits August 4, 2026 11:01
Arguments in a "field or expression" position kept String values as string
literals, so `PipelineFunctions.startsWith('title', 'Harry')` compared the
literal text "title" against "Harry" instead of reading the `title` field.
Add `_fieldOrExpression`, mirroring the Node SDK's `fieldOrExpression`, and
apply it across the function catalog. Value positions still keep Strings as
literals, and document paths stay values as they do in Node.

Also:

- Add `PipelineSource.createFrom()` to convert a `Query` or `VectorQuery`
  into an equivalent Pipeline, translating filters, projections, implicit
  orderings, cursors, limit/limitToLast and offset.
- Add expressions released since the initial port: `coalesce`, `length`,
  `reverse`, `concat`, `getField`, `geoDistance`, `documentMatches` and
  `score`, plus `logicalMinimum`/`logicalMaximum`.
- Align `PipelineExpression.length()` and `.concat()` with the Node SDK's
  generic `length`/`concat` backend functions, and expose the
  string-specific `charLength()`/`stringConcat()` alongside them.
- Rename field-position parameters to `fieldName` to match Node.
- Hide `greaterThan`/`lessThan` from the Firestore import in two suites that
  want the `matcher` versions.
…dings

Four Pipeline stages diverged from the backend contract, verified against
the canonical Node SDK stage definitions in dev/src/pipelines/stage.ts:

- `unnest` sent only the array expression, so there was no way to name the
  emitted element. It now sends `[expr, field(alias)]`, taking the alias
  from the selectable, and encodes `index_field` as a field reference
  rather than a string.
- `replace_with` omitted the required mode argument; it now sends
  `[map, 'full_replace']`.
- `sample` passed the rate as a `documents`/`percentage` option; it now
  sends `[rate, mode]` with mode `documents` or `percent`.
- `distinct` sent a positional list of expressions; it now sends a single
  map keyed by alias, reusing the same projection map as `select` and
  `aggregate`.

`unnest` and `sample` are breaking signature changes.

Also make `PipelineFunctions.minimum`/`maximum` aggregate-only, matching
Node, now that `logicalMinimum`/`logicalMaximum` cover the element-wise
form.

Adds golden proto tests asserting each stage's arguments and options, so
this class of wire-format drift is caught without an Enterprise database.
The README covered 4 of the ~16 Pipeline stages and the example directory
had no pipeline content at all.

- Add `example/pipeline_example.dart` covering every stage: source/filter/
  sort/project/limit, aggregates with and without grouping, `unnest`,
  `replaceWith` with `addFields`/`removeFields`, `distinct`, `sample`,
  `union`, `findNearest`, and `createFrom`. Each example seeds and cleans up
  its own documents, and carries `[START]`/`[END]` region markers for docs
  ingestion.
- Expand the README: every source and stage with a runnable snippet, a
  function reference table mapping Dart helpers to backend function names,
  the field-argument vs value-argument rule, execution options, the
  `PipelineSnapshot`/`PipelineResult` surface, and a Query migration guide.
- Note the Enterprise-edition requirement and the failure behavior on
  `Pipeline.execute()` and `Firestore.pipeline()`.
- Add `example/README.md` so both examples are discoverable, and so pub.dev's
  Example tab shows representative code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants