feat: add vector-search helpers (cosine/l2/dot distance UDFs, semantic_search) - #40
Merged
Merged
Conversation
…c_search) Adds ibis_hotdata.vector: three @ibis.udf.scalar.builtin distance-function stubs plus a semantic_search(table, column, query_vector, k) query builder, compiling to ORDER BY <distance>(...) ASC LIMIT k with the vector column excluded from output -- the SQL shape the engine's HNSW index-selection rule requires. Verified via ibis.to_sql compile-only tests, and live against a local-cluster workspace (examples/06_semantic_search.py).
3 tasks
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| qvec = ibis.literal(list(query_vector)) | ||
| return ( | ||
| table.select(*other_cols, **{distance_name: distance_fn(col, qvec)}) | ||
| .order_by(ibis.asc(distance_name)) |
There was a problem hiding this comment.
super nit: (not blocking) if table already has a column named distance_name (default "distance"), it ends up in both other_cols and the **{distance_name: ...} kwarg, and select raises a duplicate-column error. Not worth guarding against for the common case, but a one-line note in the docstring that distance_name must not collide with an existing column would save a confusing traceback.
Review nit on #40: if table already has a column named distance_name (default "distance"), select raises a duplicate-column error since it ends up in both other_cols and the aliased distance kwarg.
rohan-hotdata
added a commit
that referenced
this pull request
Jul 22, 2026
The conflict resolution merging #40 dropped the blank line between the Fixed and Added sections under Unreleased.
rohan-hotdata
added a commit
that referenced
this pull request
Jul 22, 2026
…41) * fix: migrate create_table's upload path to the presigned upload flow runtimedb PR #952 removed the legacy POST /v1/files endpoint in favor of the presigned direct-to-storage flow (POST /v1/uploads -> PUT -> finalize). hotdata-ibis's create_table/upload_file called the now-dead endpoint via the SDK's generated UploadsApi.upload_file, breaking every write against any runtimedb deployment past that commit (confirmed on production). Switches to hotdata.uploads.UploadsApi (the SDK's own hand-written orchestration of the presigned flow: session create, storage PUT, finalize, with retries/multipart handling) instead of hand-rolling the flow here. No SDK version bump needed -- hotdata.uploads is already present across the whole >=0.7,<0.9 pin range. upload_file's return value now carries upload_id (FinalizeUploadResponse) instead of id (the old UploadResponse). Test fixtures updated to mock the three-stage presigned flow instead of the single POST /v1/files call. Verified: full offline suite (101 tests) against the updated flow, and live against production (api.hotdata.dev, where /v1/files is confirmed 404) -- examples 01, 03, 04, 05, and the vector-search example all pass end to end, including the create_database -> create_table -> query -> drop_database write/read cycle. * docs: restore blank line between CHANGELOG sections The conflict resolution merging #40 dropped the blank line between the Fixed and Added sections under Unreleased.
Contributor
|
approved |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ibis_hotdata.vector:cosine_distance,l2_distance,negative_dot_product(@ibis.udf.scalar.builtinstubs) plussemantic_search(table, column, query_vector, k, ...), a query builder for HNSW-indexed vector columns.ORDER BY <distance>(col, ARRAY[...]) ASC LIMIT kwith the vector column excluded from the output — the SQL shape the engine'sUSearchRuleindex-selection requires (query vector as a literalARRAY[...], distance aliased, embedding column never in the output — see engine issue #508).Verification
tests/test_vector.py) assert the exact SQL shape: literal array, aliased distance, source column excluded from output,ORDER BY ... ASC LIMIT k, customdistance_fn/distance_namehonored.examples/06_semantic_search.py): correct nearest-neighbor ordering for bothcosine_distanceandl2_distanceagainst a toy embedding table, through the full create → load → query → drop lifecycle.Note
This branch and #41 (presigned-upload-flow fix) both touch
CHANGELOG.md's[Unreleased]section at the same anchor point — expect a small, easy textual conflict when the second one merges; no code overlap.Test plan
uv run pytestuv run ruff check src tests examples🤖 Generated with Claude Code