Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 33 additions & 13 deletions docs/guides/extensions/curator/metadata_contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ By following this guide, you will:

- List curation tasks in a Synapse project
- Create a Grid session for a record-based curation task
- Synchronize the Grid session to pick up schema changes made after the session was created
- Download metadata from the Grid to a local CSV
- Edit the metadata locally
- Upload the metadata back into the Grid
Expand Down Expand Up @@ -108,7 +109,24 @@ Each option in Step 2 leaves you with a single `curation_task`. Start a new Grid
latest_grid = curation_task.create_grid_session()
```

### Step 4: Download record-based metadata as a local CSV
### Step 4: Synchronize the grid session to pick up schema updates

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I currently place the synchronization step right after session creation (Step 3), framed around picking up a schema change before starting to edit. There's also a valid case for calling synchronize after importing local CSV edits (Step 6). Please let me know if you would want me to move it. @cconrad8


A Grid session captures the JSON schema in place at the moment it's created — it does not automatically pick up a newer schema version. If the administrator adds or changes a column in the schema *after* you created your session in Step 3, synchronize the session to pull in the latest schema and data from the RecordSet before you continue working.

For record-based grids, `sync_type` is required — you must explicitly choose `"PULL"` or `"PULL_PUSH"`:

```python
latest_grid = latest_grid.synchronize(sync_type="PULL")
```

- **`"PULL"`** — refreshes the grid session with the latest schema and data from the RecordSet, without writing anything back. Use this to preview an incoming schema or data change (for example, a newly added column) before you've made any edits of your own, or simply to catch the session up to the current RecordSet state.
- **`"PULL_PUSH"`** — does the same pull, then immediately writes the grid session's current data back to the RecordSet as a new version. Use this once you're ready to commit your in-progress edits together with the refreshed schema/data.

> **Note:** Run this step any time you suspect the schema has changed since you opened the session — for example, if the administrator mentions they've published a new schema version, or if a field you expect to see is missing from your downloaded CSV in Step 5.

If your session is already current (no schema changes since Step 3), you can skip this step entirely.

### Step 5: Download record-based metadata as a local CSV

Download the current grid contents so you can edit them locally — in pandas, Excel, or any tool that reads CSV.

Expand All @@ -128,7 +146,7 @@ print(df)

# Example only: fills 4 rows with random integers regardless of column type.
# Replace this with real edits that match your task's schema before importing —
# schema validation runs in Step 6 and will reject values that don't fit.
# schema validation runs in Step 7 and will reject values that don't fit.
df = pd.DataFrame(
np.random.randint(0, 100, size=(4, len(df.columns))),
columns=df.columns,
Expand All @@ -138,7 +156,7 @@ edited_path = "./grid_edited.csv"
df.to_csv(edited_path, index=False)
```

### Step 5: Import edited record-based metadata to Synapse
### Step 6: Import edited record-based metadata to Synapse

`import_csv` upserts rows into the grid based on the `upsert_keys` the administrator configured when setting up the `RecordSet`. Existing rows matching on those keys are updated; new rows are inserted.

Expand All @@ -147,7 +165,7 @@ latest_grid = latest_grid.import_csv(path=edited_path)
print(f"Upserted edits into grid session: https://www.synapse.org/Grid:default?sessionId={latest_grid.session_id}")
```

### Step 6: Export the grid back to the RecordSet
### Step 7: Export the grid back to the RecordSet

> **Important:** Until you call `export_to_record_set()`, your edits live only inside the Grid session — they aren't visible on the RecordSet and won't be validated. Apply changes whenever you reach a logical checkpoint.

Expand All @@ -158,9 +176,9 @@ latest_grid.export_to_record_set()
print(f"Exported to RecordSet version: {latest_grid.record_set_version_number}")
```

### Step 7: Review your validation results
### Step 8: Review your validation results

When you exported the grid in Step 6, Synapse validated each row against the JSON schema bound to the RecordSet and generated a row-level report. Reviewing this report before handing the task back to the administrator lets you catch and fix problems in your own data first — saving a round trip.
When you exported the grid in Step 7, Synapse validated each row against the JSON schema bound to the RecordSet and generated a row-level report. Reviewing this report before handing the task back to the administrator lets you catch and fix problems in your own data first — saving a round trip.

#### Prerequisites for validation results

Expand All @@ -170,7 +188,7 @@ A validation report is only generated when **all** of the following are true:
2. You have entered data through a Grid session
3. The Grid session has been exported back to the RecordSet — this is the step that triggers validation and populates the RecordSet's validation_file_handle_id

If the Grid was never exported (Step 6), there is nothing to review yet.
If the Grid was never exported (Step 7), there is nothing to review yet.

#### Retrieve and inspect the results

Expand All @@ -185,7 +203,7 @@ if isinstance(curation_task.task_properties, RecordBasedMetadataTaskProperties):
validation_df = record_set.get_detailed_validation_results()

if validation_df is None:
print("No validation results yet — make sure the Grid was exported in Step 6.")
print("No validation results yet — make sure the Grid was exported in Step 7.")
else:
total = len(validation_df)
valid = validation_df["is_valid"].sum()
Expand Down Expand Up @@ -230,11 +248,11 @@ Row 2:

#### Fix and re-export

If any rows are invalid, recreate a Grid session against the RecordSet (see Step 3), correct the offending rows, and re-run Steps 4–6 to re-export. The validation report is regenerated on each export, so iterate until the report is clean before letting the administrator know your task is ready.
If any rows are invalid, recreate a Grid session against the RecordSet (see Step 3), correct the offending rows, and re-run Steps 5–7 to re-export. The validation report is regenerated on each export, so iterate until the report is clean before letting the administrator know your task is ready.

> **If get_detailed_validation_results returns None after exporting:** check that record_set.validation_file_handle_id is set after the re-fetch. If it isn't, the export did not complete — re-run export_to_record_set() on an active Grid session against the same RecordSet.

### Step 8: Mark the curation task as COMPLETED
### Step 9: Mark the curation task as COMPLETED

Once your validation report is clean and you've cleaned up the Grid session, transition the curation task to COMPLETED. This signals the administrator that the task is ready for their review — they can list tasks in the project and pick up the ones whose status is COMPLETED.

Expand All @@ -244,11 +262,11 @@ curation_task.set_task_state(state="COMPLETED")

## File-Based Curation Tasks

File-based tasks follow the same overall flow as record-based tasks (Steps 1–8 above), with three key differences:
File-based tasks follow the same overall flow as record-based tasks (Steps 1–9 above), with three key differences:

**No CSV import.** `import_csv` is not currently supported for file-based grids. Instead, you can either:

- Download the CSV (Step 4) as a local reference, make your edits locally, then copy-paste the values back into the Grid UI
- Download the CSV (Step 5) as a local reference, make your edits locally, then copy-paste the values back into the Grid UI
- Make edits directly in the Synapse Grid UI — Step 3 prints the session URL (`https://www.synapse.org/Grid:default?sessionId=...`) after creating the session

**Use `synchronize()` instead of `export_to_record_set()`.** After editing in the Grid UI, push your changes back to the underlying files:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For discussing, should we deprecate the export to record set function and push people to use this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually noticed some problems if we remove export_to_record_set_async. As an example, the following is working as expected:

grid = Grid(session_id="MTM4Mzc0")
grid = await grid.import_csv_async(path="test_grid_import.csv")
await grid.export_to_record_set_async()
grid = await task.synchronize_active_grid_session_async(sync_type="PULL")
grid.download_csv(destination="/Users/lpeng/Downloads")

But if I remove export_to_record_set_async and change sync_type to PULL_PUSH, then I noticed that I can't actually "push" the changes from my CSV to the grid. I will ask other devs in curator-dev channel after Sage week.

Expand All @@ -259,6 +277,8 @@ latest_grid.synchronize()

This writes the Grid annotation values back to each file as Synapse annotations. There is no versioned RecordSet — the files themselves are updated in place.

Note: for file-based grids, `sync_type` is not required and always behaves as `"PULL_PUSH"` — there is no separate preview (`"PULL"`) step.

**No per-row validation report.** Validation is enforced by the JSON schema bound to the folder containing the files, not by a row-level export report. After you call `synchronize()`, the administrator verifies schema compliance on their end — there is nothing to retrieve from the contributor side. If the administrator reports violations, correct the flagged annotations in the Grid UI and re-synchronize.

## Appendix
Expand All @@ -283,7 +303,7 @@ Deleting is permanent — you can no longer re-export from this session. If you
- [Grid.download_csv][synapseclient.models.Grid.download_csv] - Download Grid contents as a local CSV
- [Grid.import_csv][synapseclient.models.Grid.import_csv] - Upsert CSV edits back into a Grid session (record-based grids only)
- [Grid.export_to_record_set][synapseclient.models.Grid.export_to_record_set] - Export Grid data back to RecordSet and generate validation results
- [Grid.synchronize][synapseclient.models.Grid.synchronize] - Synchronize a file-based Grid against its source file view
- [Grid.synchronize][synapseclient.models.Grid.synchronize] - Synchronize a Grid session against its source RecordSet or file view, pulling in schema/data changes and (for `PULL_PUSH`) writing edits back
- [Grid.delete][synapseclient.models.Grid.delete] - Delete a Grid session
- [RecordSet.get_detailed_validation_results][synapseclient.models.RecordSet.get_detailed_validation_results] - Retrieve the row-level validation report for a RecordSet
<!-- markdownlint-enable MD052 -->
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/experimental/async/curator.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
- list_async
- create_grid_session_async
- set_task_state_async
- synchronize_active_grid_session_async
---

[](){ #RecordSet-reference-async }
Expand Down Expand Up @@ -56,6 +57,12 @@
inherited_members: true
members:
---
[](){ #SyncType-reference-async }
::: synapseclient.models.SyncType
options:
inherited_members: true
members:
---
[](){ #grid-reference-async }
::: synapseclient.models.Grid
options:
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/experimental/sync/curator.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
- list
- create_grid_session
- set_task_state
- synchronize_active_grid_session
---

[](){ #RecordSet-reference }
Expand Down Expand Up @@ -56,6 +57,12 @@
inherited_members: true
members:
---
[](){ #SyncType-reference }
::: synapseclient.models.SyncType
options:
inherited_members: true
members:
---
[](){ #grid-reference }
::: synapseclient.models.Grid
options:
Expand Down
2 changes: 2 additions & 0 deletions synapseclient/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
Grid,
GridExecutionDetails,
RecordBasedMetadataTaskProperties,
SyncType,
TaskExecutionDetails,
TaskState,
)
Expand Down Expand Up @@ -106,6 +107,7 @@
"TaskState",
"Grid",
"GridExecutionDetails",
"SyncType",
"TaskExecutionDetails",
"UserProfile",
"UserPreference",
Expand Down
Loading
Loading