Skip to content

Use synthetic data for scripts under src/maxtext/configs/tpu#4329

Draft
aireenmei wants to merge 1 commit into
mainfrom
aireen/tpu_script_syn
Draft

Use synthetic data for scripts under src/maxtext/configs/tpu#4329
aireenmei wants to merge 1 commit into
mainfrom
aireen/tpu_script_syn

Conversation

@aireenmei

@aireenmei aireenmei commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Description

Currently, the dataset type used by scripts under src/maxtext/configs/tpu falls into this pattern:

  • Category A (Newer scripts, v6e/, v5p/, v5e/{128b,gpt3_175b}.sh): already use dataset_type=synthetic, but the script has legacy docstring stating the $DATASET_PATH is required, but is actually not used in the command.
  • Category B (v5e/{16b,32b,64b}.sh, v5p/{32b,64b,128b,256b,512b,1024b}.sh): has both dataset_type=synthetic and dataset_path=$DATASET_PATH in the command, and dataset_path=$DATASET_PATH is a no-op. Also has the legacy docstring.
  • Category C (older scripts v4/*, v5e/{llama2_7b,llama2_13b,llama2_70b}.sh): use the default dataset_type=tfds and requires user to provide DATASET_PATH.

This PR:

  • Migrate category C scripts to use dataset_type=synthetic, which aligns with the new convention and allows user to do quick test without dataset preparation.
  • Remove the redundant dataset_path=$DATASET_PATH for category B
  • Remove legacy docstring for category A and B

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant