Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ This guide explains how automation agents and human contributors should work wit
- **Routes** – `GetRoutes`, `GetRoutesMinimal`. The minimal variant returns `RouteMinimalResponse` with deduplicated `LineMinimal` data; paging tokens are currently empty (pagination not implemented).
- **Train types** – `GetTrainTypesByStationId`, `GetRouteTypes`. Train types aggregate by line group and include related lines plus optional train type metadata. Rail variants use `TrainTypeKind::{Default, Branch, Rapid, Express, LimitedExpress, HighSpeedRapid, CommuterRapid}` (0-6); bus variants use `BusRoute` (7), which represents a `(route_id, shape_id)` operation pattern (e.g. 循環 / 短ターン / 支線) generated automatically from the configured GTFS bus feeds (Toei Bus, Seibu Bus, Keio Bus) and the converted Tokyu Bus JSON.
- **GTFS bus integration** – At startup, `src/import.rs::integrate_gtfs_to_stations()` ingests GTFS feeds into `gtfs_*` tables and then projects them onto the shared `stations` / `lines` / `types` / `station_station_types` tables. Every configured GTFS feed is imported, including Seibu Bus and Keio Bus (both downloaded from ODPT with `ODPT_ACCESS_TOKEN`). Tokyu Bus ordinary-route `BusroutePattern`, `BusstopPole`, and `BusTimetable` JSON are converted into the same `gtfs_*` representation; pattern IDs become `shape_id` values so route variants remain queryable as bus TrainTypes. The Tokyu-operated Ota, Shinagawa, and Meguro community buses use their official GTFS feeds and matching JSON routes are excluded to prevent duplicates. `ODPT_ACCESS_TOKEN` is required for authenticated sources. Stops whose Tokyu JSON records omit coordinates remain available to name and route queries but not coordinate searches. `transport_type` (0: rail, 1: bus) on both `stations` and `lines` keeps rail and bus records queryable side by side. GTFS IDs are namespaced per feed before import to avoid cross-operator collisions. `line_cd` (100,000,000+), `station_cd` / `station_g_cd` (200,000,000+), and bus `type_cd` / `line_group_cd` (100,000,000+) are all deterministic fnv1a hashes that stay clear of the rail data ranges. Disable the entire bus pipeline with `DISABLE_BUS_FEATURE=true`.
- **Bus English-name fallback** – Keio Bus GTFS and Tokyu Bus JSON ship a kana reading for every stop but no English (`en`) name, so `gtfs_stops.stop_name_r` would otherwise be empty. `src/domain/romaji.rs::romaji_display_name()` derives a modified-Hepburn romanization (with macrons for long vowels, matching the curated rail style: Tōkyō / Kyōto / Shin-Ōsaka) from that reading, and `import.rs` fills `stop_name_r` with it whenever the feed provides no `en` value. Because `stop_name_r` is the single upstream source that fans out into the `stations` projection, `search_by_name`, and the romanized bus route/headsign names, this supplements every English-facing surface at once. The fallback never overwrites a real `en` value, and a reading with no convertible kana stays `NULL` rather than emitting a partial transcription. When projecting into `stations`, `station_name_rn` is filled with the plain-ASCII spelling via `romaji::strip_macrons()` (Tōkyō → Tokyo), mirroring the rail dataset's `_r` (macron) / `_rn` (macron-free) column pair.
- **Bus stop translations (readings & English)** – GTFS-JP `translations.txt` layouts differ per feed, so `load_gtfs_translations` resolves columns by header name (Seibu ships 6 columns without `record_sub_id`; Keio and the Tokyu community feeds ship 7) and indexes each `stop_name` translation under both keys it may use: `record_id` (== the stop_id, Seibu — with the "-NN" pole suffix also mapped to the parent stop_id) and `field_value` (== the Japanese stop_name, Keio / Tokyu community, where `record_id` is left empty). `import_gtfs_stops` then looks a stop's translation up by stop_id first, then by name. Keying only by `record_id` (the previous behavior) silently dropped every field_value-keyed feed, leaving `station_name_k` filled with the kanji stop_name and `station_name_r` empty. Readings arriving as half-width katakana (`ニシハチオウジ`, Keio / Tokyu community) are folded to full-width via `romaji::to_fullwidth_katakana()` before storage.
- **Bus English-name fallback** – When a feed provides no English (`en`) translation for a stop — e.g. Tokyu Bus ordinary-route JSON, which carries only `dc:title` and `odpt:kana` — `src/domain/romaji.rs::romaji_display_name()` derives a modified-Hepburn romanization (with macrons for long vowels, matching the curated rail style: Tōkyō / Kyōto / Shin-Ōsaka) from the kana reading, and `import.rs` fills `stop_name_r` with it. The fallback never overwrites a real `en` value, and a reading with no convertible kana stays `NULL` rather than emitting a partial transcription. Because `stop_name_r` is the single upstream source that fans out into the `stations` projection, `search_by_name`, and the romanized bus route/headsign names, this supplements every English-facing surface at once. When projecting into `stations`, `station_name_rn` is filled with the plain-ASCII spelling via `romaji::strip_macrons()` (Tōkyō → Tokyo), mirroring the rail dataset's `_r` (macron) / `_rn` (macron-free) column pair.
- **TTS metadata** – `Station`, `StationMinimal`, `Line`, and `TrainType` expose `name_ipa` / `name_roman_ipa` plus `name_tts_segments` for multi-segment pronunciation output. Use `name_tts_segments` when clients need per-token SSML construction for mixed-language names such as `Kasai-Rinkai Park`.
- **Connected routes** – `GetConnectedRoutes`. `QueryInteractor::get_connected_stations` is not implemented yet and returns an empty vector; update the use-case and infrastructure layers together when adding real logic.
- Changes to the service contract require coordinated updates to `proto/stationapi.proto`, regenerated code via `tonic-build`, and corresponding adjustments in both presentation and use-case layers.
Expand Down
172 changes: 162 additions & 10 deletions stationapi/src/domain/romaji.rs
Original file line number Diff line number Diff line change
Expand Up @@ -257,18 +257,148 @@ fn is_separator(c: char) -> bool {
matches!(c, '\u{30FB}' | ' ' | '\u{3000}' | '/' | '\u{FF0F}')
}

/// Normalize hiragana to katakana so both reading styles convert identically.
/// Normalize a kana string to full-width katakana so every reading style
/// converts identically. Delegates to [`to_fullwidth_katakana`].
fn normalize_kana(input: &str) -> String {
input
.chars()
.map(|c| {
if ('\u{3041}'..='\u{3096}').contains(&c) {
char::from_u32(c as u32 + 0x60).unwrap_or(c)
} else {
c
to_fullwidth_katakana(input)
}

/// Normalize hiragana and half-width katakana to full-width katakana.
///
/// Some ODPT bus GTFS feeds ship their `ja-Hrkt` readings as half-width
/// katakana with combining voiced/semi-voiced marks (e.g. `ニシハチオウジ`), which
/// must be folded to full-width before they can be stored in the katakana
/// column or romanized. Full-width katakana passes through unchanged, so the
/// function is idempotent.
pub fn to_fullwidth_katakana(input: &str) -> String {
let chars: Vec<char> = input.chars().collect();
let mut out = String::with_capacity(chars.len());
let mut i = 0;
while i < chars.len() {
let c = chars[i];

// Hiragana → katakana (same +0x60 offset used across the module).
if ('\u{3041}'..='\u{3096}').contains(&c) {
out.push(char::from_u32(c as u32 + 0x60).unwrap_or(c));
i += 1;
continue;
}

// Half-width katakana → full-width, folding a trailing voiced (゙) or
// semi-voiced (゚) mark into the preceding kana.
if let Some(base) = halfwidth_katakana_to_fullwidth(c) {
match chars.get(i + 1).copied() {
Some('\u{FF9E}') => {
if let Some(voiced) = apply_dakuten(base) {
out.push(voiced);
i += 2;
continue;
}
}
Some('\u{FF9F}') => {
if let Some(semi) = apply_handakuten(base) {
out.push(semi);
i += 2;
continue;
}
}
_ => {}
}
})
.collect()
out.push(base);
i += 1;
continue;
}

out.push(c);
i += 1;
}
out
}

/// Map a half-width katakana code point to its full-width base form.
fn halfwidth_katakana_to_fullwidth(c: char) -> Option<char> {
let full = match c {
'\u{FF65}' => '\u{30FB}', // ・ middle dot
'ヲ' => 'ヲ',
'ァ' => 'ァ',
'ィ' => 'ィ',
'ゥ' => 'ゥ',
'ェ' => 'ェ',
'ォ' => 'ォ',
'ャ' => 'ャ',
'ュ' => 'ュ',
'ョ' => 'ョ',
'ッ' => 'ッ',
'ー' => 'ー',
'ア' => 'ア',
'イ' => 'イ',
'ウ' => 'ウ',
'エ' => 'エ',
'オ' => 'オ',
'カ' => 'カ',
'キ' => 'キ',
'ク' => 'ク',
'ケ' => 'ケ',
'コ' => 'コ',
'サ' => 'サ',
'シ' => 'シ',
'ス' => 'ス',
'セ' => 'セ',
'ソ' => 'ソ',
'タ' => 'タ',
'チ' => 'チ',
'ツ' => 'ツ',
'テ' => 'テ',
'ト' => 'ト',
'ナ' => 'ナ',
'ニ' => 'ニ',
'ヌ' => 'ヌ',
'ネ' => 'ネ',
'ノ' => 'ノ',
'ハ' => 'ハ',
'ヒ' => 'ヒ',
'フ' => 'フ',
'ヘ' => 'ヘ',
'ホ' => 'ホ',
'マ' => 'マ',
'ミ' => 'ミ',
'ム' => 'ム',
'メ' => 'メ',
'モ' => 'モ',
'ヤ' => 'ヤ',
'ユ' => 'ユ',
'ヨ' => 'ヨ',
'ラ' => 'ラ',
'リ' => 'リ',
'ル' => 'ル',
'レ' => 'レ',
'ロ' => 'ロ',
'ワ' => 'ワ',
'ン' => 'ン',
_ => return None,
};
Some(full)
}

/// Apply a dakuten (voiced mark) to a full-width katakana base, if voiceable.
fn apply_dakuten(c: char) -> Option<char> {
let voiced = match c {
'ウ' => 'ヴ',
'カ' | 'キ' | 'ク' | 'ケ' | 'コ' | 'サ' | 'シ' | 'ス' | 'セ' | 'ソ' | 'タ' | 'チ'
| 'ツ' | 'テ' | 'ト' | 'ハ' | 'ヒ' | 'フ' | 'ヘ' | 'ホ' => {
char::from_u32(c as u32 + 1)?
}
_ => return None,
};
Some(voiced)
}

/// Apply a handakuten (semi-voiced mark) to a full-width katakana base.
fn apply_handakuten(c: char) -> Option<char> {
match c {
'ハ' | 'ヒ' | 'フ' | 'ヘ' | 'ホ' => char::from_u32(c as u32 + 2),
_ => None,
}
}

/// Two-character katakana combinations (palatalised sounds and loanword kana).
Expand Down Expand Up @@ -522,6 +652,28 @@ mod tests {
assert_eq!(romaji("とうきょう"), "tōkyō");
}

#[test]
fn halfwidth_katakana_is_folded_to_fullwidth() {
// Real Keio Bus ja-Hrkt readings (half-width with combining dakuten).
assert_eq!(
to_fullwidth_katakana("ニシハチオウジエキミナミグチ"),
"ニシハチオウジエキミナミグチ"
);
assert_eq!(to_fullwidth_katakana("アキバジンジャ"), "アキバジンジャ");
assert_eq!(to_fullwidth_katakana("アサガヤジュウタク"), "アサガヤジュウタク");
// Handakuten and prolonged mark.
assert_eq!(to_fullwidth_katakana("パーク"), "パーク");
// Full-width katakana and hiragana are unchanged / folded (idempotent).
assert_eq!(to_fullwidth_katakana("トウキョウ"), "トウキョウ");
assert_eq!(to_fullwidth_katakana("しぶや"), "シブヤ");
}

#[test]
fn halfwidth_reading_romanizes_end_to_end() {
// Keio Bus 阿佐ヶ谷住宅東 reading → Hepburn (used only when `en` is absent).
assert_eq!(romaji("アサガヤジュウタクヒガシ"), "asagayajūtakuhigashi");
}

#[test]
fn separators_become_spaces() {
assert_eq!(romaji("シブヤ・エキ"), "shibuya eki");
Expand Down
Loading
Loading