[Java][FlightSQL] JDBC Driver fails when Looker formats the URL with ':direct=' instead of '://'

[amirhzd]

Describe the enhancement requested

I am running a customer-hosted Looker instance and trying to connect to Dremio using the
Apache Arrow Flight SQL JDBC driver (flight-sql-jdbc-driver), registered in Looker via a
custom_jdbc_config.yml entry so the native Dremio dialect can be kept:

- name: dremio
  dir_name: dremio
  module_path: org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver
  override_jdbc_url_subprotocol: arrow-flight-sql

override_jdbc_url_subprotocol only swaps the subprotocol token Looker plugs into its
existing, dialect-specific URL template for Dremio — it does not rewrite the rest of the
template. Looker's native Dremio dialect builds connection strings as:

jdbc:dremio:direct=<host>:<port>

so after the subprotocol override, Looker generates:

jdbc:arrow-flight-sql:direct=<host>:<port>

instead of the driver's expected format:

jdbc:arrow-flight-sql://<host>:<port>

Because ArrowFlightJdbcDriver parses the URL with java.net.URI.create(...) and strictly
expects the :// delimiter to separate scheme from host, it cannot extract a host from the
:direct= form, and the connection fails.

I tried a workaround by switching Looker's dialect dropdown to "Trino" (which emits the
standard :// format). While queries did run correctly over Arrow Flight, it broke the
Looker SQL Runner UI, since Looker then sends Trino-specific metadata queries to Dremio.

To use this driver properly with Looker, the native Dremio dialect needs to stay selected,
which means the driver needs to tolerate Looker's :direct= URL format.

Steps to Reproduce / Current Behavior

Passing a connection string containing :direct= to the driver:

jdbc:arrow-flight-sql:direct=<dremio-host>:<port>

fails instantly with:

Java::JavaSql::SQLException: URL must have a host. Expected format: jdbc:arrow-flight-sql://[host][:port][?param1=value&...]

Fix

Implemented and verified locally against flight/flight-sql-jdbc-core. Added a normalization
step to ArrowFlightJdbcDriver.java that rewrites the legacy :direct= delimiter to ://
before any URL parsing occurs, applied consistently at every entry point that parses the
connection string (acceptsURL, connect, getUrlsArgs):

// Some BI tools (e.g. Looker) generate connection strings using the legacy
// "jdbc:arrow-flight-sql:direct=<host>:<port>" syntax instead of the standard "://" delimiter.
// Normalize it so the URL can still be parsed as a valid URI.
private static final String LEGACY_DIRECT_DELIMITER = ":direct=";

private static String normalizeUrl(final String url) {
  if (url != null && url.contains(LEGACY_DIRECT_DELIMITER)) {
    return url.replace(LEGACY_DIRECT_DELIMITER, "://");
  }
  return url;
}

• acceptsURL(String url) now normalizes before checking the jdbc:arrow-flight-sql:// /
  jdbc:arrow-flight:// prefixes.
• connect(String url, Properties info) normalizes once up front and uses the normalized URL
  for both argument parsing (getUrlsArgs) and ArrowFlightConnection.createNewConnection.
• getUrlsArgs(String url) normalizes defensively as well, since it is @VisibleForTesting
  and may be called directly.

This is a purely additive, backward-compatible change — URLs that don't contain :direct=
are returned unmodified.

Testing / Verification

Added two regression tests to ArrowFlightJdbcDriverTest:

• testDriverUrlParsingMechanismShouldAcceptLegacyDirectDelimiter — verifies acceptsURL and
  getUrlsArgs correctly parse host/port/params from a jdbc:arrow-flight-sql:direct=... URL.
• testShouldConnectWhenProvidedWithLegacyDirectDelimiterUrl — verifies a live connection can
  be established end-to-end using the :direct= URL format against the test Flight SQL server.

Full ArrowFlightJdbcDriverTest suite: 18/18 tests passing, 0 failures, 0 errors.

Also validated end-to-end against a real customer-hosted Looker instance connecting to Dremio
with the native Dremio dialect (config above) — connection succeeded after applying the fix.

Additional deployment note (not part of this fix, but relevant to anyone testing it with Looker)

The flight-sql-jdbc-driver shaded jar is built with Java 17 bytecode by default
(maven.compiler.release=17 in the root POM), but Looker's bundled JRE is Java 11. Loading
the default build in Looker fails with:

UnsupportedClassVersionError: ... class file version 61.0, this version of the Java Runtime
only recognizes class file versions up to 55.0

To build a Looker-compatible jar, rebuild the driver and its Arrow-owned dependency closure
(arrow-format, arrow-memory-core, arrow-memory-netty(-buffer-patch), arrow-vector,
flight-core, flight-sql, flight-sql-jdbc-core, flight-sql-jdbc-driver) with:

mvn -pl format,memory/memory-core,memory/memory-netty,memory/memory-netty-buffer-patch,vector,flight/flight-core,flight/flight-sql,flight/flight-sql-jdbc-core,flight/flight-sql-jdbc-driver \
  -DskipTests -Dmaven.compiler.release=11 -Dmaven.compiler.source=11 -Dmaven.compiler.target=11 \
  clean install

(clean is required — Maven's incremental compiler otherwise skips recompilation when only
compiler flags change.) This may be worth calling out in the driver's documentation for users
deploying it to JVM-hosted BI tools that ship an older bundled runtime.

[lidavidm]

(1) You filed an issue, but this seems to (have been generated) for a PR, has this been verified at all?
(2) Isn't it possible for Looker to fix things? AFAIK, Dremio officially uses the Flight SQL driver as their JDBC driver.

[amirhzd]

Thank you for the questions — and apologies for the confusion. I'm fairly new to open-source contribution, and I leaned on an AI assistant to write this up, which made it read like a PR when I'd only meant to report the problem. Let me clarify honestly:

(1) The change is real — I did apply it locally and the driver's test suite passed, and it fixed a live Looker→Dremio connection for me. If I understood you correctly, I'd rather let CI verify it properly than have you take my word for it.

(2) Totally fair point. I actually opened a feature request on Looker's portal over two months ago, but it's still sitting in "Awaiting Review," so a fix from their side doesn't look imminent. In principle, the malformed URL is arguably a Looker (or Dremio-integration) concern rather than something the standard Arrow driver should absorb — I genuinely don't know where it best belongs, and that's your call.
That said, I'd like to make the case for accepting a driver-side workaround: since the upstream fix is blocked and this is likely to trip up anyone connecting from Looker (or similar systems that don't emit the standard :// form). And also tolerating the :direct= format lowers the barrier for more people to adopt the driver. But I'll defer to your judgment on the trade-off.

[amirhzd]

Would you prefer I open a PR so CI can validate the change and we can discuss it there, or keep this as an issue for now? Happy to go whichever way you think is best.

[lidavidm]

@jbonofre is there someone who can comment from the Dremio side?

Unless this is really a standard URI format for JDBC, I'd rather not build in workarounds for Looker specifically? Also we have no way of testing Looker ourselves

[jbonofre]

@lidavidm I will.

[xborder]

I agree with @lidavidm . This seems like an integration issue between Looker and Dremio, not an issue on the FlightSQL driver.

From the JDBC spec:

The standard syntax for JDBC URLs is shown here. It has three parts, which are separated by colons.

jdbc:<subprotocol>:<subname>
<subname>-a way to identify the data source. The subname can vary, depending on the subprotocol, and it can have any internal syntax the driver writer chooses, including a subsubname.

This means that :direct=<host>:<port> is specific to Dremio's legacy driver. There's also jdbc:dremio:zk=<host>:<port> (docs) which we probably don't want to add to FlightSQL driver.

I'm not familiar with Looker but it seems that you'd always need to place a jar in a server. Would forking this repository, applying the fix and creating a custom JAR work as a temporary solution for you?
Can you also link to where you raised this issue on Looker so that we can chip in?

[amirhzd]

Thanks for the comment, @xborder.

The feature request I created on Looker's Pendo portal is titled "Support for Apache Arrow Flight SQL driver (via Dremio) in self-hosted Looker." Here's the link: https://portal.feedback.us.pendo.io/app/#/case/534655?currentProductId=53ee6715-9dcb-471f-bd41-7846ac4d9e41&cpid=8141b66d-4d40-4b81-90a8-a88fd509df27.

You'll need to sign in to Pendo to view it, as far as I know. I'd really appreciate it if you could upvote it or otherwise help give it more visibility.

Forking Arrow Java has been discussed internally, but it didn't receive enough support in the first round. That said, it doesn't change how important it is for us to use the Arrow Flight SQL driver.

Another option would be to write a small custom JDBC driver in Java that acts as a bridge between Looker and the official Arrow Flight SQL driver. I may give that a try. If anyone has built something similar before, I'd really appreciate any advice or lessons learned.