Databricks
Connect a Databricks SQL warehouse with a read-only service principal, what it costs, how to set it up, and what each error means.
A Databricks connection reads selected tables and views from one Unity Catalog
catalog. Xentropy talks to your workspace over HTTPS: it lists tables through
the Unity Catalog API and runs its own single SELECT statements on a SQL
warehouse that you choose. It never writes to your workspace.
Availability
Databricks is available as early access on selected accounts. It is not
generally available yet. Until it is enabled for your account, it does not
appear in the engine list and its reads fail with PROVIDER_DISABLED.
What it costs
Warehouse costs
Test and listing tables use Unity Catalog metadata only. This does not start your warehouse and costs nothing. The data sample and every read run SQL on your warehouse, which starts it if it is stopped. Databricks bills you for the time the warehouse runs, and an idle warehouse keeps billing until it auto-stops.
- Use a serverless SQL warehouse. It starts in about 7 seconds, so a read on a stopped warehouse still finishes in time.
- Set auto-stop low. In the Databricks UI the shortest auto-stop is 5 minutes on serverless and 10 minutes on Pro and Classic.
- Pro and Classic warehouses can take about 4 minutes to start. A read
waits at most 30 seconds, so the first read after the warehouse has been idle
fails with
DATABRICKS_WAREHOUSEand asks you to retry. Retry after a few minutes. - Scheduled refreshes each run one read, and the shortest schedule is every 15 minutes. If the warehouse has stopped in between, each refresh starts it again and it then runs until it auto-stops.
- To see what Xentropy runs, filter
system.query.historyby the query tagapp=xentropy. Each statement also carries aconnectiontag, a short hash that tells your connections apart.
Databricks has no cost cap per statement. Xentropy limits each read to 50,001 rows, cancels any statement that is still running at the read deadline, and sorts only by a declared primary key.
Before you start
- You need a Unity Catalog workspace on AWS, Azure or Google Cloud, a SQL warehouse, and a workspace admin to create the service principal.
- The
hive_metastorecatalog is not supported. Workspaces reachable only through Private Link (public access turned off) are not supported. - If your workspace has an IP access list, see Outgoing IP addresses.
Set up a read-only service principal
These are the steps we verified on a live workspace. The form shows them too, with the SQL filled in from what you enter.
xentropy-reader. On Azure, choose a Databricks-managed service principal.databricks-sql-access). Without it, Databricks refuses every request.-- Run as a catalog admin. Use the service principal's client ID (application ID).
GRANT USE CATALOG ON CATALOG <catalog> TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<client-id>`;
GRANT SELECT ON SCHEMA <catalog>.<schema> TO `<client-id>`;The service principal needs nothing else: no MODIFY, no CREATE, and no
admin rights.
When Xentropy signs in, it asks Databricks for the OAuth scope
sql:read:restricted-query unity-catalog. With that scope, Databricks itself
refuses any statement that would change data, and Unity Catalog can be read
for the table list. The token lasts one hour and is kept only in memory; it
is never stored.
Personal access token (legacy)
You can use a personal access token instead of a service principal. Databricks calls these tokens legacy: a workspace admin can turn them off, and Databricks revokes a token that has not been used for 90 days.
- Use a dedicated user or service principal that has the same entitlement, Can use and grants as above.
- Create the token in Settings → Developer → Access tokens, with a
lifetime. For a service principal, an admin runs
databricks token-management create-obo-token. - If Databricks lets you choose scopes for the token, include
sqlandunity-catalog.
Set up the connection
| Field | What to enter |
|---|---|
| Workspace URL | The address you use to open Databricks, such as https://dbc-a1b2345c-d6e7.cloud.databricks.com, https://adb-1234567890123456.7.azuredatabricks.net or https://1234567890123456.7.gcp.databricks.com. You can paste the whole URL; Xentropy keeps only the host name. Account console URLs (accounts.…) and regional URLs are refused. |
| SQL warehouse | The warehouse ID, or its HTTP path (/sql/1.0/warehouses/<id>) from the warehouse's Connection details tab. |
| Catalog | The Unity Catalog catalog to read. One catalog per connection. |
| Schema (optional) | Only list relations from this schema. Leave it empty to list every schema the principal can read. |
| Authentication | Service principal (OAuth), recommended, or Personal access token. |
| Client ID and Client secret | From the service principal (OAuth only). |
| Access token | The personal access token (token only). |
| TIMESTAMP_NTZ time zone | The time zone your TIMESTAMP_NTZ values are written in. The default is UTC. |
The client secret and the token are encrypted at rest and never shown again. If you later replace the configuration and leave the secret empty, Xentropy keeps the stored one only while the workspace, sign-in method and client ID stay the same.
What Xentropy lists and reads
- Relations are listed as
schema.tablefrom the chosen catalog, using Unity Catalog. Only what the principal can see is listed, withoutinformation_schema, up to 500. - Tables (managed, external and shallow clones), views, materialized views, streaming tables and foreign tables can be selected. Metric views cannot.
- When you save, Xentropy reads each selected table's Unity Catalog entry. That
proves the grants without starting the warehouse. A table whose columns
Unity Catalog does not return (such as Delta Sharing tables) is described
with a zero-row
SELECT, which does start the warehouse. - Declared primary keys set the order and grain, marked as not verified, because Databricks does not enforce them. Declared foreign keys to other selected tables in the same catalog become relationships.
Type mapping
| Databricks type | Dataset type |
|---|---|
BOOLEAN | boolean |
TINYINT, SMALLINT, INT, FLOAT, DOUBLE | number |
BIGINT, DECIMAL | decimal (exact, as text) |
STRING, CHAR, VARCHAR, TIME, INTERVAL | string |
DATE | date |
TIMESTAMP | datetime (UTC) |
TIMESTAMP_NTZ | datetime, read in the connection's time zone and returned in UTC |
ARRAY, MAP, STRUCT, VARIANT | Flattened. A column whose values are all objects becomes dotted fields (up to 4 levels). Lists and mixed values stay one field, as JSON text |
BINARY, GEOGRAPHY, GEOMETRY, and other types | excluded |
Limits
| Limit | Value |
|---|---|
| Rows per read | 50,000 (Xentropy asks for 50,001 to tell when there are more) |
| Data per read | 5 MiB |
| Readable fields | 100, including flattened fields |
| Read deadline | 30 s. Databricks has no per-statement timeout, so Xentropy cancels the statement just before the deadline |
| Data sample | 100 rows, 30 s |
| Test and metadata calls | 10 s |
| Relations listed / selected | 500 / 100 |
An agent reads a Databricks connection with connections:read or
databricks:read.
Outgoing IP addresses
Databricks IP access lists apply to API calls too. They are not available on every Databricks plan; if your workspace has one, Databricks refuses requests from addresses it does not list.
When Xentropy publishes its outgoing IP addresses (egress IPs), the form shows them under
the fields, each with a copy button, and agents get the same list as
egressIps from GET /api/v1/data/providers. Add them to the workspace's IP
access list. If the form says Xentropy does not publish fixed IP addresses yet,
a workspace with an IP access list cannot be connected yet.
Errors
Test reports the first stage that fails, and the form labels the stage. Messages name only what you typed: the workspace, client ID, warehouse ID, catalog or relation. Databricks' own error text is never shown.
| Code | Cause | What to do |
|---|---|---|
DATABRICKS_DNS | The workspace host does not resolve | Check the workspace URL |
DATABRICKS_NETWORK_POLICY | The workspace host resolves to a private or reserved address (403). Or the workspace refused the request before sign-in (403, details.stage = ip_access_list): its IP access list does not allow Xentropy, or the principal lacks permission for the API | Use the public workspace URL. Check the IP access list and the principal's entitlement |
DATABRICKS_TCP | The workspace could not be reached, or the request timed out (503) | Retry later |
DATABRICKS_TLS | The workspace's certificate could not be verified | Check the workspace URL |
DATABRICKS_AUTH | The client ID and secret were rejected, the token is revoked, expired or turned off, or Databricks did not grant the sql and unity-catalog scopes | Check the client ID and secret, or create a new secret or token |
DATABRICKS_PERMISSION | The principal lacks the databricks-sql-access entitlement, or the catalog or a selected table does not exist or is not granted | Turn on the entitlement, grant USE CATALOG, USE SCHEMA and SELECT, or deselect the table |
DATABRICKS_WAREHOUSE | The warehouse ID is wrong, the principal has no Can use on it, it is deleted or unhealthy, or it was still starting at the read deadline (503) | Grant Can use or fix the ID. If it was starting, retry in a minute (Pro and Classic: a few minutes) |
INCOMPLETE_DATA | The relation has more than 50,000 rows, or more than 5 MiB | Select a smaller view |
LIMIT_EXCEEDED | More than 100 readable fields | Select a view with fewer columns |
BUDGET_EXCEEDED | The statement was still running at the read deadline and was cancelled | Select a smaller view, or use a larger warehouse |
PROVIDER_UNAVAILABLE | Databricks was busy, under maintenance or failed for another reason (503) | Retry later |
Scheduled refreshes retry DATABRICKS_WAREHOUSE automatically. A failed
refresh keeps the previous data.
Snowflake
Connect Snowflake with a key-pair service user and a SELECT-only role, how warehouse credits are used, and how column types map.
Google BigQuery
Connect BigQuery with a read-only service account key, read tables at no query cost, and run SELECT queries under a dry-run estimate and a hard cost cap.