Docs
Connections

Databricks

Connect a Databricks SQL warehouse with a read-only service principal, what it costs, how to set it up, and what each error means.

A Databricks connection reads selected tables and views from one Unity Catalog catalog. Xentropy talks to your workspace over HTTPS: it lists tables through the Unity Catalog API and runs its own single SELECT statements on a SQL warehouse that you choose. It never writes to your workspace.

Availability

Databricks is available as early access on selected accounts. It is not generally available yet. Until it is enabled for your account, it does not appear in the engine list and its reads fail with PROVIDER_DISABLED.

What it costs

Warehouse costs

Test and listing tables use Unity Catalog metadata only. This does not start your warehouse and costs nothing. The data sample and every read run SQL on your warehouse, which starts it if it is stopped. Databricks bills you for the time the warehouse runs, and an idle warehouse keeps billing until it auto-stops.

  • Use a serverless SQL warehouse. It starts in about 7 seconds, so a read on a stopped warehouse still finishes in time.
  • Set auto-stop low. In the Databricks UI the shortest auto-stop is 5 minutes on serverless and 10 minutes on Pro and Classic.
  • Pro and Classic warehouses can take about 4 minutes to start. A read waits at most 30 seconds, so the first read after the warehouse has been idle fails with DATABRICKS_WAREHOUSE and asks you to retry. Retry after a few minutes.
  • Scheduled refreshes each run one read, and the shortest schedule is every 15 minutes. If the warehouse has stopped in between, each refresh starts it again and it then runs until it auto-stops.
  • To see what Xentropy runs, filter system.query.history by the query tag app = xentropy. Each statement also carries a connection tag, a short hash that tells your connections apart.

Databricks has no cost cap per statement. Xentropy limits each read to 50,001 rows, cancels any statement that is still running at the read deadline, and sorts only by a declared primary key.

Before you start

  • You need a Unity Catalog workspace on AWS, Azure or Google Cloud, a SQL warehouse, and a workspace admin to create the service principal.
  • The hive_metastore catalog is not supported. Workspaces reachable only through Private Link (public access turned off) are not supported.
  • If your workspace has an IP access list, see Outgoing IP addresses.

Set up a read-only service principal

These are the steps we verified on a live workspace. The form shows them too, with the SQL filled in from what you enter.

Create a service principal. A workspace admin opens Settings → Identity and access → Service principals and adds one, for example xentropy-reader. On Azure, choose a Databricks-managed service principal.
Turn on SQL access. On the service principal's Configurations tab, turn on the Databricks SQL access entitlement (databricks-sql-access). Without it, Databricks refuses every request.
Let it use the warehouse. Open the SQL warehouse, choose Permissions, add the service principal and give it Can use.
Grant read access. A catalog admin runs the grants below. Repeat the last two statements for each schema you want to read.
Create an OAuth secret. On the service principal's Secrets tab, choose Generate secret, with a lifetime of up to 730 days (180 is a good start). Copy the client ID and the secret now: Databricks shows the secret only once.
-- Run as a catalog admin. Use the service principal's client ID (application ID).
GRANT USE CATALOG ON CATALOG <catalog> TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<client-id>`;
GRANT SELECT ON SCHEMA <catalog>.<schema> TO `<client-id>`;

The service principal needs nothing else: no MODIFY, no CREATE, and no admin rights.

When Xentropy signs in, it asks Databricks for the OAuth scope sql:read:restricted-query unity-catalog. With that scope, Databricks itself refuses any statement that would change data, and Unity Catalog can be read for the table list. The token lasts one hour and is kept only in memory; it is never stored.

Personal access token (legacy)

You can use a personal access token instead of a service principal. Databricks calls these tokens legacy: a workspace admin can turn them off, and Databricks revokes a token that has not been used for 90 days.

  • Use a dedicated user or service principal that has the same entitlement, Can use and grants as above.
  • Create the token in Settings → Developer → Access tokens, with a lifetime. For a service principal, an admin runs databricks token-management create-obo-token.
  • If Databricks lets you choose scopes for the token, include sql and unity-catalog.

Set up the connection

Open Connections, choose New connection, then Database, and choose Databricks as the engine.
Fill in the fields below.
Choose Test and list relations. Test checks that the warehouse exists and is healthy, without starting it.
Select the relations to read (up to 100), then choose Save connection.
FieldWhat to enter
Workspace URLThe address you use to open Databricks, such as https://dbc-a1b2345c-d6e7.cloud.databricks.com, https://adb-1234567890123456.7.azuredatabricks.net or https://1234567890123456.7.gcp.databricks.com. You can paste the whole URL; Xentropy keeps only the host name. Account console URLs (accounts.…) and regional URLs are refused.
SQL warehouseThe warehouse ID, or its HTTP path (/sql/1.0/warehouses/<id>) from the warehouse's Connection details tab.
CatalogThe Unity Catalog catalog to read. One catalog per connection.
Schema (optional)Only list relations from this schema. Leave it empty to list every schema the principal can read.
AuthenticationService principal (OAuth), recommended, or Personal access token.
Client ID and Client secretFrom the service principal (OAuth only).
Access tokenThe personal access token (token only).
TIMESTAMP_NTZ time zoneThe time zone your TIMESTAMP_NTZ values are written in. The default is UTC.

The client secret and the token are encrypted at rest and never shown again. If you later replace the configuration and leave the secret empty, Xentropy keeps the stored one only while the workspace, sign-in method and client ID stay the same.

What Xentropy lists and reads

  • Relations are listed as schema.table from the chosen catalog, using Unity Catalog. Only what the principal can see is listed, without information_schema, up to 500.
  • Tables (managed, external and shallow clones), views, materialized views, streaming tables and foreign tables can be selected. Metric views cannot.
  • When you save, Xentropy reads each selected table's Unity Catalog entry. That proves the grants without starting the warehouse. A table whose columns Unity Catalog does not return (such as Delta Sharing tables) is described with a zero-row SELECT, which does start the warehouse.
  • Declared primary keys set the order and grain, marked as not verified, because Databricks does not enforce them. Declared foreign keys to other selected tables in the same catalog become relationships.

Type mapping

Databricks typeDataset type
BOOLEANboolean
TINYINT, SMALLINT, INT, FLOAT, DOUBLEnumber
BIGINT, DECIMALdecimal (exact, as text)
STRING, CHAR, VARCHAR, TIME, INTERVALstring
DATEdate
TIMESTAMPdatetime (UTC)
TIMESTAMP_NTZdatetime, read in the connection's time zone and returned in UTC
ARRAY, MAP, STRUCT, VARIANTFlattened. A column whose values are all objects becomes dotted fields (up to 4 levels). Lists and mixed values stay one field, as JSON text
BINARY, GEOGRAPHY, GEOMETRY, and other typesexcluded

Limits

LimitValue
Rows per read50,000 (Xentropy asks for 50,001 to tell when there are more)
Data per read5 MiB
Readable fields100, including flattened fields
Read deadline30 s. Databricks has no per-statement timeout, so Xentropy cancels the statement just before the deadline
Data sample100 rows, 30 s
Test and metadata calls10 s
Relations listed / selected500 / 100

An agent reads a Databricks connection with connections:read or databricks:read.

Outgoing IP addresses

Databricks IP access lists apply to API calls too. They are not available on every Databricks plan; if your workspace has one, Databricks refuses requests from addresses it does not list.

When Xentropy publishes its outgoing IP addresses (egress IPs), the form shows them under the fields, each with a copy button, and agents get the same list as egressIps from GET /api/v1/data/providers. Add them to the workspace's IP access list. If the form says Xentropy does not publish fixed IP addresses yet, a workspace with an IP access list cannot be connected yet.

Errors

Test reports the first stage that fails, and the form labels the stage. Messages name only what you typed: the workspace, client ID, warehouse ID, catalog or relation. Databricks' own error text is never shown.

CodeCauseWhat to do
DATABRICKS_DNSThe workspace host does not resolveCheck the workspace URL
DATABRICKS_NETWORK_POLICYThe workspace host resolves to a private or reserved address (403). Or the workspace refused the request before sign-in (403, details.stage = ip_access_list): its IP access list does not allow Xentropy, or the principal lacks permission for the APIUse the public workspace URL. Check the IP access list and the principal's entitlement
DATABRICKS_TCPThe workspace could not be reached, or the request timed out (503)Retry later
DATABRICKS_TLSThe workspace's certificate could not be verifiedCheck the workspace URL
DATABRICKS_AUTHThe client ID and secret were rejected, the token is revoked, expired or turned off, or Databricks did not grant the sql and unity-catalog scopesCheck the client ID and secret, or create a new secret or token
DATABRICKS_PERMISSIONThe principal lacks the databricks-sql-access entitlement, or the catalog or a selected table does not exist or is not grantedTurn on the entitlement, grant USE CATALOG, USE SCHEMA and SELECT, or deselect the table
DATABRICKS_WAREHOUSEThe warehouse ID is wrong, the principal has no Can use on it, it is deleted or unhealthy, or it was still starting at the read deadline (503)Grant Can use or fix the ID. If it was starting, retry in a minute (Pro and Classic: a few minutes)
INCOMPLETE_DATAThe relation has more than 50,000 rows, or more than 5 MiBSelect a smaller view
LIMIT_EXCEEDEDMore than 100 readable fieldsSelect a view with fewer columns
BUDGET_EXCEEDEDThe statement was still running at the read deadline and was cancelledSelect a smaller view, or use a larger warehouse
PROVIDER_UNAVAILABLEDatabricks was busy, under maintenance or failed for another reason (503)Retry later

Scheduled refreshes retry DATABRICKS_WAREHOUSE automatically. A failed refresh keeps the previous data.

On this page