Google BigQuery
Connect BigQuery with a read-only service account key, read tables at no query cost, and run SELECT queries under a dry-run estimate and a hard cost cap.
A BigQuery connection reads tables and standard-SQL SELECT queries with a
service account. Tables are read without a query job, so they cost no query
bytes. Every query is checked with a free dry run first and always runs with a
hard cap on the bytes it may bill.
Availability
BigQuery is not yet generally available on xentropy.ai. Contact
support@xentropy.ai to request access. Until it is enabled for your account,
it does not appear in the catalogue and its reads fail with
PROVIDER_DISABLED.
Create a service account and key
xentropy-reader.Keep the key file safe. Xentropy encrypts the whole key when you save it and never shows or returns the private key again.
Required roles
| Role | Grant it | Required |
|---|---|---|
roles/bigquery.dataViewer | On the datasets to read | Yes |
roles/bigquery.jobUser | On the billing project (queries and dry runs) | Yes |
Do not grant editor or admin roles. Data Viewer can read but not change data.
Xentropy also refuses every statement other than a single SELECT, whatever
the roles allow.
Set up the connection
US, EU or a region such as europe-west3.DATETIME values are written in (default UTC).auth (the key), permissions (listing datasets) and jobs (a free dry run of SELECT 1 in the location). Then choose Save connection.By default the connection reads the billing project. Through the API you can
list up to 10 projects to read (projects). Up to 500 tables are listed.
BigQuery needs no IP allowlist: Xentropy connects only to Google's API hosts.
Tables
Pick a table in the connection's resources, then its fields. Tables are read with BigQuery's table data API: no query job runs and no query bytes are billed. When a table has not changed since the last capture, the capture is reused.
Views, external tables and tables that require a partition filter cannot use the table data API. Xentropy reads them through a generated query, which has the same dry run and cost cap as your own queries. A table that requires a partition filter needs a partition range.
Queries
Choose BigQuery query in the resources, write one standard-SQL SELECT
statement, and reference tables as project.dataset.table.
- Choose Estimate. Xentropy runs a free dry run and shows the bytes the query would process next to the cap.
- Save as dataset stays disabled while the estimate is over the cap or the
statement is not a
SELECT. - Every read runs the dry run again before the query. The query job always
carries the cap (
maximumBytesBilled), so BigQuery fails a job that would bill more, without charging for it.
Scripts, DML (INSERT, UPDATE, DELETE, MERGE), DDL, EXPORT DATA and
CALL fail with QUERY_NOT_READ_ONLY. Queries always use standard SQL.
Cost
BigQuery bills your billing project, not Xentropy. A query is billed for at least 10 MiB. Scheduled refreshes run the query again each time, so keep queries selective and the cap low.
Type mapping
| BigQuery type | Dataset type |
|---|---|
INT64 | number. A value beyond ±2^53 fails with TYPE_MISMATCH: bind the field as a decimal |
NUMERIC, BIGNUMERIC | decimal (exact, as text) |
FLOAT64 | number. NaN and infinity become empty with a NONFINITE_NULLED warning |
BOOL | boolean |
STRING | string |
DATE | date |
DATETIME | datetime, read in the connection's time zone and returned in UTC |
TIMESTAMP | datetime (UTC) |
TIME, INTERVAL, GEOGRAPHY, RANGE | string |
STRUCT | flattened into dotted fields such as address.city |
ARRAY, JSON | JSON text |
BYTES | excluded |
Decimal fields
INT64 values above 2^53 cannot be held exactly as a number. If a read meets
one, it fails with TYPE_MISMATCH and names the field and row. Choose Bind as
decimal for that field in the field picker, or set decimalFields in the
dataset's BigQuery resource. The field is then read as an exact decimal.
Limits
| Limit | Value |
|---|---|
| Cost cap per read | Default 1 GiB, at least 10 MiB, at most 100 GiB |
| Rows per read | 50,000 |
| Fields per dataset | 100 |
| Pages per read | Free 100, Pro 600 |
| Projects read | 10 |
| Tables listed | 500 |
| Read deadline | 30 s interactive, 300 s scheduled |
Errors
| Code | Cause | What to do |
|---|---|---|
PROVIDER_ACCESS_DENIED, stage auth | The key was rejected, disabled or deleted | Create a new key and replace the configuration |
PROVIDER_ACCESS_DENIED, stage permissions | The service account cannot list or read the datasets | Grant BigQuery Data Viewer |
PROVIDER_ACCESS_DENIED, stage jobs | The service account cannot run jobs in the billing project | Grant BigQuery Job User on the billing project |
COST_LIMIT_EXCEEDED | The estimate is over the cap, or BigQuery stopped the job at the cap. Nothing was billed | Narrow the query (fewer columns, a partition filter) or raise the cap |
QUERY_NOT_READ_ONLY | The SQL is not a single SELECT | Rewrite it as one SELECT |
TYPE_MISMATCH | An INT64 value beyond ±2^53 in a number field | Bind the field as a decimal |
RATE_LIMITED | BigQuery's rate limit or quota is used up | Retry after details.retryAt |
LIMIT_EXCEEDED | The result is too large, or more than 50,000 rows | Select fewer columns or rows |
PERMISSION_DENIED | The table is outside the projects the connection reads | Add the project, or choose another table |
VALIDATION_FAILED | BigQuery rejected the SQL or a parameter, or the key is not a service account key | Fix the SQL or the key |