Docs
Connections

S3-compatible storage

Read Parquet, Avro, ORC, CSV, TSV, JSON, NDJSON and XML objects from Cloudflare R2, Amazon S3 or another S3-compatible endpoint.

An S3-compatible connection reads objects from one bucket, optionally under a prefix. It uses a read-only client that can only list objects and GET byte ranges, over the shared egress policy (verified TLS, pinned address, redirects refused). See Connections.

Availability

S3-compatible storage is not yet generally available on xentropy.ai. Contact support@xentropy.ai to request access. Until it is enabled for your account, it does not appear in the catalogue and its reads fail with PROVIDER_DISABLED.

Before you start

Create an access key that can only read the bucket, or the prefix within it. The key needs to list objects (ListBucket) and read them (GetObject). If your bucket policy restricts source IPs, allow the egress IP addresses shown in the form.

Set up the connection

Open Connections, choose New connection, then S3-compatible storage.
Choose the storage provider: Cloudflare R2, Amazon S3 or a custom endpoint.
Enter the endpoint or region, the bucket, an optional prefix, the access key ID and the secret access key. If the endpoint uses a private CA, paste its certificate.
Choose Test and list objects. Xentropy lists one key under the prefix to verify the endpoint, credentials and bucket, then lists the objects.
Pick an object, or write a glob, check the format and the field mapping, and save the dataset.
ProviderEndpointRegionAddressing
Cloudflare R2https://<account-id>.r2.cloudflarestorage.com (the eu and fedramp jurisdiction hosts are accepted)autoPath-style
Amazon S3Derived from the region (https://s3.<region>.amazonaws.com), or your own amazonaws.com endpointRequired, for example eu-west-1Virtual-hosted
Custom (MinIO and other S3-compatible services)Any HTTPS originDefault us-east-1Path-style
  • The endpoint must be an HTTPS origin, with no path, query or credentials. The bucket is a separate field.
  • Bucket names with dots always use path-style addressing, because a dotted name cannot match a wildcard certificate.
  • Leading slashes are removed from the prefix. A dataset can only read keys under the connection's prefix.

Objects and globs

A dataset reads either one key or one glob:

  • * and ? match within one path segment. ** matches across segments. Character classes ([...]) and braces ({...}) are not supported.
  • Hive-style partition directories (date=2026-01/…) become string fields, named after the partition key.
  • Every object a glob matches must have the same format and the same columns. Otherwise the read fails with SCHEMA_MISMATCH.

The format comes from the file extension. When the extension says nothing, Xentropy checks the object's first bytes: discovery checks up to 25 such objects. You can always set the format explicitly. For CSV and TSV you can also set the delimiter and whether there is a header row. For JSON and XML you can set the record path. See File uploads for the text mapping rules, which are the same here.

Formats

FormatLimitsNotes
CSV, TSV, JSON, NDJSON, XML, each optionally .gzSame size and row limits as uploadsUTF-8 only. Read as a stream with the same rules as uploads
Parquet, Avro (object container files), ORCEach object must decode within 60 s and 512 MiB of memoryDecoded in an isolated sandbox that sees only a private copy of the object, never your credentials

Columnar types map like this:

Source typeDataset type
DECIMAL (including Avro bytes decimals)decimal
64-bit and larger integersnumber when every value is within 2^53, otherwise decimal
Smaller integers, floats, doublesnumber
Booleanboolean
Datedate
Timestamp with time zone, Avro timestamp-millis and timestamp-microsdatetime (UTC)
Timestamp without time zonedatetime. You must set the field's time zone, otherwise the read fails with TIMEZONE_REQUIRED
StructFlattened to dotted fields (shipping.city)
List, map, unionJSON text
String, enum, UUID, time, intervalstring
BinaryBase64 text

Consistency

  • Every range read of one capture is pinned to the object's ETag. If the object is overwritten during the read, the read fails with SOURCE_CHANGED instead of mixing two versions. Retry the read.
  • A refresh records the key, ETag and size of every object it read. When none of them has changed, the previous snapshot is reused without reading the objects again.

Limits

LimitValue
One object64 MiB
Objects matched by a glob200
Total size of a glob256 MiB
Keys listed (discovery, or under a glob's literal prefix)5,000, in at most 20 pages
Decompressed gzip5× the compressed size, and at most 512 MiB per read
One range request8 MiB
Read deadline (all objects)120 s
Rows / fields / normalized data50,000 / 100 / 5 MiB

Errors

CodeCauseWhat to do
S3_DNSThe endpoint does not resolveCheck the endpoint hostname
S3_NETWORK_POLICYThe endpoint resolves to a private or reserved address (403)Use the public endpoint, or ask your administrator to allow it
S3_TLSThe endpoint's certificate does not verifyPaste the CA certificate for a private CA
S3_AUTHThe access key or secret was rejectedCheck both. For AWS, also check the region
S3_BUCKET_NOT_FOUNDThe bucket does not exist at this endpointCheck the bucket name, endpoint and region
S3_ACCESS_DENIEDThe key may not list or read this bucket or prefixGrant ListBucket and GetObject
S3_REDIRECTThe endpoint answered with a redirectFor AWS, choose the bucket's own region
NOT_FOUNDThe key does not exist, or the glob matches nothingCheck the key or glob
LIMIT_EXCEEDEDAn object, glob, row or size limit was exceededNarrow the glob, or split the objects
INCOMPLETE_DATAMore than 5,000 keys share the glob's prefixUse a longer literal prefix
SCHEMA_MISMATCHGlob members differ in format or columnsMatch objects of one schema
FORMAT_UNSUPPORTEDThe format could not be detected, or columnar decoding is not availableChoose the format, or use a text format
DECOMPRESSION_LIMITGzip expands beyond its limitsStore the object uncompressed, or split it
SOURCE_CHANGEDThe object changed during the read (409)Retry the read
TIMEZONE_REQUIREDA timestamp without zone has no time zone setSet the field's time zone
BUDGET_EXCEEDEDThe read ran past 120 sRead fewer or smaller objects
PERMISSION_DENIEDThe key is outside the connection's prefixUse a key under the prefix

On this page