S3-compatible storage
Read Parquet, Avro, ORC, CSV, TSV, JSON, NDJSON and XML objects from Cloudflare R2, Amazon S3 or another S3-compatible endpoint.
An S3-compatible connection reads objects from one bucket, optionally under a
prefix. It uses a read-only client that can only list objects and GET byte
ranges, over the shared egress policy (verified TLS, pinned address, redirects
refused). See Connections.
Availability
S3-compatible storage is not yet generally available on xentropy.ai. Contact
support@xentropy.ai to request access. Until it is enabled for your account,
it does not appear in the catalogue and its reads fail with
PROVIDER_DISABLED.
Before you start
Create an access key that can only read the bucket, or the prefix within it.
The key needs to list objects (ListBucket) and read them (GetObject). If
your bucket policy restricts source IPs, allow the egress IP addresses shown in
the form.
Set up the connection
| Provider | Endpoint | Region | Addressing |
|---|---|---|---|
| Cloudflare R2 | https://<account-id>.r2.cloudflarestorage.com (the eu and fedramp jurisdiction hosts are accepted) | auto | Path-style |
| Amazon S3 | Derived from the region (https://s3.<region>.amazonaws.com), or your own amazonaws.com endpoint | Required, for example eu-west-1 | Virtual-hosted |
| Custom (MinIO and other S3-compatible services) | Any HTTPS origin | Default us-east-1 | Path-style |
- The endpoint must be an HTTPS origin, with no path, query or credentials. The bucket is a separate field.
- Bucket names with dots always use path-style addressing, because a dotted name cannot match a wildcard certificate.
- Leading slashes are removed from the prefix. A dataset can only read keys under the connection's prefix.
Objects and globs
A dataset reads either one key or one glob:
*and?match within one path segment.**matches across segments. Character classes ([...]) and braces ({...}) are not supported.- Hive-style partition directories (
date=2026-01/…) become string fields, named after the partition key. - Every object a glob matches must have the same format and the same columns.
Otherwise the read fails with
SCHEMA_MISMATCH.
The format comes from the file extension. When the extension says nothing, Xentropy checks the object's first bytes: discovery checks up to 25 such objects. You can always set the format explicitly. For CSV and TSV you can also set the delimiter and whether there is a header row. For JSON and XML you can set the record path. See File uploads for the text mapping rules, which are the same here.
Formats
| Format | Limits | Notes |
|---|---|---|
CSV, TSV, JSON, NDJSON, XML, each optionally .gz | Same size and row limits as uploads | UTF-8 only. Read as a stream with the same rules as uploads |
| Parquet, Avro (object container files), ORC | Each object must decode within 60 s and 512 MiB of memory | Decoded in an isolated sandbox that sees only a private copy of the object, never your credentials |
Columnar types map like this:
| Source type | Dataset type |
|---|---|
DECIMAL (including Avro bytes decimals) | decimal |
| 64-bit and larger integers | number when every value is within 2^53, otherwise decimal |
| Smaller integers, floats, doubles | number |
| Boolean | boolean |
| Date | date |
Timestamp with time zone, Avro timestamp-millis and timestamp-micros | datetime (UTC) |
| Timestamp without time zone | datetime. You must set the field's time zone, otherwise the read fails with TIMEZONE_REQUIRED |
| Struct | Flattened to dotted fields (shipping.city) |
| List, map, union | JSON text |
| String, enum, UUID, time, interval | string |
| Binary | Base64 text |
Consistency
- Every range read of one capture is pinned to the object's ETag. If the
object is overwritten during the read, the read fails with
SOURCE_CHANGEDinstead of mixing two versions. Retry the read. - A refresh records the key, ETag and size of every object it read. When none of them has changed, the previous snapshot is reused without reading the objects again.
Limits
| Limit | Value |
|---|---|
| One object | 64 MiB |
| Objects matched by a glob | 200 |
| Total size of a glob | 256 MiB |
| Keys listed (discovery, or under a glob's literal prefix) | 5,000, in at most 20 pages |
| Decompressed gzip | 5× the compressed size, and at most 512 MiB per read |
| One range request | 8 MiB |
| Read deadline (all objects) | 120 s |
| Rows / fields / normalized data | 50,000 / 100 / 5 MiB |
Errors
| Code | Cause | What to do |
|---|---|---|
S3_DNS | The endpoint does not resolve | Check the endpoint hostname |
S3_NETWORK_POLICY | The endpoint resolves to a private or reserved address (403) | Use the public endpoint, or ask your administrator to allow it |
S3_TLS | The endpoint's certificate does not verify | Paste the CA certificate for a private CA |
S3_AUTH | The access key or secret was rejected | Check both. For AWS, also check the region |
S3_BUCKET_NOT_FOUND | The bucket does not exist at this endpoint | Check the bucket name, endpoint and region |
S3_ACCESS_DENIED | The key may not list or read this bucket or prefix | Grant ListBucket and GetObject |
S3_REDIRECT | The endpoint answered with a redirect | For AWS, choose the bucket's own region |
NOT_FOUND | The key does not exist, or the glob matches nothing | Check the key or glob |
LIMIT_EXCEEDED | An object, glob, row or size limit was exceeded | Narrow the glob, or split the objects |
INCOMPLETE_DATA | More than 5,000 keys share the glob's prefix | Use a longer literal prefix |
SCHEMA_MISMATCH | Glob members differ in format or columns | Match objects of one schema |
FORMAT_UNSUPPORTED | The format could not be detected, or columnar decoding is not available | Choose the format, or use a text format |
DECOMPRESSION_LIMIT | Gzip expands beyond its limits | Store the object uncompressed, or split it |
SOURCE_CHANGED | The object changed during the read (409) | Retry the read |
TIMEZONE_REQUIRED | A timestamp without zone has no time zone set | Set the field's time zone |
BUDGET_EXCEEDED | The read ran past 120 s | Read fewer or smaller objects |
PERMISSION_DENIED | The key is outside the connection's prefix | Use a key under the prefix |