SynapCores
Install free

SynapCores

SynapCores v2.4.0-ce: write your query results straight to the lake

Published on October 27, 2026

SynapCores Community Edition v2.4.0-ce

Community Edition · free · seamless upgrade from v2.3.x

SynapCores v2.4.0-ce is out. The headline: COPY ... TO LAKE — materialize the result of any SELECT (or WITH) as plain, partitioned Parquet in your own object store, in one SQL statement. No Airflow DAG, no Spark job, no separate export service. The query is the pipeline.

The job this removes

"Take this transformed slice of the database and land it in S3 as Parquet, partitioned by region, so the lakehouse and everything downstream can read it" is a task most teams solve with a second system — an orchestrator, a warehouse UNLOAD, or hand-rolled Python. That system has its own schedule, its own credentials, its own failure modes, and its own copy of your schema to keep in sync.

v2.4.0-ce makes it a statement:

COPY (SELECT id, metro, amount FROM permits WHERE amount > 0)
TO LAKE 'matches/'
WITH (CONNECTION = 'werkor_lake', PARTITION_BY = (metro),
      MODE = 'overwrite_partitions', COMPRESSION = 'zstd');

It streams the query straight to the writer — no REST response cap — and returns job_id, run_id, rows_written, files_written, bytes_written and partitions_touched. Partition columns become Hive directories (metro=nyc/…) and are dropped from the data files, so the output is standard foreign Parquet any engine can read. Register it back with CREATE EXTERNAL TABLE and query it in place:

CREATE EXTERNAL TABLE permit_matches STORED AS PARQUET
LOCATION 'matches/' PARTITIONED BY (metro TEXT)
WITH (CONNECTION = 'werkor_lake');

SELECT COUNT(*) FROM permit_matches WHERE metro = 'nyc';

Why it's safe to point at production storage

COPY TO LAKE writes real data to real buckets, so it is gated, not permissive:

  • A fresh, full-data-write principal — not an agent's operating privileges, not a read-only or table-scoped key.
  • An explicit destination write_allowlist grant. An empty allowlist — including on destinations upgraded from older versions — denies COPY entirely. Absolute paths, arbitrary S3 URLs, and managed-table ancestry are refused.
  • overwrite_partitions replaces only the partitions the query actually produced — an empty result deletes nothing. append only adds files.
  • Durable publication. Complete files are staged outside reader visibility, then published; an interrupted run is reconciled from a recovery journal before the next COPY. Inspect or cancel in flight via /v1/lake/copy/jobs and /v1/lake/runs.

Exact decimals and supported temporal types stay typed in the Parquet; writers emit row-group statistics, page indexes and high-cardinality bloom filters, so downstream partition pruning and predicate pushdown just work.

Honest limits. This is a one-shot export, not a writable external table or a continuous materialized view. Object-store listing is not a multi-file transaction — concurrent readers can briefly see a mix while new files replace old — and multi-partition publication is not a single table-wide transaction. Derived column-alias lists (... AS t(a, b)) over a non-VALUES relation aren't supported yet; COPY fails closed rather than guessing.

Your database answers questions out of the box now

A field test on a fresh install surfaced a boot-bricking default: a database question to the chat model came back 503 "requires an operator-approved SQL model" because the database_ask model allowlist shipped locked to one coder model. v2.4.0-ce ships that allowlist empty = any installed model. A clean install answers database questions on its configured chat model (qwen2.5:7b on ≥8 GiB) with zero config edits. A non-empty allowlist is now an optional operator policy knob — the generated SQL is always gated by read-only validation and table grants, so model choice was never the safety boundary.

Alongside it, better model lifecycle: native model eviction/swap (evict the idle LRU instead of returning a reservation-full 503, honor keep_alive=0 to unload, never evict an in-flight model), and the Ollama-compatible /api/version (pre-auth) and /api/tags (now works for scoped keys) endpoints behave correctly for drop-in clients.

Eight fixes from the field

From an M4 field test of v2.3.3-ce:

  • Calendar functions in RAG-scoped reads. CURRENT_DATE, DATE literals and INTERVAL no longer get a spurious 403 under the restricted-read gate — they read no extra data. Writes, DDL and impure functions stay denied.
  • No placeholder leaks. The answer step detects an unresolved slot and declines honestly instead of surfacing raw template text (SQL citations preserved).
  • SUM over integers returns BIGINT, not a float — on both the row and columnar paths.
  • Learning Loop console unblocked: the Database dropdown lists real databases (not the synthetic tenant_<id>), the plan view unwraps correctly instead of white-screening, and read-only authorization checks no longer bloat the audit chain — so "Verify audit" walks to completion and reports intact / partial / tampered distinctly.
  • A guard so a raw tool-call JSON blob can never be surfaced as a chat answer.

Get it

docker pull synapcores/community:latest    # v2.4.0-ce
# or
docker pull synapcores/community:v2.4.0-ce

Linux x86_64 / aarch64 and macOS (Apple Silicon) binaries are on the release page. COPY TO LAKE is in Community Edition — free. Full syntax is in the SQL reference and the in-engine sql_manual tool.

Point a SELECT at your lake and skip the pipeline.