SynapCores v2.4.7-ce: data-lake incremental exports catch late-arriving rows

SynapCores Community Edition v2.4.7-ce

Community Edition · free · seamless upgrade from v2.4.x

v2.4.7-ce fixes a quiet data-completeness gap in incremental data-lake exports: late-arriving rows no longer get stranded. If you export a day's data to Parquet early and more rows for that day land afterward, the lookback window now goes back and picks them up — so the partition reflects the full day once it closes, not just the slice that existed at first capture. One targeted engine fix, drop-in, no on-disk format change.

Trust the lookback window to actually backfill

Scheduled lake exports advertise a lookback_days window to re-visit recent days. Until now, once a day's partition had been written and verified, later runs skipped it — so the lookback window did nothing, and any rows that arrived after the first export for that day were never picked up.

As of v2.4.7-ce, within the lookback window an incremental export re-reads a verified partition when the source has grown for that day (or when the source can't be counted cheaply up front) and atomically replaces the partition with the complete day. The swap is safe for concurrent readers, and a partition that is genuinely unchanged is still skipped — no wasted re-reads.

Why it matters: event streams, ping and telemetry pipelines, and any ingest that buffers and flushes rows carrying their original timestamps are now captured in full on the next scheduled run, instead of being frozen at the fraction present at first export. If you've been seeing a Parquet day that reads lighter than the source for that same day, this is the release that closes the gap — going forward, each day completes the night after it closes.

Proven before shipping

• End-to-end: a partition first exported partial is re-read and completed on the following run; a genuinely unchanged partition is still skipped, and replacements are atomic for concurrent readers.
• The full lake test suite is green except one pre-existing, unrelated read-path case — identical pass/fail set to the previous release, so zero regression from this change.
• Clean boot plus model run on a non-AVX-512 CPU; embeddings serve at the default dimension.
• Drop-in upgrade: no on-disk format change. Existing Parquet partitions are read as-is, and the first run after upgrade simply completes any days still within the lookback window.

Get it

docker pull synapcores/community:v2.4.7-ce

Or upgrade in place — existing data and lake partitions are read as-is.

LM