### What problem does this PR solve?
Problem Summary:
When a non-GTID MySQL transaction is being delivered and the event
listener is blocked by backpressure, the keepalive thread can reconnect
after the in-memory binlog position has advanced into the transaction.
The reconnect then resumes from the middle of the transaction and can
silently skip rows.
This change tracks the start of an incomplete non-GTID transaction and
rewinds to that position before a keepalive reconnect. Completed
transactions clear the replay state. It also separates the CDC progress
heartbeat from the connection keepalive interval and uses the configured
progress heartbeat as the MySQL protocol heartbeat.
The Failsafe test classpath now uses the compiled classes directory so
integration tests exercise locally overridden dependency classes instead
of the repackaged application artifact.
### What problem does this PR solve?
The cdc_client builds debezium's `ChangeEventQueue` with only a
count-based bound (`max.queue.size=8192`) while the byte bound
(`max.queue.size.in.bytes`) defaults to `0` (disabled). With wide rows
(e.g. ~2MB each), the in-memory queue can grow to `2MB * 8192 ≈ 16GB`
and OOM the process. Both PostgreSQL and MySQL paths build the queue
from `getMaxQueueSizeInBytes()`, so a single property covers both, and
it applies to both the snapshot and streaming phases.
### What this PR does
Set a heap-adaptive byte cap on the queue buffer in
`ConfigUtil.getDefaultDebeziumProps()`, which is shared by the Postgres
and MySQL source readers:
- Default cap is `clamp(heap/16, 64MB, 256MB)`: heap 1G -> 64MB, 2G ->
128MB, >= 4G -> 256MB.
- The cap is intentionally conservative because a single cdc_client JVM
can run many queues concurrently (one per split, across multiple jobs),
and the real batching/backpressure happens downstream in the sink rather
than in this queue.
- Escape hatch: `-Dcdc.max.queue.size.in.bytes=<bytes>` overrides the
adaptive value (absolute bytes; `<= 0` disables the byte bound).
Narrow tables are unaffected: 8192 rows stay well under 64MB, so the
count bound is reached first and behavior is unchanged.
## What
Two test layers for the self-written logic in `fs_brokers/cdc_client`.
### Unit tests (no database / no Spring, plain `mvn test`)
- **Deserialization** (`DebeziumJsonDeserializer`): value conversion for
dates (incl. ancient / zero / pre-epoch — guards against calendar
drift), time, timestamp, zoned timestamp, decimal
(precise/string/double), bits, geometry (point → GeoJSON), arrays and
binary; plus the full `deserialize()` path — create/update/delete,
`__DORIS_DELETE_SIGN__`, column exclusion, non-data-change
short-circuit.
- **Postgres schema-change detection**
(`PostgresDebeziumJsonDeserializer`): WAL-based ADD/DROP column branches
— passthrough when no stored schema / no change, pure ADD, pure DROP,
simultaneous ADD+DROP (rename) skip, and the missing-refresher guard.
- **Sink helpers**: `BatchRecordBuffer` (delimiter / counting / clear),
`HttpPutBuilder` (headers, auth, txn, build precondition),
`LoadStatistic` (accumulation).
- **Factory / result**: `SourceReaderFactory`, `DeserializeResult`.
- **Config utils** (`ConfigUtil`): server-timezone parsing (mysql / pg /
default), `isJson`, `is13Timestamp`, exclude-columns / target-table
parsing, `toStringMap`.
### Testcontainers ITCases (JUnit5 + Testcontainers, real read path)
Drive the real production read path end to end against containerized
MySQL / PostgreSQL — `/api/fetchSplits` → `writeRecords` (from-to) and
`fetchRecordStream` (TVF) — with a mock stream-load / commit endpoint,
so no Doris cluster is needed. Intended to run on PR only when
`fs_brokers/cdc_client/**` changes.
- **from-to `writeRecords` path**: snapshot full-load, incremental
insert/update/delete, at-least-once offset resume, chunk-boundary
resume, data-type families, ancient dates, startup modes (latest /
specific-offset), per-column charset, server timezone, empty table,
Postgres schema change, Postgres default REPLICA IDENTITY.
- **TVF `fetchRecordStream` path**: MySQL snapshot + streamed
update/delete + offset resume; PostgreSQL full chain (snapshot +
incremental + pgoutput slot/publication).
- **Version smoke**: MySQL 5.7 / 8.4 and PostgreSQL 15 / 16 / 17 each
run a snapshot + DML chain (the default ITCases already cover 8.0 / 14).
Both read paths share the same reader / split preparation /
`deserialize()`, so type/charset/timezone/version decoding is covered
once and exercised from both entry points. Tests are isolated by a
unique jobId per method and close the process-wide `Env` reader in
teardown; incremental assertions poll with a timeout (no sleeps) to stay
non-flaky.
One production-side change rides along: `MySqlSourceReader` sets
`enable.time.adjuster=false` so genuinely ancient (<100) DATE/DATETIME
years are preserved — the ancient-date ITCase depends on it.
### What problem does this PR solve?
Related PR: #58898#59461
This PR primarily optimizes the speed of incremental and snapshot reads.
1. For incremental reads:
- Binding the fetch logic to an interval allows fetching data within
that interval.
- Splitting the fetch and write logic asynchronously.
2. For snapshot reads:
- Introducing the `snapshot_split_size` and `snapshot_parallelism`
parameters.
- `snapshot_split_size`: Adjusts the size of each chunk during the split
phase, allowing each split to fetch more data.
- `snapshot_parallelism`: The degree of parallelism during the snapshot
read phase, i.e., how many chunks can run simultaneously, and how many
chunks are scheduled in a single task.
### What problem does this PR solve?
This Issues (https://github.com/apache/doris/issues/58896) implements
multi-table synchronization in MySQL, The main purpose of this PR is to
extend the data source to Postgres.
wudi