mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
wudi
d5aec29f76
### What problem does this PR solve? Previously, CDC streaming jobs could skip an unsupported source schema change and advance the source schema baseline without notifying the user. The job might then fail only when a later row could no longer be written to Doris. Users also had no explicit option to keep the target schema under manual control. This PR adds configurable schema-change handling for MySQL, PostgreSQL, and OceanBase **FROM ... TO ...** streaming jobs, with an explicit pause-and-resume recovery flow. It also allows changing two snapshot tuning properties through `ALTER JOB`. #### Schema-change behavior The new source property `schema_change_behavior` accepts: | Value | Behavior | | --- | --- | | `evolve` (default) | Apply supported ADD/DROP COLUMN changes. Pause on detected unsupported column changes and report the affected table and change reason. | | `ignore` | Skip target column DDL while updating the source schema baseline used to decode subsequent records. Normal data loading and its error handling still apply. | The property can be set at creation or changed while the job is paused. Columns excluded through `exclude_columns` are filtered before target column DDL and column-level unsupported-change checks; primary-key checks still apply. MySQL/OceanBase classify column operations from source DDL, including CHANGE, MODIFY, and RENAME. PostgreSQL uses relation-schema differences: native type OID changes and primary-key changes are unsupported, and simultaneous ADD/DROP is treated as a possible rename. PostgreSQL changes that retain the same type OID, such as length or precision adjustments, are not newly classified as unsupported by this PR. Index changes and table-level operations do not gain target-side replication in this change. In particular, source TRUNCATE or DROP TABLE does not truncate or drop the Doris table. #### Pause and recovery On a detected unsupported change, pending data is flushed and the job pauses with `[SCHEMA_CHANGE_UNSUPPORTED]`; automatic resume is disabled for this error. Detailed source schema information is logged for diagnosis. The user reviews the target schema, adjusts it if necessary, and runs `RESUME JOB`. This explicitly accepts unsupported changes until the next successful task commit. During that recovery round: - Required, non-excluded source column names must exist in the mapped Doris target table. Missing columns cause another pause with a diagnostic message. - Extra target columns are allowed. Target types and keys are deliberately not compared, so users can retain intentional source/target differences; subsequent writes still have to succeed. - The acceptance flag is cleared after a successful task and is not persisted across FE restarts. It can cover multiple unsupported changes encountered before that commit. For example, MySQL `MODIFY COLUMN` now pauses even if the target already has the column. If the user considers the target compatible, `RESUME JOB` permits replay after the column-name check. A rename requires manual target preparation and may also require keeping the old column available for DML replay from the last committed offset. An event containing both supported and unsupported operations is rejected before emitting any target DDL for that event. For example, ADD+MODIFY does not partially apply the ADD before pausing; recovery validates the full post-change column set. Supported ADD/DROP execution remains retryable: an already-present added column or already-absent dropped column is accepted after a target-schema check. #### Mutable snapshot settings `ALTER JOB` now accepts `snapshot_parallelism` and `snapshot_split_size` for CDC FROM-TO jobs and `cdc_stream` TVF jobs: - `snapshot_parallelism` affects subsequent snapshot task dispatch. - `snapshot_split_size` affects newly generated splits; existing persisted split boundaries are unchanged. - When `snapshot_parallelism` or `server_id` changes, validation uses the merged source properties so the retained MySQL server ID range still covers the effective parallelism. Source identity properties and `snapshot_split_key` remain immutable.