2.3.13 Release Note

This is a great milestone for Apache SeaTunnel project! The release of version 2.3.13 with its so many new features and fixes will significantly boost the efficiency of data integration workflows for our users.

We’re really impressed by the dedication and hard work our contributors have put into this update, and we're confident this new version will be widely welcomed by the community.

Feature

• [Feature][Connector-V2] Support sync_mode=update for FtpFile/SftpFile/LocalFile source (binary) (#10437)
• [Feature][Connector-V2][Elasticsearch] Add Runtime Fields support (Elasticsearch 7.11+) (#10201)
• [Feature][Connector-v2]Resolve null first column in CSV Reader (#10383)
• [Feature][Core][Checkpoint] Add checkpoint api (#10065)
• [Feature][Connector-v2][Kafka] Add support for Schema Registry wire format in Protobuf deserialization (#10183)
• [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read (#10332)
• [Feature] [Connector-V2][HBase] Support time-range scan with min/max timestamp in HBaseSource (#10318)
• [Feature][Iceberg] Support schema.partition_keys and ${partition_keys} placeholder (#10389)
• [Feature][Zeta] Support the min-pause configuration for checkpoints (#9804)
• [Feature][Connector-V2] Support hdfs file multi table source read (#9816)
• [Feature][Transform-V2] Support multimodal embeddings (#9673)
• [Feature][Transform-V2] Support RegexExtract Transform (#9829)
• [Feature][Connector-V2] Support `` for the SQL To Paimon converter (#10206)
• [Feature][Kafka source] Inject Kafka record timestamp as EventTime metadata (#9994)
• [Feature][Connector-V2] Postgresql support read TIMESTAMP_TZ (#10048)
• [Feature][Connector-V2] Support sink connector for Apache HugeGraph (#10002)
• [Feature][Connector-V2] Support HDFS ViewFs Schema (#10117)
• [Feature][Core] Add REST API for viewing Pending queue details (#10078)
• [Feature][Connector-V2] Hive sink support SchemaSaveMode and DataSaveMode (#9743)
• [Feature][Connector-V2] Support the dynamic options of the paimon table read (#9981)
• [Feature][Connector-V2][Milvus] Sink writer flush by interval (#9961)
• [Feature][Core] Add PendingJobs info in the response of GetOverviewOperation (#9902)
• [Feature][Connector-V2] Support sink connector for AWS DSQL [#9726] (#9739)
• [Feature][Connector-V2] Support row range boundaries for HBaseSource (#9983)
• [Feature][Hive] Kerberos renew for >24h Hive and HDFS writes (#9989)
• [Feature][Flink] Support CDC source schema evolution on Flink engine (#9867)
• [Feature][Connector-redis] fix redis cluster bug and add cluster e2e (#9869)
• [Feature][Format] Improve maxwell_json,canal_json,debezium_json format support merge update_before and update_after (#9805)
• [Feature][File] Add markdown parser documentation (#9834)
• [Feature][File] Add markdown parser for RAG support [#9714] (#9760)
• [Feature][Connectors-v2] Clean up temporary files for paimon sink (#9819)
• [Feature][Connector-V2] Support CDC mode for databend sink connector (#9661)
• [Feature][Engine]Metalake support for data source information storage and management (#9688)
• [Feature] [Connector-V2] [IoTDB] Added source and sink connectors to support IoTDB 2.X (#9872)
• [Feature][connector-jdbc] Support source and sink of DuckDB (#10285)
• [Feature][Connector-V2] Support write data into lance dataset (#9894)
• [Feature][Connector-File-Hadoop] Support sync_mode=update for HdfsFile source (binary) (#10268)
• [Feature][connectors-v2/connector-file] Support logical Parquet splits (#10239)
• [Feature][Connector-V2][Hive] Support multiple Hive metastore URIs for automatic failover (#10253)
• [Feature][Core][Metrics] Add sink committed metrics and calculate commit rate in job metrics summary (#10233)
• [Feature][connector-hive] Support regex and whole-database table_name for source (#10250)
• [Feature][connector-file-local] Localfile supports splitting and parallel reading of large files of csv, text, and json (#10142)
• [Feature][Flink] Add Flink version 1.20.1 support (#9576)
• [Feature][connector-kudu] Support regex and whole-database table_name for source (#10180)
• [Feature][Connector-V2] Mongodb Sink Support Multi Table (#9958)
• [Feature][Connector-V2] MongodbSink Support SaveMode (#9883)
• [Feature][Core] Support define any nested array and map type in Zeta (#9881)
• [Improve][Zeta] Improve job metrics handling with partitioning support (#9833)
• [Improve][Connector-Clickhouse] improve ck batch parallel read by using last batch row sorting value approach, instead of limit offset. (#9801)
• [Improve][Connectors-v2] Add support for re-signaling NoMoreSplitsEvent after reader re-registration (#10208)
• [Improve][Core][Metrics] Refactor and enhance job metrics handling for multiple sinks with updated tests (#10273)
• [Improve][Zeta] engine-server and seantunnel-ui support remote paginated queries (#9951)
• [Improve][Common] Improve SinkFlowTestUtils Checkpoint Logic (#10071)
• [Improve][seatunnel-api] Unify non-relational data source multi-table mode schema configuration parameters (#10370)
• [Improve][Docs] Add S3 Compatible Storage IMAP configuration example. Closes [#10308] (#10342)
• [Improve] cdc related options (#10372)
• [Improve][doc]Optimize the introduction of schema parameters in documents (#10385)
• [Improve][doc] move duckdb doc (#10382)
• [Improve][E2E] Improve redis e2e for delete (#10018)
• [Improve][Core][Metrics] Optimize Sink Write Data Statistics (#10100)
• [Improve][Connector-V2][WeChat] Optimize StringBuffer to StringBuilder for better performance (#10127)
• [Improve][Fink] NO_CDC source support checkpoint (#10094)
• [Improve][Docs] Supplement Chinese documentation for SeaTunnel connectors (#10109)
• [Improve][Connector-v2][MySQL] Optimize shard calculation strategy (#9975)
• [Improve] Support JSON multi-level nested parsing (#10000)
• [Improve][Docs] Add incompatible changes documentation for version upgrades (#10068)
• [Improve][e2e] Change Docker images to use seatunnelhub repository (#10072)
• [Improve][Connector-V2][HBase] Support configurable range scan boundary inclusion policies (#10011)
• [Improve][E2E] update jdk docker image to seatunnelhub (#10036)
• [Improve][seatunnel-connectors-v2][connector-mongodb] Improve MongodbWriter close (#10051)
• [Imporve][Connector-v2][MySQL-CDC]Schema evolution optimization waiting time (#10040)
• [Improve][E2E] Improve redis test for delete hash (#9946)
• [Improvement][Connector] Upgrade Kafka client version from 3.2.0 to 3.4.0 (#9941)
• [Improve][CI] Optimize CI (#9900)
• [Improve][Core][Starter] Support encryption and decryption for transform config (#9932)
• [Improve][CI] Increase timeout for kafka-connector-it from 120 to 140 minutes (#9893)
• [Improve][Connector-V2] MongodbSinkFactory implements TableSinkFactory (#9879)
• [Improve] Update snapshot version to 2.3.13 (#9858)
• [Improve][Connector-V2][Hive] Deduplicate Hive option definitions (#10251)
• [Improve][Connector-V2][Elasticsearch] Enhance Scroll API handling with resource cleanup (#10124)
• [Improve][Connector-v2][Hive] Add socket and connection timeout options for Hive JDBC connections (#10254)
• [Improve][Core] Auto-create GC log directory to avoid startup failures (#10255)
• [Improve][e2e] Increase timeout for DatabendCDCSinkIT to reduce test flakiness (#10269)
• [Improve][Connector-V2][Jdbc] Increase timeout for schema change IT to fix flaky test (#10258)
• [Improve][connector-elasticsearch] Optimize Elasticsearch Sink vector parameters and documents (#10260)
• [Improve][Zeta] Remove unused fields (#10261)
• [improve] the substring and substr functions support Date,LocalDate,L… (#9969)
• [Improve][seatunnel-api] Optimize metadata lake related code (#10347)
• [Improve][Docs][seatunnel-engine] Add namespace to plugin-config (#10271)
• [Improve][Connector-V2][HBase] Support DATE/TIME/TIMESTAMP/DECIMAL in sink and fix DECIMAL deserialization (#10291)
• [Improve][Docs][connector-file] Document SaveMode options for HdfsFile/LocalFile (#10283)
• [Improve][Docs] Add multiple table sync examples for MySQL Sink connector (#10241)
• [Improve][seatunnel-connectors-v2][connector-jdbc] Remove some obsolete code in JdbcSinkFactory (#10026)
• [Improve][CI] Increase kudu connector timeout to 60 minutes (#10240)
• [Improve][Doc] Update table_path examples for clarity and consistency (#10202)
• [Improve][Connector] Replace printStackTrace with proper logging (#10176)
• [Improve][Connecotr-v2][Databend] Optimize CI judgment logic (#10130)
• [Improve][Docs] Fix word spelling errors (#10170)
• [Improve][Zeta] Filter tasks and pipelines by state (#9926)

Docs

• [DOC]SQL UDF docs hardcode 2.3.2 dependencies causing runtime mismatc… (#10445)
• [Feature][Doc] correct some issues for Readme.md (#10424)
• [Doc] Adjust the file structure under concepts (#10420)
• Update README.md
• [Doc] Fix the Deepwiki display in READE.md (#10438)
• [Doc][MongoDB] Update CDC documentation for role and user creation with clearer examples (#10423)
• docs: align EN architecture docs with Zeta implementation
• [Feature][Doc] correct some issues for Readme.md
• [Doc]Adjust Seatunnel Doc Structure (#10395)
• [Doc] Seatunnel Doc Structure Adjustment (#10351)
• [Add][Docs] Introduce SeaTunnel MCP and x2SeaTunnel documentation on website (#10108)
• [doc][transforms-v2] improve meatadata and rowkind doc (#10010)
• [Docs][Transform-v2][Embedding] Fix transform embedding option name (embedding_model_provider -> model_provider) (#9925)
• [DOC]fix Elasticsearch source docs (options table, typos) (#10259)
• [Chore] fix typos coolumn ---> column (#9830)
• [Docs][Connector-V2][S3File] Fix connector s3file option name (access_secret -> secret_key) (#9892)
• [DOC] Optimize Seatunnel Document Structure (#10262)
• [DOC][connector-hbase] Add Kerberos configuration notes/examples and fix h

Bug Fix

• [Fix][Connector-V2] Fix wrong column discovery when tableNamePattern is treated as LIKE (#10422)
• [Fix][Connectors-v2] Fix transaction cancellation failure after broker recovery (#10013)
• [Fix][connector-cdc] Fix NPE when finishing snapshot split due to null splitId (#10404)
• [Fix][Transform-V2] Fix multiTable SQL transform (#10263)
• [Fix][Connector-V2][SelectDBCloud] Fix NotSerializableException in engine sink action (#10412)
• [Fix][Connector-v2][MongoDB] There is a problem with using Cache for multi-task submission (#10116)
• [Fix][Zeta] Fix memory leak when cancelling pending job (#10315)
• [Fix][Zeta] Fix Write Count Display Issue in Multi-Sink Scenarios (#10376)
• [Fix][Connector-V2] Fix NPE when heartbeat.action.query is configured (#10278)
• [Fix][docs]fix schema-feature.md doc error (#10413)
• [Fix][connector-hbase] Fix namespace handling for HBase source (#10295)
• [Fix][Connector-v2][SQLServer] Handle database names with special characters in SQL queries and add related tests (#10327)
• [Fix][Connector-Jdbc] Bump Redshift JDBC driver from 2.1.0.9 to 2.1.0.30 to fix OOM issues (#10393)
• [Fix][connector-fake] Signal no more splits to avoid FakeSourceReader wait-split hang after restore (#10275)
• [Fix][Connector-V2] Fix partitioning column selection logic (#10319)
• [Fix][Connector-V2][Databend] Ensure CDC final merge on committer close (#10349)
• [Fix][Transform-V2][DataValidator] Fix ROUTE_TO_TABLE routing when tableId has db/schema prefix (#10368)
• [Fix][connector-jdbc] Fix CatalogUtils getCatalogTable(Connection, String, ...) losing primary key for query-only sources (#10093)
• [Fix][Connector-v2][Doris] Array type data parsing failed (#10095)
• [Fix][Connector-v2][MongoDB-CDC] When starting multiple tasks to connect to different MongoDB services, only the first one takes effect (#10064)
• [Fix][Connector-V2] Ignore JdbcConnectorException on JDBC statement close (#9942)
• [Fix][doc] Add missing closing braces in SqlServer-CDC documentation (#10073)
• [Fix][Connector-V2][Clickhouse] Add primary key support in ClickhouseSourceFactory (#10067)
• [Fix][seatunnel-spark-starter] Fix when Spark submits tasks in the fo… (#10033)
• [Fix][Connector-V2][Http]Fix setting of HTTP parameters in HttpSink.java (#10053)
• [Fix][Connector-v2][Clickhouse] Use actual table schema instead of DESC results for PhysicalColumn creation (#10019)
• [Fix][Connector-v2][Clickhouse] Support clickhouse log engine (#10035)
• [Fix][Transform] Fix ClassCastException in SQL Transform with TRIM(Expression) (#10031)
• [Fix] logs http basic issue # 9755 (#9968)
• [Fix][Connector-v2][Clickhouse]Fix filter_query not working in SQL batch read strategy (#10009)
• [Fix][e2e] Fix flaky Redis tests (#10024)
• [Fix][Connecotr][Hive] correct password parameter assignment in HiveJdbcConnectionProvider (#10025)
• [Fix][Core]update JDK docker image version (#10023)
• [Fix][Config] Fix type conversion errors caused by conflicting field names (#9960)
• [Fix][Zeta] Handle null log file name response (#9977)
• Fix[CI] Update the MySQL Docker image version to 8.0.43 (#9903)
• [Fix][Spark] Ensure order-independent schema merging by sorting catalog tables (#9911)
• [Fix][CI] Fix code style error (#9919)
• [Fix][Core] Add shade module for apache commons lang3 (#9895)
• [Fix][Connector-V2] Solve the reading of special excel (#9882)
• [Fix][Zeta] Fix pendingJobMasterMap resource leak (#9749)
• [Fix][UI] Optimize detail and finish-jobs components (#9888)
• [Fix][Connector-V2] Fix db2 table lower case bug (#9568)
• [Fix][UI] Optimize UI package (#9884)
• [Fix][Doc] Fix file_filter_modified_start and file_filter_modified_end format (#9880)
• [Fix][connector-hive] Fix the file_name_expression does not take effect in Hive sink. (#9823)
• [Fix][doc] Fix the ./mvnw install error according to the official documentation (#9853)
• [Fix][engine-server] clean BaseServletTest unit case logs dir (#9795)
• [Fix][Zeta] Fix JobStateEventTest to use await conditions for accessCounter assertions (#9842)
• [Fix][Zeta] Fix typo in isPhysicalDAGInfo (#9840)
• [Fix][Flink]Delete duplicate code (#9839)
• [Fix][Connector-xugu] Fix several bugs in the xugu connector (#9820)
• [Fix][Zeta] Fix seatunnel.sh - j command not working (#9832)
• [Fix][Zeta] Handle null startTimestamp in BaseService to fix flaky MysqlCDCWithBinlogDeleteIT test (#9815)
• [Fix][Connectors-v2] Fix UT for connector-http (#9821)
• [Fix][Connector-V2][ClickHouse] Fix ThreadLocal memory leak in ClickhouseCatalogUtil (#10264)
• [Fix][Connector-V2][Hbase] Fix ERROR_WHEN_DATA_EXISTS NPE on empty table (#10336)
• [Fix][Connector-V2] Bump jsch from 0.1.55 to 0.2.20 to fix buffer issues (#10298)
• [Fix][Connector-V2][GraphQL] Clarify subscription modes and JSON-only response (#10303)
• [Fix][Connector-V2][Hbase] Fix HBase sink binary rowkey handling (#10300)
• [Fix][HiveSink] Implement overwrite semantics for streaming commits to prevent multiple deletions of target directories (#10279)
• [Fix][Connector-V2][Hbase] Fix source reader only scanning first split (#10287)
• [Fix][Connecotr][MongoDB] Fix mongodb maxtime naming (#10267)
• [Fix][Connector-V2][Postgres-CDC] Fix PostgreSQL GEOMETRY handling with JDBC sink (#10186)
• [Fix][Zeta] Fix memory leak in SinkAggregatedCommitterTask (#10189)
• [Fix][Connector-V2][SqlServer-CDC] Fix earliest startup.mode offset and align tests (#10207)
• [Fix][E2E] Limit Flink restart attempts to avoid infinite retries masking real errors (#10152)
• [Fix][Transform-V2] Fix datetime/numeric/string function edge cases and update tests (#10168)
• [Fix][Connector-Kudu] Fix Doris auto table from Kudu STRING being created as CHAR(16) (#10175)
• [Fix][Connector-V2][Clickhouse] Handle special characters in column comments (#10128)
• [Fix][Connector-V2][HBase] Ensure fully qualified table name is used in tableExists method and add unit tests (#10126)
• [Fix][Connector-V2][Clickhouse] Fix ClickHouse catalog nullable column type and add tests (#10119)
• [Fix][Connector-V2] Fix 'file_filterpattern' cannot filter directories (#10049)
• [Fix][Bugfix] Doris Sink should not throw exception when the transaction is already committed/visible (#10079)
• [Fix][fix][e2e] Fix Databend e2e (#9985)
• [Fix][E2E] Fix DruidIT ConditionTimeout (#9979)
• [Fix][Connector-V2][Doris] Fix DECIMAL scale=0 conversion to avoid unintended MAX_SCALE (#9947)
• [Fix][Connector-V2] Fixed the problem of complex path reading of ftp (#9781)
• [Bug][Connector-V2][Hive] Fix init failure when Hive table/partition directory is empty (#10331)
• [Bug][connector-jdbc] Fix integer overflow when merging table schema with large column length (#10387)
• [Bug][Connector-V2][File] Fix IndexOutOfBoundsException when reading empty directories (#10373)
• [Bug][Connector-V2][Email] Fix NPE on null values, add configurable attachment name and delimiter (#10112)
• [Bugfix][seatunnel-connectors-v2][connector-elasticsearch] ElasticsearchSink encounters a writing error and the task does not exit (#10038)
• [Bugfix][seatunnel-e2e] Increase timeout (#10066)
• [Hotfix][CI] Fix TestSQLIT (#10087)
• [Hotfix][e2e] Fix MongoDB CI failure (#9997)
• [Hotfix] [connector-jdbc-e2e-part-7] CI failure caused by a conflict during container deletion (#9922)
• [HotFix][CI] Fix mysql image not found error (#9921)

Finally, let's remember the contributors who participated in contributing PRs this time. They are (in no particular order)

• Adam Wang
• AzkabanWarden.Gf
• Bo Schuster
• cloud456
• CloverDew
• corgy-w
• CosmosNi
• Cyanty
• David Zollo
• dotfive-star
• dy102
• dyp12
• Frui Guo
• Jarvis
• Jast
• Jeremy
• JeremyXin
• Jia Fan
• Joonseo Lee
• krutoileshii
• 老王
• Leon Yoah
• Li Dongxu
• LiJie20190102
• limin
• LimJiaWenBrenda
• liucongjy
• loupipalien
• mengxpgogogo-eng
• misi
• 巧克力黑
• shfshihuafeng
• silenceland
• Sim Chou
• Steven Zhao
• wanmingshi
• wtybxqm
• xiaochen
• yzeng1618
• zhan7236
• zhangdonghao
• zhuxt2015
• zy

Installation package download link: https://seatunnel.apache.org/download