Commit Graph

71 Commits

Author SHA1 Message Date
bd51223855 feat: add truncate query handling in migration process; update job configuration to remove commented truncate SQL 2026-04-21 11:40:02 -05:00
9eb8800864 feat: add range configuration to job and update extractors for inclusive range handling 2026-04-21 11:29:34 -05:00
16c232762e feat: refactor extractor interface and implement Consume function for ETL process 2026-04-20 00:19:41 -05:00
7bd80d4180 feat: enhance logging configuration to use dynamic log level from environment variable 2026-04-19 23:06:13 -05:00
846a49d40c feat: implement GenericLoader for batch processing and utility functions 2026-04-17 15:58:08 -05:00
93b302db8e feat: refactor job result handling and remove unused files 2026-04-16 16:47:35 -05:00
0d0511716f feat: add row count mismatch error handling in processMigrationJob and update SQL query to exclude additional graph-related columns 2026-04-16 12:46:55 -05:00
df4c3bc390 feat: refactor db handling to use db-wrapper package; enhance connection management and result handling for MSSQL and Postgres 2026-04-16 08:48:29 -05:00
33c9cd9c3e feat: implement database wrapper interfaces for MSSQL and Postgres; enhance migration job processing with pre and post SQL execution 2026-04-13 07:57:18 -05:00
85074da2ec feat: add max partition and chunk error limits to extractor and loader error handlers 2026-04-12 20:57:31 -05:00
f126d5bbd0 feat: implement exponential backoff strategy for error handling in extractor and loader processes; enhance retry configuration options 2026-04-12 20:35:29 -05:00
01780b4b02 refactor: remove unused ColumnType and inspect-columns files; update migration job to use separate table analyzers for source and target databases 2026-04-12 19:16:14 -05:00
4d3cd6e4cf feat: add MSSQL table analyzer and integrate partition range generation for improved data migration 2026-04-11 01:23:13 -05:00
7830ae862d refactor: rename batch-related variables and functions for consistency and clarity 2026-04-11 00:44:12 -05:00
955bc65ce9 refactor: rename Batch to Partition in error handling and processing functions for consistency 2026-04-11 00:32:50 -05:00
9eb9821daf refactor: rename Batch to Partition and update related types and channels for consistency 2026-04-11 00:09:28 -05:00
cd0e53b1d2 feat: implement MSSQL extractor, transformer, and Postgres loader for enhanced data migration 2026-04-10 23:39:37 -05:00
1be7018ba3 feat: refactor configuration to include source and target database types 2026-04-10 22:58:57 -05:00
c1bae79f98 feat: implement Postgres loader and refactor migration job processing 2026-04-10 20:40:01 -05:00
053e6bd673 feat: add MSSQL extractor and transformer implementations for improved data migration 2026-04-10 19:59:44 -05:00
eb3c3bbfce feat: refactor error handling to use custom_errors.LoaderError for improved error management 2026-04-10 19:35:38 -05:00
9493a2d32f feat: refactor error handling to accept max retry attempts as a parameter for improved flexibility 2026-04-10 19:32:12 -05:00
d228a048b8 feat: update extractor error handling to use models.UnknownRowValues for improved type consistency 2026-04-10 19:29:07 -05:00
ca621352c9 feat: refactor models to improve type handling and enhance error management across migration processes 2026-04-10 19:27:27 -05:00
6345a0d694 feat: enhance migration job processing with detailed metrics and error handling 2026-04-09 21:55:19 -05:00
1db35c796c feat: enhance migration job processing with parallel execution and improved logging 2026-04-09 20:02:04 -05:00
0d9f955b2f feat: enhance batch processing by adding rowsPerBatch parameter and improving logging messages 2026-04-09 19:46:45 -05:00
524d892a60 feat: refactor migration job structure to use SourceTableInfo and TargetTableInfo for improved configuration handling 2026-04-09 19:20:50 -05:00
a0b51f40c1 feat: add context support to migration job processing for improved cancellation and error handling 2026-04-09 00:43:11 -05:00
b64a76ca45 feat: improve error handling and job cancellation in migration process 2026-04-09 00:38:16 -05:00
51480015ba feat: enhance concurrency management by adding WaitGroup support in extractors and loaders 2026-04-09 00:22:30 -05:00
dc632361e5 feat: implement loader error handling and refactor chunk processing in migration job 2026-04-08 23:42:31 -05:00
0ee5d9032c feat: add context support to error handlers for improved cancellation and error management 2026-04-08 23:07:41 -05:00
d3a3b26bb3 feat: enhance error handling and context management in MSSQL extraction process 2026-04-08 22:39:07 -05:00
554618daad feat: refactor transformation logic in MSSQL processing to use context and improve error handling 2026-04-08 21:47:24 -05:00
7924dd3af7 feat: refactor chunk handling in loader and transformer for improved data processing 2026-04-08 21:11:26 -05:00
f6dfcd390f feat: refactor chunk handling in extractor and transformer for improved data processing 2026-04-08 21:09:26 -05:00
853be4a5a6 feat: update chunk size for MSSQL processing and enhance error handling in transformation functions 2026-04-08 20:48:36 -05:00
eeef3bc813 fix: correct variable name for job error in MSSQL extraction 2026-04-08 20:39:16 -05:00
e158986947 feat: enhance error handling with JobError struct and update extractor logic 2026-04-08 20:21:58 -05:00
bc6f9a6a70 feat: implement batch processing for MSSQL with improved structure and logging 2026-04-08 19:20:09 -05:00
75b04d4b2e feat: add MANZANA migration job and update related processing logic 2026-04-08 10:16:27 -05:00
4434054b21 feat: implement data generation and loading for MANZANA with improved structure and logging 2026-04-08 09:57:40 -05:00
1e2a37e59f fix: adjust migration parameters for improved performance and consistency 2026-04-07 23:52:49 -05:00
8afdb45318 feat: implement batch processing for MSSQL extraction and transformation with range handling 2026-04-07 23:39:55 -05:00
c318428563 feat: update chunk size for MSSQL data loading and add utility functions for database operations 2026-04-07 17:57:36 -05:00
8738ef92b7 feat: implement MSSQL bulk loading functionality with transaction handling 2026-04-07 16:21:10 -05:00
270a66dbbf feat: enhance logging and performance metrics for migration processes 2026-04-07 15:53:17 -05:00
7bb67ddfcf feat: refactor data extraction and loading functions for improved context handling 2026-04-07 12:55:05 -05:00
6da321c4bb feat: implement MSSQL row transformation and loading functions 2026-04-07 11:47:27 -05:00