Commit Graph

35 Commits

Author SHA1 Message Date
212d3663e2 refactor: remove unused error channels and enhance job configuration; add max failed batches load parameter 2026-05-08 22:28:31 -05:00
c4e233401b refactor: update GenericLoader to use ProcessBatchWithRetries; enhance error handling and retry logic 2026-05-08 07:50:54 -05:00
a216a8016f refactor: update extractor methods to support FromJsonColumns; enhance data processing capabilities 2026-05-07 08:17:25 -05:00
80babf24f2 refactor: implement Consume method in GenericLoader; enhance error handling in ProcessBatch 2026-05-06 18:56:50 -05:00
2a5f703f3c refactor: enhance retry handling in extractor processes; unify backoff delay computation 2026-05-05 23:16:13 -05:00
6414943cf3 refactor: streamline error handling and processing in GenericExtractor; implement partition processing with retries 2026-04-27 01:11:45 -05:00
00459e42e6 refactor: update column field names in ExtractionQuery struct; enhance logging in consume and process methods 2026-04-27 00:25:48 -05:00
52fe083ab7 refactor: add consume and process methods for GenericExtractor; streamline data extraction logic 2026-04-26 19:39:14 -05:00
9a00d6af04 refactor: replace specific extractor implementations with a generic extractor; remove mssql and postgres extractor files 2026-04-26 19:33:59 -05:00
7bde77dcc5 refactor: rename Extractor Exec method 2026-04-26 01:17:40 -05:00
6ad25e5889 feat: update job configuration for infraestructura_site_holder__attach; enhance storage handling and retry logic 2026-04-22 15:23:19 -05:00
3b1371a270 feat: integrate Azure storage handling in migration process; update transformers and job processing logic 2026-04-21 14:15:20 -05:00
aa71eeb5c1 feat: enhance range handling in MSSQL and Postgres extractors; update partition range generator logic 2026-04-21 11:32:52 -05:00
9eb8800864 feat: add range configuration to job and update extractors for inclusive range handling 2026-04-21 11:29:34 -05:00
16c232762e feat: refactor extractor interface and implement Consume function for ETL process 2026-04-20 00:19:41 -05:00
846a49d40c feat: implement GenericLoader for batch processing and utility functions 2026-04-17 15:58:08 -05:00
39c0d99502 feat: extend context timeout to 1 minute for database queries in mssql.go and postgres.go 2026-04-16 13:19:46 -05:00
b418ded78b feat: update SQL query to filter out specific column names in mssql.go 2026-04-16 13:14:39 -05:00
0d0511716f feat: add row count mismatch error handling in processMigrationJob and update SQL query to exclude additional graph-related columns 2026-04-16 12:46:55 -05:00
df4c3bc390 feat: refactor db handling to use db-wrapper package; enhance connection management and result handling for MSSQL and Postgres 2026-04-16 08:48:29 -05:00
1ce3d9e153 refactor: update partition handling to use Range struct for better clarity and consistency 2026-04-15 20:23:45 -05:00
803f8988b8 refactor: update extractor interfaces to return row counts instead of using pointers for rows read 2026-04-13 19:25:18 -05:00
f126d5bbd0 feat: implement exponential backoff strategy for error handling in extractor and loader processes; enhance retry configuration options 2026-04-12 20:35:29 -05:00
5633dc98d0 fix: enhance error handling in extractor and loader processes; ensure proper job error propagation and logging 2026-04-12 20:11:19 -05:00
01780b4b02 refactor: remove unused ColumnType and inspect-columns files; update migration job to use separate table analyzers for source and target databases 2026-04-12 19:16:14 -05:00
aded502ee4 feat: implement Postgres table analyzer with column type querying and metadata retrieval 2026-04-12 13:38:36 -05:00
4d3cd6e4cf feat: add MSSQL table analyzer and integrate partition range generation for improved data migration 2026-04-11 01:23:13 -05:00
7830ae862d refactor: rename batch-related variables and functions for consistency and clarity 2026-04-11 00:44:12 -05:00
955bc65ce9 refactor: rename Batch to Partition in error handling and processing functions for consistency 2026-04-11 00:32:50 -05:00
9eb9821daf refactor: rename Batch to Partition and update related types and channels for consistency 2026-04-11 00:09:28 -05:00
cd0e53b1d2 feat: implement MSSQL extractor, transformer, and Postgres loader for enhanced data migration 2026-04-10 23:39:37 -05:00
a5b5a04feb feat: update extractor and transformer constructors to return Extractor interface 2026-04-10 20:42:16 -05:00
c1bae79f98 feat: implement Postgres loader and refactor migration job processing 2026-04-10 20:40:01 -05:00
053e6bd673 feat: add MSSQL extractor and transformer implementations for improved data migration 2026-04-10 19:59:44 -05:00
c2ea84bfcf feat: implement extractor error handling and batch processing for MSSQL and Postgres 2026-04-10 19:06:41 -05:00