Commit Graph

10 Commits

Author SHA1 Message Date
bf15979c00 refactor: consolidate error handling and retry configuration
Centralize the error channel size, retry policy and max-failed batches/partition limits in the job config.
2026-05-05 16:00:00 -05:00
f8e6c19431 refactor: switch config to cleanenv and remove unused dependencies
Migrate config loading to cleanenv and drop unused dependencies and legacy config plumbing.
2026-04-25 11:00:00 -05:00
933a7572d5 feat: integrate azure storage handling in migration pipeline
Stream transformer output to an Azure blob container when a job declares ToStorage, with per-job prefix and credential configuration.
2026-04-21 15:00:00 -05:00
f0caa55a21 feat: add source and target database type fields
Add source_db_type and target_db_type to the migration config so the same binary can drive any supported direction.
2026-04-17 13:00:00 -05:00
ecf0937a53 feat: add database wrapper interfaces for mssql and postgres
Introduce a DB dialect interface and per-dialect wrappers that expose connection, query and column type operations with pre/post SQL hooks.
2026-04-15 19:00:00 -05:00
74abf12dcf feat: add exponential backoff retry strategy
Introduce a retry policy with exponential backoff and jitter for extractor and loader errors, with configurable max attempts and delay caps.
2026-04-13 14:00:00 -05:00
7f3d2b8cc4 refactor: rename Batch to Partition across pipeline
Rename internal types and channels from Batch to Partition for consistency with the data model.
2026-04-11 18:00:00 -05:00
92f010a9a6 feat: add logging and migration runner skeleton
Wire a structured logger and a top-level migration job that drives extract, transform and load stages.
2026-04-06 13:00:00 -05:00
f7b8dd73d4 refactor: unify database url configuration
Consolidate source/target connection handling around a single URL-based configuration model.
2026-04-03 14:00:00 -05:00
69ea51a8b6 feat: add config module with env and yaml loading
Introduce a typed configuration layer loaded from environment variables and a YAML file.
2026-03-31 14:00:00 -05:00