Commit Graph

Select branches

Hide Pull Requests

feat/mssql-to-mssql

main

refactor/etl-interfaces

refactor/extractor-retry-logic

refactor/retry-backoff

Mono Color

• 86258718d8 refactor: update benchmark results and configuration for improved performance in go-migrate main refactor/etl-interfaces Kylesoda 2026-05-29 14:38:25 -05:00
• 13cd02a824 refactor: add reverse configuration and enhance transformation logic for PostgreSQL to SQL Server migration Kylesoda 2026-05-29 13:23:00 -05:00
• f844004942 refactor: add comprehensive README documentation for go-migrate tool Kylesoda 2026-05-29 13:16:13 -05:00
• 4ba26092a9 refactor: implement bidirectional transformation support with PostgreSQL integration Kylesoda 2026-05-29 12:17:16 -05:00
• 537b7fbd28 refactor: remove obsolete skill registry documentation Kylesoda 2026-05-29 12:01:20 -05:00
• d534314cff feat: add Docker Compose configuration for database services and storage Kylesoda 2026-05-29 11:32:43 -05:00
• b386965bb8 refactor: enhance database configuration handling with individual parameters and URL resolution methods Kylesoda 2026-05-15 09:59:42 -05:00
• 961fa48025 refactor: implement dry run feature to validate connections and count source rows without migrating Kylesoda 2026-05-15 08:27:43 -05:00
• 837fdc7abb refactor: add validation feature to compare row counts between source and target databases Kylesoda 2026-05-12 11:01:50 -05:00
• 9c7662d5cb refactor: add logging for successful database connections in main function Kylesoda 2026-05-12 10:57:05 -05:00
• 13bbd4e82c refactor: update SOURCE_DB_URL in .env.example to include connection timeout settings Kylesoda 2026-05-12 08:33:38 -05:00
• b3e1979bfb refactor: implement expiry check to manage database connection interruptions Kylesoda 2026-05-12 08:28:41 -05:00
• b98c998820 refactor: update MockTableAnalyzer methods to improve parameter handling and capture range constraints Kylesoda 2026-05-11 11:26:27 -05:00
• fe35d2a34c refactor: update RangeConfig to use pointers for min and max; adjust partition calculation logic to handle nil values Kylesoda 2026-05-11 11:21:59 -05:00
• 0784458106 refactor: add prefix support for storage column configuration and update blob path generation Kylesoda 2026-05-11 11:04:36 -05:00
• 6f2e3e28f1 refactor: enhance error handling in ProcessPartition by tracking last row values Kylesoda 2026-05-11 10:02:58 -05:00
• 604702ef43 refactor: add unit tests for loaderAccumulator and consume functions; enhance error handling and batch processing logic Kylesoda 2026-05-11 08:36:54 -05:00
• 34a3122e5b refactor: add comprehensive tests for batch processing and error handling in Consume Kylesoda 2026-05-11 08:22:35 -05:00
• 68220e4c41 refactor: implement loaderAccumulator for batch processing; streamline error handling in Consume Kylesoda 2026-05-11 01:32:21 -05:00
• ab9a2d8694 refactor: simplify batch processing by removing partition dependency and introducing batch accumulator Kylesoda 2026-05-11 00:38:42 -05:00
• 16217f6ee2 refactor: update debug log to display partition count instead of partitions Kylesoda 2026-05-09 12:16:48 -05:00
• 723041ff2c refactor: add partition calculation strategy and implement estimation logic; enhance table analyzers for max/min column queries Kylesoda 2026-05-09 10:35:36 -05:00
• 8f8d2d11a4 refactor: enhance config handling in main; unify config parsing logic across scripts Kylesoda 2026-05-09 08:36:30 -05:00
• 0c59d06af6 refactor: adjust configuration parameters for extractors and transformers; optimize batch processing settings Kylesoda 2026-05-09 02:07:09 -05:00
• 6fa9b21b1c refactor: update totalRows and chunkSize constants for improved performance Kylesoda 2026-05-09 01:49:39 -05:00
• a8be31c18b refactor: adjust configuration parameters for extractors and loaders; enhance logging messages for clarity Kylesoda 2026-05-09 01:48:36 -05:00
• 5a8bce7701 refactor: update totalRows constant and add siteHolderAttach data generation logic; enhance row generation and loading process Kylesoda 2026-05-09 01:33:12 -05:00
• b690e580c5 refactor: enhance logging and batch processing in migration; adjust configuration parameters for improved performance Kylesoda 2026-05-09 01:16:34 -05:00
• 68d983ea57 refactor: remove unused Extractor and Loader interfaces from types.go; streamline code structure Kylesoda 2026-05-09 00:32:32 -05:00
• 1bc7b67643 refactor: replace Exec with Consume method in MssqlTransformer; enhance retry handling and streamline transformation logic Kylesoda 2026-05-09 00:30:49 -05:00
• d124da8b20 refactor: enhance error messages for max retries in partition and batch processing Kylesoda 2026-05-08 23:36:59 -05:00
• b5fd6d0534 refactor: remove unused LoaderError type and associated file; streamline error handling structure Kylesoda 2026-05-08 23:32:07 -05:00
• 85d7d69da9 refactor: streamline error handling in migration process; consolidate failed partitions and batches tracking Kylesoda 2026-05-08 23:22:53 -05:00
• d54108d5e5 refactor: move max failed batches configuration to retry section; clean up unused error handling code Kylesoda 2026-05-08 23:00:23 -05:00
• 212d3663e2 refactor: remove unused error channels and enhance job configuration; add max failed batches load parameter Kylesoda 2026-05-08 22:28:31 -05:00
• c4e233401b refactor: update GenericLoader to use ProcessBatchWithRetries; enhance error handling and retry logic Kylesoda 2026-05-08 07:50:54 -05:00
• a216a8016f refactor: update extractor methods to support FromJsonColumns; enhance data processing capabilities Kylesoda 2026-05-07 08:17:25 -05:00
• 46ddd0d6b7 refactor: enhance extraction query handling; add support for JSON column extraction and wildcard patterns Kylesoda 2026-05-07 07:40:21 -05:00
• 80babf24f2 refactor: implement Consume method in GenericLoader; enhance error handling in ProcessBatch Kylesoda 2026-05-06 18:56:50 -05:00
• f12937a1c3 refactor: standardize job error channel size; update batch size for transformed batches Kylesoda 2026-05-05 23:20:34 -05:00
• 2a5f703f3c refactor: enhance retry handling in extractor processes; unify backoff delay computation Kylesoda 2026-05-05 23:16:13 -05:00
• 7cb959a103 refactor: update migration job configuration parameters; rename queue and batch size fields for clarity Kylesoda 2026-05-05 23:04:27 -05:00
• 6414943cf3 refactor: streamline error handling and processing in GenericExtractor; implement partition processing with retries Kylesoda 2026-04-27 01:11:45 -05:00
• 00459e42e6 refactor: update column field names in ExtractionQuery struct; enhance logging in consume and process methods Kylesoda 2026-04-27 00:25:48 -05:00
• 52fe083ab7 refactor: add consume and process methods for GenericExtractor; streamline data extraction logic Kylesoda 2026-04-26 19:39:14 -05:00
• 9a00d6af04 refactor: replace specific extractor implementations with a generic extractor; remove mssql and postgres extractor files Kylesoda 2026-04-26 19:33:59 -05:00
• 33af391986 refactor: correct import path for db_dialects package in mssql and postgres files; add db_dialects package Kylesoda 2026-04-26 19:25:16 -05:00
• 2b2d740d2e feat: implement QueryFromObject method for mssql and postgres wrappers; enhance query building with limits and geometry support Kylesoda 2026-04-26 19:24:22 -05:00
• fbe17b3842 refactor: unify db dialect definition in dbdialects package Kylesoda 2026-04-26 01:33:34 -05:00
• 7bde77dcc5 refactor: rename Extractor Exec method Kylesoda 2026-04-26 01:17:40 -05:00
• 6ad25e5889 feat: update job configuration for infraestructura_site_holder__attach; enhance storage handling and retry logic Kylesoda 2026-04-22 15:23:19 -05:00
• 0ac5f01b65 feat: update configuration handling to use cleanenv; remove unused dependencies and improve error logging Kylesoda 2026-04-22 12:31:31 -05:00
• 3b1371a270 feat: integrate Azure storage handling in migration process; update transformers and job processing logic Kylesoda 2026-04-21 14:15:20 -05:00
• bb7b35619a feat: add ToStorage configuration to JobConfig for enhanced data handling Kylesoda 2026-04-21 13:43:03 -05:00
• cd2efb8692 feat: add new job configuration for red_terminal__attach with source and target definitions Kylesoda 2026-04-21 13:38:11 -05:00
• 9964ef819b feat: refactor migration job to use preSQL and postSQL from TargetTable; update config structure for job definitions Kylesoda 2026-04-21 12:30:40 -05:00
• bd51223855 feat: add truncate query handling in migration process; update job configuration to remove commented truncate SQL Kylesoda 2026-04-21 11:40:02 -05:00
• aa71eeb5c1 feat: enhance range handling in MSSQL and Postgres extractors; update partition range generator logic Kylesoda 2026-04-21 11:32:52 -05:00
• 9eb8800864 feat: add range configuration to job and update extractors for inclusive range handling Kylesoda 2026-04-21 11:29:34 -05:00
• 09bd364976 feat: implement Azure Blob Storage client and refactor configuration structure Kylesoda 2026-04-21 11:03:56 -05:00
• f2e6edd8fa feat: add caarlos0/env package for environment variable management and refactor appConfig structure Kylesoda 2026-04-21 10:41:38 -05:00
• c5a18f6d95 chore: update .env.example to set LOG_LEVEL to INFO and add Azure storage configuration variables Kylesoda 2026-04-21 10:15:01 -05:00
• 8217b13d08 feat: add azblob client implementation for file uploads Kylesoda 2026-04-21 10:01:42 -05:00
• 16c232762e feat: refactor extractor interface and implement Consume function for ETL process Kylesoda 2026-04-20 00:19:41 -05:00
• b4a846575d Update .env.example with new variables Kylesoda 2026-04-19 23:08:09 -05:00
• 5bd730f026 feat: update .gitignore to include .vscode directory Kylesoda 2026-04-19 19:37:35 -05:00
• 7bd80d4180 feat: enhance logging configuration to use dynamic log level from environment variable Kylesoda 2026-04-19 19:32:27 -05:00
• 63cf26e1ab feat: update target database type to sqlserver and clean up unused code in mssql.go feat/mssql-to-mssql Kylesoda 2026-04-17 16:29:32 -05:00
• 846a49d40c feat: implement GenericLoader for batch processing and utility functions Kylesoda 2026-04-17 15:58:08 -05:00
• ec96532d04 refactor: optimize row handling in mssql extractor and transformer refactor/extractor-retry-logic Kylesoda 2026-04-17 01:10:45 -05:00
• 46597c4ffd refactor: implement extractor retry logic and streamline extractor interface Kylesoda 2026-04-17 00:33:49 -05:00
• 15d1b96849 refactor: streamline error handling and remove redundant code in mssql extractor Kylesoda 2026-04-17 00:23:01 -05:00
• 73b65e2a3f refactor: remove extractor error channel and simplify retry logic in mssql and postgres extractors Kylesoda 2026-04-17 00:07:51 -05:00
• 1c3db39b21 update extractor interface DiegoAlessandroMotta 2026-04-16 23:49:23 -05:00
• f65e67e02f feat: add backoff package for retry logic and implement example usage in new backoff-test script refactor/retry-backoff Kylesoda 2026-04-13 22:26:53 -05:00
• 93b302db8e feat: refactor job result handling and remove unused files Kylesoda 2026-04-16 16:47:35 -05:00
• 39c0d99502 feat: extend context timeout to 1 minute for database queries in mssql.go and postgres.go Kylesoda 2026-04-16 13:19:46 -05:00
• b418ded78b feat: update SQL query to filter out specific column names in mssql.go Kylesoda 2026-04-16 13:14:39 -05:00
• 0d0511716f feat: add row count mismatch error handling in processMigrationJob and update SQL query to exclude additional graph-related columns Kylesoda 2026-04-16 12:46:55 -05:00
• 67fb0148ae feat: add Makefile for building binaries across platforms Kylesoda 2026-04-16 11:51:05 -05:00
• 098cf36e3c feat: comment out TRUNCATE statements in pre_sql for MANZANA and PUERTO jobs Kylesoda 2026-04-16 11:46:24 -05:00
• 5484716b81 feat: add source and target database type fields to MigrationConfig Kylesoda 2026-04-16 09:08:21 -05:00
• df4c3bc390 feat: refactor db handling to use db-wrapper package; enhance connection management and result handling for MSSQL and Postgres Kylesoda 2026-04-16 08:48:29 -05:00
• ea41a7c218 feat: register MSSQL and Postgres drivers in db-wrapper for improved factory pattern support Kylesoda 2026-04-15 23:09:56 -05:00
• f09284ecdc feat: enhance db-wrapper with improved MSSQL and Postgres implementations; add row result handling and dialect support Kylesoda 2026-04-15 22:55:14 -05:00
• 0384d5423f feat: add range configuration to job settings for enhanced data processing control Kylesoda 2026-04-15 20:23:21 -05:00
• 1ce3d9e153 refactor: update partition handling to use Range struct for better clarity and consistency Kylesoda 2026-04-13 22:36:07 -05:00
• ed889b740a add db-wrapper package types DiegoAlessandroMotta 2026-04-15 20:22:23 -05:00
• 803f8988b8 refactor: update extractor interfaces to return row counts instead of using pointers for rows read Kylesoda 2026-04-13 19:25:18 -05:00
• 33c9cd9c3e feat: implement database wrapper interfaces for MSSQL and Postgres; enhance migration job processing with pre and post SQL execution Kylesoda 2026-04-13 07:57:18 -05:00
• 85074da2ec feat: add max partition and chunk error limits to extractor and loader error handlers Kylesoda 2026-04-12 20:57:31 -05:00
• f126d5bbd0 feat: implement exponential backoff strategy for error handling in extractor and loader processes; enhance retry configuration options Kylesoda 2026-04-12 20:35:29 -05:00
• 5633dc98d0 fix: enhance error handling in extractor and loader processes; ensure proper job error propagation and logging Kylesoda 2026-04-12 20:11:19 -05:00
• 01780b4b02 refactor: remove unused ColumnType and inspect-columns files; update migration job to use separate table analyzers for source and target databases Kylesoda 2026-04-12 19:16:14 -05:00
• aded502ee4 feat: implement Postgres table analyzer with column type querying and metadata retrieval Kylesoda 2026-04-12 13:38:36 -05:00
• 4d3cd6e4cf feat: add MSSQL table analyzer and integrate partition range generation for improved data migration Kylesoda 2026-04-11 01:23:13 -05:00
• 7830ae862d refactor: rename batch-related variables and functions for consistency and clarity Kylesoda 2026-04-11 00:44:12 -05:00
• 955bc65ce9 refactor: rename Batch to Partition in error handling and processing functions for consistency Kylesoda 2026-04-11 00:30:54 -05:00
• 9eb9821daf refactor: rename Batch to Partition and update related types and channels for consistency Kylesoda 2026-04-11 00:09:28 -05:00
• cd0e53b1d2 feat: implement MSSQL extractor, transformer, and Postgres loader for enhanced data migration Kylesoda 2026-04-10 23:39:37 -05:00