go-migrate
Data migrator between SQL Server and PostgreSQL with parallel ETL processing.
Build
go build -o go-migrate ./cmd/go_migrate
Usage
./go-migrate [options] [<config-path>]
Options
| Flag | Description |
|---|---|
-config <path> |
Path to the YAML configuration file. Can also be passed as a positional argument. Defaults to config.yaml in the current directory. |
-validate |
Compares the row count between source and target for each job. Does not migrate data. |
-dry-run |
Validates connections, storage access (if applicable), and counts source rows without migrating. |
Examples
# Migrate using the default config.yaml
./go-migrate
# Use a specific configuration file
./go-migrate -config production.yaml
# Validate that source and target have the same row count
./go-migrate -validate -config production.yaml
# Check connectivity without migrating
./go-migrate -dry-run -config production.yaml
Configuration
The tool reads credentials and parameters from environment variables or a .env file.
Key environment variables
| Variable | Description |
|---|---|
SOURCE_DB_URL |
Source database connection URL. Alternatively, set SOURCE_DB_HOST, SOURCE_DB_NAME, SOURCE_DB_USER, and SOURCE_DB_PWD. |
TARGET_DB_URL |
Target database connection URL. Alternatively, set TARGET_DB_HOST, TARGET_DB_NAME, TARGET_DB_USER, and TARGET_DB_PWD. |
LOG_LEVEL |
Log level: DEBUG, INFO, WARN, ERROR (default: INFO). |
To migrate binary data to Azure Blob Storage, also set AZ_STORAGE_ENABLED, AZ_ACCOUNT_NAME, AZ_CONTAINER, and AZ_ACCOUNT_KEY.
Migration config file (YAML)
Defines the migration jobs. Minimal example:
source_db_type: sqlserver
target_db_type: postgres
max_parallel_workers: 4
defaults:
batches_per_partition: 10
extractor_batch_size: 1000
max_extractors: 2
max_loaders: 2
retry:
attempts: 3
base_delay_ms: 500
max_delay_ms: 5000
jobs:
- name: demo_users
enabled: true
source:
schema: dbo
table: users
primary_key: id
target:
schema: public
table: users
See the config.yaml in this repository for the full set of supported options (from_json, to_storage, partition slicing via range, per-job overrides, pre_sql/post_sql, etc.).
Execution modes
- Migrate (default): extracts, transforms, and loads data in parallel.
- Validate (
-validate): counts and compares rows between source and target. - Dry run (
-dry-run): validates connections and reports the source row count without migrating.
Description
Languages
Go
98.3%
Makefile
1.7%