go-migrate

Data migrator between SQL Server and PostgreSQL with parallel ETL processing.

Build

go build -o go-migrate ./cmd/go_migrate

Usage

./go-migrate [options] [<config-path>]

Options

Flag Description
-config <path> Path to the YAML configuration file. Can also be passed as a positional argument. Defaults to config.yaml in the current directory.
-validate Compares the row count between source and target for each job. Does not migrate data.
-dry-run Validates connections, storage access (if applicable), and counts source rows without migrating.

Examples

# Migrate using the default config.yaml
./go-migrate

# Use a specific configuration file
./go-migrate -config production.yaml

# Validate that source and target have the same row count
./go-migrate -validate -config production.yaml

# Check connectivity without migrating
./go-migrate -dry-run -config production.yaml

Configuration

The tool reads credentials and parameters from environment variables or a .env file.

Key environment variables

Variable Description
SOURCE_DB_URL Source database connection URL. Alternatively, set SOURCE_DB_HOST, SOURCE_DB_NAME, SOURCE_DB_USER, and SOURCE_DB_PWD.
TARGET_DB_URL Target database connection URL. Alternatively, set TARGET_DB_HOST, TARGET_DB_NAME, TARGET_DB_USER, and TARGET_DB_PWD.
LOG_LEVEL Log level: DEBUG, INFO, WARN, ERROR (default: INFO).

To migrate binary data to Azure Blob Storage, also set AZ_STORAGE_ENABLED, AZ_ACCOUNT_NAME, AZ_CONTAINER, and AZ_ACCOUNT_KEY.

Migration config file (YAML)

Defines the migration jobs. Minimal example:

source_db_type: sqlserver
target_db_type: postgres
max_parallel_workers: 4

defaults:
  batches_per_partition: 10
  extractor_batch_size: 1000
  max_extractors: 2
  max_loaders: 2
  retry:
    attempts: 3
    base_delay_ms: 500
    max_delay_ms: 5000

jobs:
  - name: demo_users
    enabled: true
    source:
      schema: dbo
      table: users
      primary_key: id
    target:
      schema: public
      table: users

See the config.yaml in this repository for the full set of supported options (from_json, to_storage, partition slicing via range, per-job overrides, pre_sql/post_sql, etc.).

Execution modes

  • Migrate (default): extracts, transforms, and loads data in parallel.
  • Validate (-validate): counts and compares rows between source and target.
  • Dry run (-dry-run): validates connections and reports the source row count without migrating.
Description
No description provided
Readme 168 KiB
Languages
Go 98.3%
Makefile 1.7%