Files

DiegoAlessandroMotta bfe335c933 Remove obsolete MSSQL and PostgreSQL scripts and related types

- Deleted `seed-manzana.go`, `site-holder-attach.go`, `types.go`, `utils.go`, `main.go` files from both `mssql-info-test` and `mssql-test` directories.
- Removed `pg-info-test/main.go` and `test-etl/main.go` files.
- Eliminated `wkb-to-ewkb/main.go` for unused WKB to EWKB conversion functionality.
- Cleaned up associated functions and types that are no longer needed.

2026-07-16 20:23:58 -05:00

2.6 KiB

Raw Blame History

go-migrate

Data migrator between SQL Server and PostgreSQL with parallel ETL processing.

Build

go build -o go-migrate ./cmd/go_migrate

Usage

./go-migrate [options] [<config-path>]

Options

Flag	Description
`-config <path>`	Path to the YAML configuration file. Can also be passed as a positional argument. Defaults to `config.yaml` in the current directory.
`-validate`	Compares the row count between source and target for each job. Does not migrate data.
`-dry-run`	Validates connections, storage access (if applicable), and counts source rows without migrating.

Examples

# Migrate using the default config.yaml
./go-migrate

# Use a specific configuration file
./go-migrate -config production.yaml

# Validate that source and target have the same row count
./go-migrate -validate -config production.yaml

# Check connectivity without migrating
./go-migrate -dry-run -config production.yaml

Configuration

The tool reads credentials and parameters from environment variables or a .env file.

Key environment variables

Variable	Description
`SOURCE_DB_URL`	Source database connection URL. Alternatively, set `SOURCE_DB_HOST`, `SOURCE_DB_NAME`, `SOURCE_DB_USER`, and `SOURCE_DB_PWD`.
`TARGET_DB_URL`	Target database connection URL. Alternatively, set `TARGET_DB_HOST`, `TARGET_DB_NAME`, `TARGET_DB_USER`, and `TARGET_DB_PWD`.
`LOG_LEVEL`	Log level: `DEBUG`, `INFO`, `WARN`, `ERROR` (default: `INFO`).

To migrate binary data to Azure Blob Storage, also set AZ_STORAGE_ENABLED, AZ_ACCOUNT_NAME, AZ_CONTAINER, and AZ_ACCOUNT_KEY.

Migration config file (YAML)

Defines the migration jobs. Minimal example:

source_db_type: sqlserver
target_db_type: postgres
max_parallel_workers: 4

defaults:
  batches_per_partition: 10
  extractor_batch_size: 1000
  max_extractors: 2
  max_loaders: 2
  retry:
    attempts: 3
    base_delay_ms: 500
    max_delay_ms: 5000

jobs:
  - name: demo_users
    enabled: true
    source:
      schema: dbo
      table: users
      primary_key: id
    target:
      schema: public
      table: users

See the config.yaml in this repository for the full set of supported options (from_json, to_storage, partition slicing via range, per-job overrides, pre_sql/post_sql, etc.).

Execution modes

Migrate (default): extracts, transforms, and loads data in parallel.
Validate (-validate): counts and compares rows between source and target.
Dry run (-dry-run): validates connections and reports the source row count without migrating.

2.6 KiB Raw Blame History