93 lines
2.6 KiB
Markdown
93 lines
2.6 KiB
Markdown
# go-migrate
|
|
|
|
Data migrator between SQL Server and PostgreSQL with parallel ETL processing.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
go build -o go-migrate ./cmd/go_migrate
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./go-migrate [options] [<config-path>]
|
|
```
|
|
|
|
### Options
|
|
|
|
| Flag | Description |
|
|
|------|-------------|
|
|
| `-config <path>` | Path to the YAML configuration file. Can also be passed as a positional argument. Defaults to `config.yaml` in the current directory. |
|
|
| `-validate` | Compares the row count between source and target for each job. Does not migrate data. |
|
|
| `-dry-run` | Validates connections, storage access (if applicable), and counts source rows without migrating. |
|
|
|
|
### Examples
|
|
|
|
```bash
|
|
# Migrate using the default config.yaml
|
|
./go-migrate
|
|
|
|
# Use a specific configuration file
|
|
./go-migrate -config production.yaml
|
|
|
|
# Validate that source and target have the same row count
|
|
./go-migrate -validate -config production.yaml
|
|
|
|
# Check connectivity without migrating
|
|
./go-migrate -dry-run -config production.yaml
|
|
```
|
|
|
|
## Configuration
|
|
|
|
The tool reads credentials and parameters from environment variables or a `.env` file.
|
|
|
|
### Key environment variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `SOURCE_DB_URL` | Source database connection URL. Alternatively, set `SOURCE_DB_HOST`, `SOURCE_DB_NAME`, `SOURCE_DB_USER`, and `SOURCE_DB_PWD`. |
|
|
| `TARGET_DB_URL` | Target database connection URL. Alternatively, set `TARGET_DB_HOST`, `TARGET_DB_NAME`, `TARGET_DB_USER`, and `TARGET_DB_PWD`. |
|
|
| `LOG_LEVEL` | Log level: `DEBUG`, `INFO`, `WARN`, `ERROR` (default: `INFO`). |
|
|
|
|
To migrate binary data to Azure Blob Storage, also set `AZ_STORAGE_ENABLED`, `AZ_ACCOUNT_NAME`, `AZ_CONTAINER`, and `AZ_ACCOUNT_KEY`.
|
|
|
|
### Migration config file (YAML)
|
|
|
|
Defines the migration jobs. Minimal example:
|
|
|
|
```yaml
|
|
source_db_type: sqlserver
|
|
target_db_type: postgres
|
|
max_parallel_workers: 4
|
|
|
|
defaults:
|
|
batches_per_partition: 10
|
|
extractor_batch_size: 1000
|
|
max_extractors: 2
|
|
max_loaders: 2
|
|
retry:
|
|
attempts: 3
|
|
base_delay_ms: 500
|
|
max_delay_ms: 5000
|
|
|
|
jobs:
|
|
- name: demo_users
|
|
enabled: true
|
|
source:
|
|
schema: dbo
|
|
table: users
|
|
primary_key: id
|
|
target:
|
|
schema: public
|
|
table: users
|
|
```
|
|
|
|
See the `config.yaml` in this repository for the full set of supported options (`from_json`, `to_storage`, partition slicing via `range`, per-job overrides, `pre_sql`/`post_sql`, etc.).
|
|
|
|
## Execution modes
|
|
|
|
- **Migrate** (default): extracts, transforms, and loads data in parallel.
|
|
- **Validate** (`-validate`): counts and compares rows between source and target.
|
|
- **Dry run** (`-dry-run`): validates connections and reports the source row count without migrating.
|