# go-migrate

Data migrator between SQL Server and PostgreSQL with parallel ETL processing.

## Build

```bash
go build -o go-migrate ./cmd/go_migrate
```

## Usage

```bash
./go-migrate [options] [<config-path>]
```

### Options

| Flag | Description |
|------|-------------|
| `-config <path>` | Path to the YAML configuration file. Can also be passed as a positional argument. Defaults to `config.yaml` in the current directory. |
| `-validate` | Compares the row count between source and target for each job. Does not migrate data. |
| `-dry-run` | Validates connections, storage access (if applicable), and counts source rows without migrating. |

### Examples

```bash
# Migrate using the default config.yaml
./go-migrate

# Use a specific configuration file
./go-migrate -config production.yaml

# Validate that source and target have the same row count
./go-migrate -validate -config production.yaml

# Check connectivity without migrating
./go-migrate -dry-run -config production.yaml
```

## Configuration

The tool reads credentials and parameters from environment variables or a `.env` file.

### Key environment variables

| Variable | Description |
|----------|-------------|
| `SOURCE_DB_URL` | Source database connection URL. Alternatively, set `SOURCE_DB_HOST`, `SOURCE_DB_NAME`, `SOURCE_DB_USER`, and `SOURCE_DB_PWD`. |
| `TARGET_DB_URL` | Target database connection URL. Alternatively, set `TARGET_DB_HOST`, `TARGET_DB_NAME`, `TARGET_DB_USER`, and `TARGET_DB_PWD`. |
| `LOG_LEVEL` | Log level: `DEBUG`, `INFO`, `WARN`, `ERROR` (default: `INFO`). |

To migrate binary data to Azure Blob Storage, also set `AZ_STORAGE_ENABLED`, `AZ_ACCOUNT_NAME`, `AZ_CONTAINER`, and `AZ_ACCOUNT_KEY`.

### Migration config file (YAML)

Defines the migration jobs. Minimal example:

```yaml
source_db_type: sqlserver
target_db_type: postgres
max_parallel_workers: 4

defaults:
  batches_per_partition: 10
  extractor_batch_size: 1000
  max_extractors: 2
  max_loaders: 2
  retry:
    attempts: 3
    base_delay_ms: 500
    max_delay_ms: 5000

jobs:
  - name: demo_users
    enabled: true
    source:
      schema: dbo
      table: users
      primary_key: id
    target:
      schema: public
      table: users
```

See the `config.yaml` in this repository for the full set of supported options (`from_json`, `to_storage`, partition slicing via `range`, per-job overrides, `pre_sql`/`post_sql`, etc.).

## Execution modes

- **Migrate** (default): extracts, transforms, and loads data in parallel.
- **Validate** (`-validate`): counts and compares rows between source and target.
- **Dry run** (`-dry-run`): validates connections and reports the source row count without migrating.