kylesoda/go-migrate-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: remove obsolete scripts and dev artifacts

Browse Source
This commit is contained in:
Kylesoda
2026-05-30 14:14:54 -05:00
parent 3aa664324d
commit ee782a9a19
3 changed files with 70 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
README.md
View File

@@ -1,60 +1,60 @@
# go-migrate # go-migrate
Migrador de datos entre SQL Server y PostgreSQL con procesamiento en paralelo. Data migrator between SQL Server and PostgreSQL with parallel ETL processing.
## Compilar ## Build
```bash ```bash
go build -o go-migrate ./cmd/go_migrate go build -o go-migrate ./cmd/go_migrate
``` ```
## Uso ## Usage
```bash ```bash
./go-migrate [opciones] [<ruta-config>] ./go-migrate [options] [<config-path>]
``` ```
### Opciones ### Options
| Flag | Descripción | | Flag | Description |
|------|-------------| |------|-------------|
| `-config <path>` | Ruta al archivo de configuración YAML. También se puede pasar como argumento posicional. Si no se indica, se busca `config.yaml`. | | `-config <path>` | Path to the YAML configuration file. Can also be passed as a positional argument. Defaults to `config.yaml` in the current directory. |
| `-validate` | Compara la cantidad de filas entre origen y destino por cada job. No migra datos. | | `-validate` | Compares the row count between source and target for each job. Does not migrate data. |
| `-dry-run` | Valida conexiones, acceso a storage (si aplica) y cuenta filas en origen sin migrar. | | `-dry-run` | Validates connections, storage access (if applicable), and counts source rows without migrating. |
### Ejemplos ### Examples
```bash ```bash
# Migrar con config.yaml por defecto # Migrate using the default config.yaml
./go-migrate ./go-migrate
# Usar un archivo de configuración específico # Use a specific configuration file
./go-migrate -config produccion.yaml ./go-migrate -config production.yaml
# Validar que origen y destino tengan la misma cantidad de filas # Validate that source and target have the same row count
./go-migrate -validate -config produccion.yaml ./go-migrate -validate -config production.yaml
# Verificar conectividad sin migrar # Check connectivity without migrating
./go-migrate -dry-run -config produccion.yaml ./go-migrate -dry-run -config production.yaml
``` ```
## Configuración ## Configuration
La herramienta lee credenciales y parámetros desde variables de entorno o un archivo `.env`. The tool reads credentials and parameters from environment variables or a `.env` file.
### Variables clave ### Key environment variables
| Variable | Descripción | | Variable | Description |
|----------|-------------| |----------|-------------|
| `SOURCE_DB_URL` | URL de conexión a la base de datos origen (o `SOURCE_DB_HOST`, `SOURCE_DB_NAME`, `SOURCE_DB_USER`, `SOURCE_DB_PWD`). | | `SOURCE_DB_URL` | Source database connection URL. Alternatively, set `SOURCE_DB_HOST`, `SOURCE_DB_NAME`, `SOURCE_DB_USER`, and `SOURCE_DB_PWD`. |
| `TARGET_DB_URL` | URL de conexión a la base de datos destino (o `TARGET_DB_HOST`, `TARGET_DB_NAME`, `TARGET_DB_USER`, `TARGET_DB_PWD`). | | `TARGET_DB_URL` | Target database connection URL. Alternatively, set `TARGET_DB_HOST`, `TARGET_DB_NAME`, `TARGET_DB_USER`, and `TARGET_DB_PWD`. |
| `LOG_LEVEL` | Nivel de log: `DEBUG`, `INFO`, `WARN`, `ERROR` (por defecto: `INFO`). | | `LOG_LEVEL` | Log level: `DEBUG`, `INFO`, `WARN`, `ERROR` (default: `INFO`). |
Para migrar datos binarios a Azure Blob, también se requieren `AZ_STORAGE_ENABLED`, `AZ_ACCOUNT_NAME`, `AZ_CONTAINER`, `AZ_ACCOUNT_KEY`. To migrate binary data to Azure Blob Storage, also set `AZ_STORAGE_ENABLED`, `AZ_ACCOUNT_NAME`, `AZ_CONTAINER`, and `AZ_ACCOUNT_KEY`.
### Archivo de migración (YAML) ### Migration config file (YAML)
Define los jobs de migración. Ejemplo mínimo: Defines the migration jobs. Minimal example:
```yaml ```yaml
source_db_type: sqlserver source_db_type: sqlserver
@@ -72,21 +72,21 @@ defaults:
max_delay_ms: 5000 max_delay_ms: 5000
jobs: jobs:
- name: migrar_usuarios - name: demo_users
enabled: true enabled: true
source: source:
schema: dbo schema: dbo
table: Usuarios table: users
primary_key: Id primary_key: id
target: target:
schema: public schema: public
table: usuarios table: users
``` ```
Consulta el archivo `config.yaml` de tu entorno para ver los jobs disponibles y sus parámetros específicos. See the `config.yaml` in this repository for the full set of supported options (`from_json`, `to_storage`, partition slicing via `range`, per-job overrides, `pre_sql`/`post_sql`, etc.).
## Modos de ejecución ## Execution modes
- **Migración** (por defecto): extrae, transforma y carga datos en paralelo. - **Migrate** (default): extracts, transforms, and loads data in parallel.
- **Validación** (`-validate`): cuenta y compara filas entre origen y destino. - **Validate** (`-validate`): counts and compares rows between source and target.
- **Dry run** (`-dry-run`): verifica conexiones y muestra la cantidad de filas en origen. - **Dry run** (`-dry-run`): validates connections and reports the source row count without migrating.

2
benchmark-results.md
View File

@@ -1,6 +1,6 @@
# Benchmark go-migrate — 2,000,000 filas # Benchmark go-migrate — 2,000,000 filas
**Tabla**: `demo.users` **Tabla**: `public.records`
**Fecha**: 2026-05-29 **Fecha**: 2026-05-29
**Entorno**: Docker local (MSSQL 2022 Developer / PostgreSQL 16 + PostGIS) **Entorno**: Docker local (MSSQL 2022 Developer / PostgreSQL 16 + PostGIS)

36
config.yaml
View File

@@ -39,15 +39,15 @@ jobs:
# source: # source:
# schema: analytics # schema: analytics
# table: events # table: events
# primary_key: ID_events # primary_key: id
# from_json: # from_json:
# - column: $node_id* # - column: payload
# field: id # field: id
# target: # target:
# schema: analytics # schema: analytics
# table: events # table: events
# - name: analytics_audit_log # - name: storage_attachments
# source: # source:
# schema: storage # schema: storage
# table: attachments # table: attachments
@@ -75,3 +75,33 @@ jobs:
# base_delay_ms: 1000 # base_delay_ms: 1000
# max_delay_ms: 15000 # max_delay_ms: 15000
# max_jitter_ms: 500 # max_jitter_ms: 500
# - name: analytics_audit_log
# source:
# schema: analytics
# table: audit_log
# primary_key: id
# target:
# schema: analytics
# table: audit_log
# pre_sql:
# - "DROP INDEX IF EXISTS audit_log_created_at_idx"
# - "DROP INDEX IF EXISTS audit_log_user_id_idx"
# post_sql:
# - "CREATE INDEX audit_log_created_at_idx ON analytics.audit_log (created_at)"
# - "CREATE INDEX audit_log_user_id_idx ON analytics.audit_log (user_id)"
# - "ANALYZE analytics.audit_log"
# # Partition slicing: only migrate the slice [2024-01-01, 2024-12-31]
# # in this re-run. is_min_inclusive=false resumes AFTER any failed boundary.
# range:
# min: 1704067200000 # 2024-01-01 UTC, as epoch millis
# max: 1735689599999 # 2024-12-31 UTC, as epoch millis
# is_min_inclusive: false
# is_max_inclusive: true
# partition_calculation_strategy: ESTIMATION # faster than EXACT on large tables
# truncate_method: DELETE # use DELETE instead of TRUNCATE
# batches_per_partition: 8
# retry:
# attempts: 5
# max_failed_partitions: 3
# max_failed_batches_load: 10