Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2019 Aaron Longwell
Copyright (c) 2021 Aaron Longwell

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
117 changes: 71 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Schema - Embedded Database Migration Library for Go
# Schema - Database Migrations for Go

An opinionated, embeddable library for tracking and application modifications
to your Go application's database schema.
An embeddable library for applying changes to your Go application's
`database/sql` schema.

[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=for-the-badge)](https://pkg.go.dev/github.com/adlio/schema)
[![Travis Build Status](https://img.shields.io/travis/com/adlio/schema/main?style=for-the-badge)](https://travis-ci.com/github/adlio/schema)
Expand All @@ -15,24 +15,27 @@ to your Go application's database schema.
- Dependency-free (All go.mod dependencies are used only in tests)
- Unidirectional migrations (no "down" migration complexity)

## Supported Databases
# Usage Instructions

This package was extracted from a PostgreSQL project. Other databases have solid automated test coverage, but should be considered somewhat experimental in
production use cases. [Contributions](#contributions) are welcome for additional databases or feature enhancements / bug fixes.
Create a `schema.Migrator` in your bootstrap/config/database connection code,
then call its `Apply()` method with your database connection and a slice of
`*schema.Migration` structs.

- [x] PostgreSQL
- [x] SQLite (thanks [kalafut](https://github.com/kalafut)!)
- [x] MySQL / MariaDB
- [ ] SQL Server (open a Pull Request)
- [ ] CockroachDB, Redshift, Snowflake, etc (open a Pull Request)
The `.Apply()` function figures out which of the supplied Migrations have not
yet been executed in the database (based on the ID), and executes the `Script`
for each in **alphabetical order by IDe**.

## Usage Instructions (Go 1.16+)
The `[]*schema.Migration` can be created manually, but the package
has some utility functions to make it easier to parse .sql files into structs
with the filename as the `ID` and the file contents as the `Script`.

Create a `schema.Migrator` in your bootstrap/config/database connection code,
then call its `Apply()` method with your database connection and a slice of
`*schema.Migration` structs. Assuming you're using Go 1.16 or above, and you
have a directory of SQL files called `my-migrations/` next to your main.go file,
you'll run something like this:
## Using go:embed (requires Go 1.16+)

Go 1.16 added features to embed a directory of files into the binary as an
embedded filesystem (`embed.FS`).

Assuming you have a directory of SQL files called `my-migrations/` next to your
main.go file, you'll run something like this:

```go
//go:embed my-migrations
Expand All @@ -41,24 +44,51 @@ var MyMigrations embed.FS
func main() {
db, err := sql.Open(...) // Or however you get a *sql.DB

migrator := schema.NewMigrator()
migrator := schema.NewMigrator(schema.WithDialect(schema.MySQL))
err = migrator.Apply(
db,
schema.FSMigrations(MyMigrations, "my-migrations/*.sql"),
)
}
```

The `.Apply()` function figures out which of the supplied Migrations have not
yet been executed in the database (based on the ID), and executes the `Script`
for each in **alphabetical order by filename**. This procedure means its OK to call
`.Apply()` on the same Migrator with a different set of Migrations each time
(which you might do if you want to avoid the ugliness of one giant migrations.go
file with hundreds of lines of embedded SQL in it).
The `WithDialect()` option accepts: `schema.MySQL`, `schema.Postgres`, or
`schema.SQLite`. These dialects all use only `database/sql` calls, so you may
have success with other databases which are SQL-compatible with the above
dialects.

You can also provide your own custom `Dialect`. See `dialect.go` for the
definition of the `Dialect` interface, and the optional `Locker` interface. Note
that `Locker` is critical for clustered operation to ensure that only one of
many processes is attempting to run migrations simultaneously.

## Using Inline Migration Structs

If you're running in an earlier version of Go, Migration{} structs will need to
be created manually:

```go
db, err := sql.Open(...)

migrator := schema.NewMigrator() // Postgres is the default Dialect
migrator.Apply(db, []*schema.Migration{
&schema.Migration{
ID: "2019-09-24 Create Albums",
Script: `
CREATE TABLE albums (
id SERIAL PRIMARY KEY,
title CHARACTER VARYING (255) NOT NULL
)
`
},
})
```

## Constructor Options

The `NewMigrator()` function accepts option arguments to customize the dialect
and the name of the migration tracking table. By default, the tracking table
will be set to `schema.DefaultTableName` (`schema_migrations`). To change it
will be named `schema_migrations`. To change it
to `my_migrations` instead:

```go
Expand All @@ -74,27 +104,18 @@ does not exist, and then locks it to modifications while building and running
the migration plan. This means that the first-arriving process will **win** and
will perform its migrations on the database.

## Usage Instructions (pre Go 1.16)

If you're running in an earlier version of Go, Migration{} structs will need to
be created manually, such as:
## Supported Databases

```go
db, err := sql.Open(...) // Or however you get a *sql.DB
This package was extracted from a PostgreSQL project. Other databases have solid
automated test coverage, but should be considered somewhat experimental in
production use cases. [Contributions](#contributions) are welcome for
additional databases or feature enhancements / bug fixes.

migrator := schema.NewMigrator()
migrator.Apply(db, []*schema.Migration{
&schema.Migration{
ID: "2019-09-24 Create Albums",
Script: `
CREATE TABLE albums (
id SERIAL PRIMARY KEY,
title CHARACTER VARYING (255) NOT NULL
)
`
},
})
```
- [x] PostgreSQL (database/sql driver only, see [adlio/pgxschema](https://github.com/adlio/pgxschema) if you use `jack/pgx`)
- [x] SQLite (thanks [kalafut](https://github.com/kalafut)!)
- [x] MySQL / MariaDB
- [ ] SQL Server (open a Pull Request)
- [ ] CockroachDB, Redshift, Snowflake, etc (open a Pull Request)

## Package Opinions

Expand Down Expand Up @@ -122,7 +143,7 @@ particular set of opinions:

## Rules of Applying Migrations

1. **Never, ever change** the `ID` (filename) or `Script` (fille contents)
1. **Never, ever change** the `ID` (filename) or `Script` (file contents)
of a Migration which has already been executed on your database. If you've
made a mistake, you'll need to correct it in a subsequent migration.
2. Use a consistent, but descriptive format for migration `ID`s/filenames.
Expand Down Expand Up @@ -157,7 +178,11 @@ there's a good chance a different schema migration tool is more appropriate.

## Version History

### 2.0.0 (pending release/tip)
### 1.2.3 - Dec10, 2021

- BUGFIX: Restore the ability to chain NewMigrator().Apply

### 1.2.2 - Dec 9, 2021

- Add support for migrations in an embed.FS (`FSMigrations(filesystem fs.FS, glob string)`)
- Add MySQL/MariaDB support (experimental)
Expand Down
6 changes: 3 additions & 3 deletions doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
// ("migrations") as embedded functionality inside another application
// which is using a database/sql
//
// Basic usage instructions involve creating a pgxschema.Migrator via the
// pgxschema.NewMigrator() function, and then passing your pgx.Conn or
// pgxpool.Pool to its .Apply() method.
// Basic usage instructions involve creating a schema.Migrator via the
// schema.NewMigrator() function, and then passing your *sql.DB
// to its .Apply() method.
//
// See the package's README.md file for more usage instructions.
//
Expand Down
13 changes: 6 additions & 7 deletions migrator.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package schema

import (
"context"
"database/sql"
"fmt"
"time"
)
Expand All @@ -21,7 +20,7 @@ type Migrator struct {

// NewMigrator creates a new Migrator with the supplied
// options
func NewMigrator(options ...Option) Migrator {
func NewMigrator(options ...Option) *Migrator {
m := Migrator{
TableName: DefaultTableName,
Dialect: Postgres,
Expand All @@ -30,7 +29,7 @@ func NewMigrator(options ...Option) Migrator {
for _, opt := range options {
m = opt(m)
}
return m
return &m
}

// QuotedTableName returns the dialect-quoted fully-qualified name for the
Expand Down Expand Up @@ -75,7 +74,7 @@ func (m *Migrator) Apply(db DB, migrations []*Migration) (err error) {
}
defer func() { err = coalesceErrs(err, m.unlock(conn)) }()

tx, err := conn.BeginTx(m.ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
tx, err := conn.BeginTx(m.ctx, nil)
if err != nil {
return err
}
Expand Down Expand Up @@ -103,18 +102,18 @@ func (m *Migrator) lock(tx Queryer) error {
if err != nil {
return err
}
m.log("Locked %s at ", m.QuotedTableName(), time.Now().Format(time.RFC3339Nano))
m.log(fmt.Sprintf("Locked %s at %s", m.QuotedTableName(), time.Now().Format(time.RFC3339Nano)))
}
return nil
}

func (m *Migrator) unlock(tx Queryer) error {
if l, isLocker := m.Dialect.(Locker); isLocker {
err := l.Unlock(m.ctx, tx, m.TableName)
err := l.Unlock(m.ctx, tx, m.QuotedTableName())
if err != nil {
return err
}
m.log("Unlocked %s at ", m.QuotedTableName(), time.Now().Format(time.RFC3339Nano))
m.log(fmt.Sprintf("Unlocked %s at %s", m.QuotedTableName(), time.Now().Format(time.RFC3339Nano)))
}
return nil
}
Expand Down
8 changes: 7 additions & 1 deletion migrator_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -291,10 +291,16 @@ func TestRunFailure(t *testing.T) {
}
}

func TestNewMigratorApplyChain(t *testing.T) {
// This is a compilability test... it is here to confirm that
// NewMigrator()'s return value can have Apply() called on it.
_ = NewMigrator().Apply(nil, testMigrations(t, "useless-ansi"))
}

// makeTestMigrator is a utility function which produces a migrator with an
// isolated environment (isolated due to a unique name for the migration
// tracking table).
func makeTestMigrator(options ...Option) Migrator {
func makeTestMigrator(options ...Option) *Migrator {
tableName := time.Now().Format(time.RFC3339Nano)
options = append(options, WithTableName(tableName))
return NewMigrator(options...)
Expand Down