Some of the tests are hanging in CI

[simonw]

https://github.com/simonw/db-to-sqlite/actions/runs/4109081649

I hit cancel workflow when it was clear three of them were not going to finish.

[simonw]

All three of the cancelled ones reported the same almost-complete status:

platform linux -- Python 3.11.1, pytest-7.2.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.11.1/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/db-to-sqlite/db-to-sqlite
collecting ... collected 19 items

tests/test_db_to_sqlite.py::test_db_to_sqlite[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [  5%]
tests/test_db_to_sqlite.py::test_db_to_sqlite[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 10%]
tests/test_db_to_sqlite.py::test_db_to_sqlite[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 15%]
tests/test_db_to_sqlite.py::test_index_fks[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 21%]
tests/test_db_to_sqlite.py::test_index_fks[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 26%]
tests/test_db_to_sqlite.py::test_index_fks[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 31%]
tests/test_db_to_sqlite.py::test_specific_tables[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 36%]
tests/test_db_to_sqlite.py::test_specific_tables[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 42%]
tests/test_db_to_sqlite.py::test_specific_tables[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 47%]
tests/test_db_to_sqlite.py::test_sql_query[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 52%]
tests/test_db_to_sqlite.py::test_sql_query[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 57%]
tests/test_db_to_sqlite.py::test_sql_query[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 63%]
tests/test_db_to_sqlite.py::test_postgres_schema PASSED                  [ 68%]
tests/test_docs.py::test_readme_contains_latest_help PASSED              [ 73%]
tests/test_fixtures.py::test_fixture_mysql PASSED                        [ 78%]
tests/test_fixtures.py::test_fixture_postgresql PASSED                   [ 84%]
tests/test_redact.py::test_redact[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 89%]
tests/test_redact.py::test_redact[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 94%]
Error: The operation was canceled.

[simonw]

Deduped the tests that pass are:

test_db_to_sqlite.py::test_db_to_sqlite
test_db_to_sqlite.py::test_index_fks
test_db_to_sqlite.py::test_postgres_schema                  
test_db_to_sqlite.py::test_specific_tables
test_db_to_sqlite.py::test_sql_query
test_docs.py::test_readme_contains_latest_help              
test_fixtures.py::test_fixture_mysql                        
test_fixtures.py::test_fixture_postgresql                   
test_redact.py::test_redact

When I run locally there's only one test that isn't in that list:

test_postgres_schema

This test here:

db-to-sqlite/tests/test_db_to_sqlite.py

Lines 108 to 122 in a2ae06e

	@pytest.mark.skipif(psycopg2 is None, reason="pip install psycopg2")
	def test_postgres_schema(tmpdir, cli_runner):
	db_path = str(tmpdir / "test_sql.db")
	connection = POSTGRESQL_TEST_DB_CONNECTION
	result = cli_runner(
	[connection, db_path, "--all", "--postgres-schema", "other_schema"]
	)
	assert result.exit_code == 0
	db = sqlite_utils.Database(db_path)
	assert db.tables[0].schema == (
	"CREATE TABLE [other_schema_categories] (\n"
	" [id] INTEGER PRIMARY KEY,\n"
	" [name] TEXT\n"
	")"
	)

[simonw]

The SQL for creating that other schema looked odd to me:

db-to-sqlite/tests/conftest.py

Lines 133 to 139 in a2ae06e

	CREATE SCHEMA other_schema
	CREATE TABLE IF NOT EXISTS other_schema_categories (
	id int not null primary key,
	name varchar(32) not null
	);
	INSERT INTO other_schema.other_schema_categories (id, name) VALUES (1, 'Other junk');

Shouldn't that have a semicolon after CREATE SCHEMA other_schema?

The tests have been passing for a couple of years so apparently that worked OK before though.

[simonw]

Weird: this workaround got 3.10 to pass but still seems to hang for 3.9 and 3.11.

https://github.com/simonw/db-to-sqlite/actions/runs/4109240953/jobs/7090852122

[simonw]

Still hanging for 3.9 and 3.11: https://github.com/simonw/db-to-sqlite/actions/runs/4109280459/jobs/7090938290

It gets this far still:

tests/test_redact.py::test_redact[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 89%]
tests/test_redact.py::test_redact[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 94%]

It looks like the last of the series of test_redact tests might be hanging - that's the second PostgreSQL one that uses postgres:// as the connection string instead of postgresql://:

db-to-sqlite/tests/shared.py

Lines 39 to 45 in e722d7e

	# Make sure it works with postgres:// connection strings too
	pytest.param(
	POSTGRESQL_TEST_DB_CONNECTION.replace("postgresql://", "postgres://"),
	marks=pytest.mark.skipif(
	psycopg2 is None, reason="pip install psycopg2"
	),
	),

[simonw]

Maybe more likely though is that the code that tears down the PostgreSQL test servers is hanging for some reason.

[simonw]

Urgh. Removing that postgres:// thing didn't fix the problem either.

https://github.com/simonw/db-to-sqlite/actions/runs/4109349112/jobs/7091083066

platform linux -- Python 3.8.16, pytest-7.2.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.8.16/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/db-to-sqlite/db-to-sqlite
collecting ... collected 14 items

tests/test_db_to_sqlite.py::test_db_to_sqlite[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [  7%]
tests/test_db_to_sqlite.py::test_db_to_sqlite[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 14%]
tests/test_db_to_sqlite.py::test_index_fks[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 21%]
tests/test_db_to_sqlite.py::test_index_fks[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 28%]
tests/test_db_to_sqlite.py::test_specific_tables[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 35%]
tests/test_db_to_sqlite.py::test_specific_tables[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 42%]
tests/test_db_to_sqlite.py::test_sql_query[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 50%]
tests/test_db_to_sqlite.py::test_sql_query[***127.0.0.1:5432/test_db_to_sqlite] PASSED [ 57%]
tests/test_db_to_sqlite.py::test_postgres_schema PASSED                  [ 64%]
tests/test_docs.py::test_readme_contains_latest_help PASSED              [ 71%]
tests/test_fixtures.py::test_fixture_mysql PASSED                        [ 78%]
tests/test_fixtures.py::test_fixture_postgresql PASSED                   [ 85%]
tests/test_redact.py::test_redact[mysql://root@127.0.0.1:3306/test_db_to_sqlite] PASSED [ 92%]
Error: The operation was canceled.

There's still something which is causing the tests to hang towards the end of the run.