Realm instances should have a complete view of their schema even when opened as non-dynamic

[nirinchev]

When the user passes a schema to a Realm, we go through Realm::update_schema which eventually calls into Realm::set_schema which receives both the on-disk schema and the user-provided one. We should still use the on-disk schema because the user-provided one may be a subset of the actual.

The impetus for this is a case where a mixed column contains a link to a table that doesn't exist in the original user schema (e.g. because it was synced by a later version of the app). If the user attempts to access the mixed property that contains the unknown object, they can get the Core Obj for it, but attempting to then construct an OS Object will fail because that requires looking up the schema which doesn't exist.

[nirinchev]

@tgoyne and @ironage you're probably most knowledgeable about the whole Realm and schema lifecycle in OS - do you know what's the reason for choosing to store the user-provided schema in the Realm instance as opposed to the on-disk one, assuming that the user-provided one will always be a subset of the on-disk one.

[tgoyne]

The current design is a legacy of when sync needed to be able to rearrange columns at runtime and insert columns at non-end positions in tables. Originally we stored column indices all over the place, and to support that switched to instead storing property indices and making the array of properties always exactly match the user-provided schema even if the columns were in a different order. Now that we have stable column keys that indirection is no longer needed and we could probably go back to the old way of doing things.

The code related to limiting the subset of types in object schema was some weird older cocoa behavior that probably never made sense in the first place. We allow the user to specify different objectTypes arrays on different threads, and each Realm will act as if only those types exist. It's unclear if limiting the reported schema to the listed types is actually a useful thing to be doing, and even if it is it's unclear if that needed to be pushed down to ObjectStore. set_schema_subset() also does some things for performance reasons so maybe that was the motivation?

Cocoa also just stores pointers to ObjectSchema all over the place and I doubt we have correct logic to update them all when the schema changes, but that's probably a straightforwardly fixable problem.

[fealebenpae]

It sounds to me like this is something we should do, but like @tgoyne pointed out, SDKs might depend too much on the current behavior. @cmelchior @kneth what do you think?

[cmelchior]

Java definitely depends on the user being able to only specify a subset of the actual schema found in the file.

For the Mixed object case, the user can ask for the type of the object if they don't already know. If it isn't in the schema, it returns DynamicRealmObject which is then the only way to interact with that specific type.

[nirinchev]

How can you interact with an object that is not in the Core schema? You shouldn't be able to get an Object instance from OS:

realm-core/src/realm/object-store/object.cpp

Lines 95 to 102 in ae1db72

	Object::Object(SharedRealm r, Obj const& o)
	: m_realm(std::move(r))
	, m_object_schema(&*m_realm->schema().find(ObjectStore::object_type_for_table_name(o.get_table()->get_name())))
	, m_obj(o)
	{
	REALM_ASSERT(!m_obj.get_table() ||
	(&m_realm->read_group() == _impl::TableFriend::get_parent_group(*m_obj.get_table())));
	}

.

[fealebenpae]

How does DynamicRealmObject work if Realm::schema() doesn't contain the schema for the type of the object in Mixed? Does it interrogate Core directly to find the table and inspect its columns?

[cmelchior]

Yes, DynamicRealmObject goes directly to Core, so doesn't use ObjectStore Object or Schema

[fealebenpae]

goes directly to Core

Yeah, we should change that, I'm afraid. I'd like to tighten down duplicate concern APIs and it seems like we should converge on the OS Schema API which other SDKs are using and hide away the Table API. I'll be happy to also discuss how this will impact Java offline, too, so I can better understand your requirements.

[cmelchior]

Well, DynamicRealms don't have a schema so 🤷 , but there might be value in having a DynamicObject at the ObjectStore level. But requiring the full schema to be specified would be a breaking change in Java and also fundamentally changes how we normally approached Sync that definitely allows things to exist in the Realm file that isn't exposed to users.

[fealebenpae]

I think I might be not understanding the situation fully. @nirinchev please correct me if I'm wrong: from what I can tell you want Realm::schema() to always return the full realm schema as it appears on disk, even if the user only declared a subset in their configuration object, right? So we don't change how users declare the schema, or the subset of it, but we change how the schema is introspected.

[nirinchev]

That is correct. Additionally, I'd like to point out that dynamic Realms do have schema - it's the schema on disk and OS should be able to introspect that (that's how the .NET SDK works with dynamic). We should just make the necessary changes to ensure that Java can use the OS API and not work with Core directly.

[cmelchior]

I'll need to dig a bit into how we expose schema information in the typed Realms then, because it would be really awkward if type A was a model class specifically not part of the schema and then Realm.getSchema().hasClass("A") actually returns true.

[kneth]

@fealebenpae

SDKs might depend too much on the current behavior.

The JS SDK is typically calling OS' Object::get_object_schema() and SharedRealm::schema() - and no calls to SharedRealm:: set_schema_subset().