Helper function for deleting orphaned embedded objects within migration

[DominicFrei]

Current Problem

After fixing a problem in the migration of Object to EmbeddedObject it is unclear to users what happens with orphaned objects (see discussion about throwing an error vs. deleting orphaned objects).
At the moment they have to be deleted (or solved) manually to not raise an error.

Expected Results

Add a helper function to RLMMigration for deleting orphaned objects within the migration block.

[sipersso]

This helper function would be super helpful. I am pretty sure that there might be orphaned objects laying around for my app and wouldn't want it to crash in production because of this.

I am pretty sure I might have orphaned objects for some users.

[sipersso]

@DominicFrei any updates on the helper function or documentation? Is there any sample code on how I can safeguard my app from crashing on launch when I release the migration to production?

Since the objects are now child objects, I assume that I won't be able to use queries to find them? Then how can I delete them?

I noticed that @saltyskip had managed to do this, but it is not clear how you should be handling this issue. Any guidance?

[DominicFrei]

@sipersso This ticket is on our to do list but since it's a convenience function it's ranked behind more important tasks. Therefore I cannot state any ETA at the moment.

Regarding the deletion of orphaned embedded objects and taking care of multiple links:
The change to embedded is happening after the migration is done (see https://github.com/realm/realm-core/blob/master/src/realm/object-store/object_store.cpp#L834) so that you can still use the migration block of this change to work with the old non-embedded objects and take care of everything (a.k.a. making sure that every object is linked once and only once).

[sipersso]

@DominicFrei I don't mind doing the migration myself, but it is very unclear how you are supposed to be able to do this in practice. How am I supposed to be able to find the orphaned child objects?

migration.enumerateObjects(ofType: ChildObject.className()) { (oldObject, newObject) in

     let parentsAttempt1 = oldObject?["linkedParentsProperty"]  as? List<MigrationObject> //Doesn't work.
     let parentAttempt2 = oldObject?.dynamicList("linkedParentsProperty")  as? List<MigrationObject> //Doesn't work
     

It doesn't seem that the "linkedParentsProperty" can't be accessed from the child object at all? Do you have any suggestions on how to identify orphaned children?

[DominicFrei]

You could enumerate both and check their count which has to be one:

migration.enumerateObjects(ofType: ChildObjectClass.className()) { (oldChildObject, newChildObject) in
    int parentCount = 0;
    migration.enumerateObjects(ofType: ParentObjectClass.className()) { (oldChildObject, newChildObject) in
        // Look for the child in the parent and increase counter if so found.
    }
    if (parentCount == 0) {
        // Delete (maybe save necessary information first)
    } else if (parentCount > 1) {
        // Remove from all but one parent (maybe introducing copies of it).
    }
}

How you actually handle it is up to you. Making all this more convenient it what this ticket is for. :)

[sipersso]

Thanks @DominicFrei! Looking at the api, I was afraid that enumeration would be the only way. Unfortunately this will not be feasible in my case as there might be tens of thousands of grandchildren and thousands of parentes in the new structure.

The helper function won’t help at all in my case and I will have to find an alternative to making this migration, like possibly moving the data to a new property.

[sipersso]

This is a much more complicated problem than it first appears to be. In my case the structure is like the following (simplified)

public class ChildObject:Object {}
public class ParentObject:Object {
    var children = List<ChildObject>()
}
public class GrandParentObject:Object {
    var children = List<ParentObject>()
}

There can be more than 100k ChildObjects
There can be more than 10k ParentObjects
There can be more than 1k GrandParentObject

In the example, using enumerations, each child would have to loop through each of its parent objects and each parent would have to loop through its grandparents. On each level there can be orphaned objects. I have tried other migration strategies, like moving the properties to new classes, but this isn't practical and very risky as it affects the whole codebase. In addition, removing objects with linked properties is problematic (#3686).

Since this migration might be heavy I guess I should do the migration off the UI thread?

I assume that I could do the following?
1: Spin off a new thread on app launch and call perform migration on a background thread.
2: Start by enumerating ParentObjects and GrandParentObjects to detect orphaned ParentObject instances.
3: Now enumerate the ChildObjects and ParentObjects again to detect orphaned ChildObject instances.

In this case, that would mean that step 2 could be 10k* 1k = 10 million loops, and step 3 would mean 100k * 10k = 100 million loops. Is this acceptable? Is there any better way?

[sipersso]

To answer my own question. I just tested this approach using a unit test an a typical database file containing 2 MB of data. This size is fairly normal for my use case, but could be 5x larger if not more for some users. In this case, the migration took about 5 minutes to complete before it crashed because I had missed to delete an orphan.

Even thought the enumeration listed here might work for sample data, I doubt the approach will be suitable for any real world use. Is this really the only way to do it?

I have also tried to migrate the data over to new properties, but as mentioned before, this is a very huge effort. And I will have to deal with the effort of removing objects with linked properties, which seems to be problematic and there does seem to be differences in how Android and how the swift clients behave during migration.

[sipersso]

I was able to reduce the migration time from 5 minutes to 1.2 seconds using the following approach

//First find map all child identifiers to parent identifiers
var parentMap:[String:String] = [:]
migration.enumerateObjects(ofType: ParentObjectClass.className()) { (oldParentObject, newParentObject) in
        // Look for the child in the parent and increase counter if so found.
    let parentId = oldParentObject!["idProperty"] as! String

    oldParentObject!.dynamicList("chilren").forEach { oldChildObject, newChildObject in
    	let childId = oldChildObject!["idProperty"] as! String
    	parentMap[childId] = parentId
    }

}

//Now loop through the children and lookup their parent id and delete orphaned objects
migration.enumerateObjects(ofType: ChildObjectClass.className()) { (oldChildObject, newChildObject) in
    
    let childId = oldChildObject["idProperty"] as! String

    if(parentMap[childId] == null){
    	migration.deleteObject(newChildObject)
    }
}

This migration does not take the "more than one parent" into consideration. But this could be detected in the first loop.

Any pitfalls with this approach @DominicFrei?