I’m having problems understanding the concept of schema updates in the context of Atlas Device Synced Realm. The documentation doesn’t make it clear on what is the approach to migrate breaking changes. The only 2 options available are stated as:
Partner collection
Client reset
Partner collection is more for production environment - and even then I’d rather not have every breaking change to have a partner collection… 100 breaking changes = 100x write per collection? Ideally, I’d like to have a schema versioned so that my app can handle the migration on a breaking change… anyway I digress.
So I figured, option 2! Client reset each time there is a breaking change, so that the client will delete the local realm and sync up to the new schema… Nope this does not work at all. I’m currently sitting on this issue where I’ve reset the device sync service many times in App Services UI, but it is still giving me a non-descriptive error message below:
Hi @lHengl!
To handle the breaking changes you have to use clientResetHandler instead of syncErrorHandler. We recommend using the “Recover or Discard Unsynced Changes Mode” strategy in such cases, since it will try to automatically recover the changes. If this is not possible then the automatic recovery fails and it tries to discard unsynced changes. In case discarding changes fails the execution will go into the onManualResetFallback, where you can prompt the users before resetting the realm file (clientResetError.resetRealm()). You can find a detailed example about onManualResetFallback implementation in “Manual Client Reset Fallback” documentation.
Feel free to comment if anything is unclear from the documentation.
Hi @Desislava_St_Stefanova I’m coming across this issue again, however this time I really want to get to the bottom of it. Here are my finding so far:
syncErrorHandler does not get invoked
clientResetHandler does not get invoked
The error occurs during the initialisation of the Realm itself. Here’s a snippet of where the error is thrown in the realm library:
realm_core.dart
RealmHandle openRealm(Configuration config) {
final configHandle = _createConfig(config);
final realmPtr = _realmLib.invokeGetPointer(() => _realmLib.realm_open(configHandle._pointer), "Error opening realm at path ${config.path}");
return RealmHandle._(realmPtr);
}
Below is the error:
[log] RealmException: Error opening realm at path /data/data/fit.tick.fitapp.dev/files/mongodb-realm/fitapp-dev-kkccq/63f2c277511cef8ea5219c0f/default.realm. Error code: 18 . Message: The following changes cannot be made in additive-only schema mode:
Property ‘userSearches.reference’ has been changed from ‘<references>’ to ‘<ReferenceRealmEO>’.
What I’ve tried so far to resolve this:
A client reset - which did nothing because the clientResetHandler is not invoked
Deleted the schema manually on the Atlas Cloud UI and did a client reset again. I was hope that the new schema will be updated. But same error of opening realm occurs.
However, I can’t seem to delete the realm for two reasons:
Realm.deleteRealm(config.path) does not work because it tells me that the realm is running!
So I tried to close the realm, by calling realm.close(). But this seem to be a chicken and egg problem, because then it tells me that the realm cannot be closed because it hasn’t been opened! What gives?
Below is my code in an attempt to delete the realm:
late final Realm _realm;
Future<Realm> openRealm(User user) async {
log().d('openRealm : opening realm for ${user.profile.name}');
finalconfig= _flexibleConfig(user);
try {
_realm = Realm(config);
} catch (e) {
_realm.close(); // This gives late initialisation error
Realm.deleteRealm(config.path); // This gives realm is already opened error
rethrow;
}
}
Hi @lHengl,
Good to hear you are moving on with the Realm.
Did you configure the clientResetHandler as follow?
final config = Configuration.flexibleSync(currentUser, schema,
clientResetHandler: RecoverOrDiscardUnsyncedChangesHandler(
// All the following callbacks are optional
onBeforeReset: (beforeResetRealm) {
// Executed before the client reset begins.
// Can be used to notify the user that a reset is going
// to happen.
},
onAfterRecovery: (beforeResetRealm, afterResetRealm) {
// Executed if and only if the automatic recovery has succeeded.
},
onAfterDiscard: (beforeResetRealm, afterResetRealm) {
// Executed if the automatic recovery has failed
// but the discard unsynced changes fallback has completed
// successfully.
},
onManualResetFallback: (clientResetError) {
// Automatic reset failed. Handle the reset manually here.
// Refer to the "Manual Client Reset Fallback" documentation
// for more information on what you can include here.
},
));
You don’t have to delete the realm. You can call clientResetError.resetRealm() inside onManualResetFallback and then to notify your users that they have to restart the app, for example.
I will try to reproduce your issue. What is the schema change that you did? Is it only renaming a property?
I will appreciate it if you can share some sample of your code using clientResetHandler.
The change is I made was changing the property to an embedded object type:
Property 'userSearches.reference' has been changed from '<references>' to '<ReferenceRealmEO>
As I’ve mentioned, the clientResetHandler is not invoked. The log does not print.
I did remember when I did my first reset, the log did show up, but my reset code only printed the logs, so nothing was done about the reset. Now, subsequent reset does not seem to invoke the resetHandler.
I have a singleton RealmApp class below (the error occurs in openRealm method):
/// A wrapper singleton instance of a realm app for convenience of access to the realm app
class RealmApp {
static Logger log([Set<String> tags = const {}]) => LogFactory.infrastructure.service<RealmApp>(tags);
///////////////////////////////////// STATIC
static final RealmApp instance = RealmApp._internal();
static Future<void> initialiseApp(AppConfiguration realmAppConfiguration) async {
log().d('initializeApp : initialising RealmApp');
instance._app = App(realmAppConfiguration);
log().d('initializeApp : done');
}
///////////////////////////////////// INSTANCE
RealmApp._internal();
/// Holds a single instance of the realm app which must be initialized before use
late final App _app;
/// Holds the current realm for used throughout the app
late Realm _realm;
Realm get realm => _realm;
Future<Realm> openRealm(User user) async {
log().d('openRealm : opening realm for ${user.profile.name}');
final config = _flexibleConfig(user);
try {
_realm = Realm(config);
} catch (e) {
log().d('openRealm : $e'); // the error is thrown here and not in the clientResetHandler
rethrow;
}
log().d('openRealm : opened realm for ${user.profile.name} at path ${_realm.config.path}');
log().d('openRealm : updating sync subscription length : ${_realm.subscriptions.length}');
// Add subscription to sync all objects in the realm
_realm.subscriptions.update((mutableSubscriptions) {
mutableSubscriptions.add(_realm.all<TaskRealm>());
mutableSubscriptions.add(_realm.all<ActualBodyCompRealm>());
mutableSubscriptions.add(_realm.all<TargetBodyCompRealm>());
mutableSubscriptions.add(_realm.all<UserPreferencesRealm>());
mutableSubscriptions.add(_realm.all<UserSearchRealm>());
});
log().d('openRealm : updated sync subscription length : ${_realm.subscriptions.length}');
log().d('openRealm : waiting for sync subscription');
await _realm.subscriptions.waitForSynchronization();
log().d('openRealm : done');
return _realm;
}
Configuration _flexibleConfig(User user) => Configuration.flexibleSync(
user,
_flexibleSyncSchema,
syncErrorHandler: (syncError) {
log().d('syncErrorHandler : ${syncError.category} $syncError');
switch (syncError.category) {
case SyncErrorCategory.client:
break;
case SyncErrorCategory.connection:
break;
case SyncErrorCategory.resolve:
break;
case SyncErrorCategory.session:
break;
case SyncErrorCategory.system:
break;
case SyncErrorCategory.unknown:
break;
}
},
clientResetHandler: RecoverOrDiscardUnsyncedChangesHandler(
onBeforeReset: (before) {
log().d('clientResetHandler : onBeforeReset');
},
onAfterRecovery: (before, after) {
log().d('clientResetHandler : onAfterRecovery');
},
onAfterDiscard: (before, after) {
log().d('clientResetHandler : onAfterDiscard');
},
onManualResetFallback: (error) {
log().d('clientResetHandler : onManualResetFallback');
},
),
);
/// Logs in a user with the given credentials.
Future<User> logIn({required Credentials credentials}) async {
log().d('logIn : logging in with ${credentials.provider.name} credentials');
final user = await _app.logIn(credentials);
log().d('logIn : logged in as ${user.profile.name}');
await openRealm(user);
log().d('logIn : opened realm for ${user.profile.name}');
return user;
}
/// Logs out the current user, if one exist
Future<void> logOut() async => currentUser?.logOut();
/// Gets the currently logged in [User]. If none exists, `null` is returned.
User? get currentUser => _app.currentUser;
/// Gets all currently logged in users.
Iterable<User> get users => _app.users;
/// Removes a [user] and their local data from the device. If the user is logged in, they will be logged out in the process.
Future<void> removeUser({required User user}) async {
return _app.removeUser(user);
}
/// Deletes a user and all its data from the device as well as the server.
Future<void> deleteUser({required User user}) async {
return _app.deleteUser(user);
}
/// Switches the [currentUser] to the one specified in [user].
Future<Realm> switchUser({required User user}) async {
_app.switchUser(user);
return realm;
}
}
Here is the log for the above code which proves that the error is caught and not handled by the flexible sync configuration:
I/flutter ( 5782): [RealmApp] : logIn : logging in with jwt credentials
I/flutter ( 5782): [RealmApp] : logIn : logged in as YdkTtwVn9XM1UxkswW6UPvQYM7B3
I/flutter ( 5782): [RealmApp] : openRealm : opening realm for YdkTtwVn9XM1UxkswW6UPvQYM7B3
I/flutter ( 5782): [RealmApp] : openRealm : RealmException: Error opening realm at path /data/data/fit.tick.fitapp.dev/files/mongodb-realm/fitapp-dev-kkccq/63f2c277511cef8ea5219c0f/default.realm. Error code: 18 . Message: The following changes cannot be made in additive-only schema mode:
I/flutter ( 5782): - Property 'userSearches.reference' has been changed from '<references>' to '<ReferenceRealmEO>'.
I/flutter ( 5782): [INFO] Realm: Connection[1]: Session[1]: client_reset_config = false, Realm exists = true, client reset = false
[log] RealmException: Error opening realm at path /data/data/fit.tick.fitapp.dev/files/mongodb-realm/fitapp-dev-kkccq/63f2c277511cef8ea5219c0f/default.realm. Error code: 18 . Message: The following changes cannot be made in additive-only schema mode:
- Property 'userSearches.reference' has been changed from '<references>' to '<ReferenceRealmEO>'.
#0 _RealmCore.throwLastError.<anonymous closure> (package:realm/src/native/realm_core.dart:119:7)
#1 using (package:ffi/src/arena.dart:124:31)
#2 _RealmCore.throwLastError (package:realm/src/native/realm_core.dart:113:5)
#3 _RealmLibraryEx.invokeGetPointer (package:realm/src/native/realm_core.dart:2784:17)
#4 _RealmCore.openRealm (package:realm/src/native/realm_core.dart:599:32)
#5 Realm._openRealm (package:realm/src/realm_class.dart:194:22)
#6 new Realm._ (package:realm/src/realm_class.dart:149:98)
#7 new Realm (package:realm/src/realm_class.dart:147:38)
#8 RealmApp.openRealm (package:fitapp/infrastructure/mongodb/realm/app/realm_app.dart:36:16)
#9 RealmApp.logIn (package:fitapp/infrastructure/mongodb/realm/app/realm_app.dart:104:11)
<asynchronous suspension>
#10 FirebaseRealmAuthService._realmLogIn (package:fitapp/infrastructure/hybrid/auth/firebase_realm_auth_service.dart:99:5)
<asynchronous suspension>
#11 FirebaseRealmAuthService._watchAuthStateChanges.<anonymous closure> (package:fitapp/infrastructure/hybrid/auth/firebase_realm_auth_service.dart:60:13)
<asynchronous suspension>
I/flutter ( 5782): [INFO] Realm: Connected to endpoint '52.64.157.195:443' (from '10.0.2.16:40040')
I/flutter ( 5782): [INFO] Realm: Verifying server SSL certificate using 155 root certificates
I/flutter ( 5782): [INFO] Realm: Connection[1]: Connected to app services with request id: "64236ef23e632940cea87942"
D/EGL_emulation( 5782): app_time_stats: avg=36.71ms min=12.31ms max=123.14ms count=27
D/EGL_emulation( 5782): app_time_stats: avg=16.68ms min=9.88ms max=21.42ms count=60
I/flutter ( 5782): [INFO] Realm: Connection[1]: Session[1]: Received: ERROR "Invalid query (IDENT, QUERY): failed to parse query: query contains table not in schema: "userSearches"" (error_code=226, try_again=false, error_action=ApplicationBug)
I/flutter ( 5782): [INFO] Realm: Connection[1]: Disconnected
The reason why clientResetError event doesn’t occur is because you have probably changed the schema on both sides, the client and the server. clientResetError is invoked when the schema on the server is different from the schema on the client app. So that the server is not involved here.
We suppose that you already have a realm file with the old schema on the device, then you change the schema in the client app and open the same old file with the new app. In such cases the only option is to delete the local realm file as you actually do.
The reason that your realm file was not deleted could be because you had opened a realm instance to the same file somewhere.
It is easily reproducible if we open the realm with the old schema and then open the realm with the new schema. The first realm, which was successfully opened, should be closed before trying to delete the file. Even though the schemas are different both realms are sharing the same file.
It could be the Realm Studio that holds the file if you have it opened.
@lHengl You may wish to launch a script to delete the realm file on the clients, or if necessary terminate sync, wait 10 minutes, and re-initiate sync, but I would advice contacting MongoDB Support directly, and have them look at what’s going on in a formal manner, because if you terminate sync, all unsynced data will be lost. But it will remove all local realm files from the apps/devices.
The biggest issue with destructive changes, is all changes you need to make you want to plan for, and alert your users whether via an in-app push notification or the like that you will be shutting down the app on X day at X time with the time zone, and then you in a controlled manner, shut down the app, terminate sync, do you destructive changes and push your updates, and then reinitiate sync.
A lot of companies do this as a part of their routine maintenance cycles and setup days/times with the least impact to their customers.
@lHengl by the way if you can not delete the file even though all the realm instances are closed, be sure to logout the users before deleting the file.
You can read about client reset handlers and find some code snippets in the documentation. If you have concrete questions, feel free to create a new post and we’ll try to help.
