Issue with 2.18 to 2.19 NuGet Upgrade of MongoDB C# Driver

Mark_Mann · 2023-02-03T16:29:51.886Z

I recently tried to update the MongoDB C# driver from 2.18 to 2.19 using the standard Visual Studio NuGet package updating process.

While the update itself went smoothly, I had system wide failures every where with the following exception:

“(x as ARoot) is not supported” I went through the patch notes but could not find anything directly related to this other than a small blurb about switching from LINQ2 to the LINQ3. I followed the instructions to manually set to LINQ2 but the issue still persisted. I have now rolled back to 2.18 but I would like to figure this out.

Essentially all objects in my system which are stored in mongo inherit from “AMongoThing”, which has some basic properties like the Mongo ObjectID, CreatedBy/CreatedDate, etc. The specific properties are not important.

There are a number of queries I make in the system, both get and set, where I don’t care what is actually stored in Mongo(Car, Person, Animal) because I am updating one of those root properties so my mongo call looks something like:

collection.Find( x => (x as AMongoThing).Created >= DateTime.Now.AddHours(-24))

This is obviously a super silly example but I can replicate the issue described above with this one line. That line works in 2.18 and fails in 2.19

I’m not sure if this is truly no longer support or I have some serializer or setting as part of my connection process which is causing the issue.

James_Kovacs · 2023-02-04T01:23:35.330Z

Hi, @Mark_Mann,

Thank you for reporting this issue. This was not an intentional breaking change. Please file a bug in our CSHARP JIRA project with a self-contained repro and we will investigate further.

Sincerely,
James

EMD_LAB · 2023-02-04T01:49:24.680Z

We are encountering a large number of ExpressionNotSupportedException errors in the version 2.19. Could you provide some insight into what may be causing this issue?

James_Kovacs · 2023-02-04T02:14:34.851Z

Hi, @EMD_LAB,

In 2.19.0, we changed the default LINQ provider from our older LINQ2 provider to our new implementation LINQ3. We updated our test suite to run all the LINQ2 tests using LINQ3, but there are inevitably gaps in test coverage. It would be helpful and appreciated if you provided examples of ExpressionNotSupportedExceptions with stack traces so that we can investigate further. Please file these bugs in our CSHARP JIRA project.

Note that you are still able to switch back to LINQ2:

var connectionString = "mongodb://localhost";
var clientSettings = MongoClientSettings.FromConnectionString(connectionString);
clientSettings.LinqProvider = LinqProvider.V2;
var client = new MongoClient(clientSettings);

Sincerely,
James

EMD_LAB · 2023-02-04T02:17:57.797Z

James_Kovacs:

clientSettings.LinqProvider = LinqProvider.V2;

Great news! We have successfully rolled back to version 2.18 and are looking forward to upgrading to 2.19 in our upcoming product releases. Thank you for your support.

Mark_Mann · 2023-02-06T21:55:47.716Z

Thank you everyone for the feedback and comments.

I ended up creating a simple program using both 2.18 and 2.19, and it seems the LinqProvider.V2 did fix the problem. The problem when I originally tried that was in the way I was setting the linqProvider property/value.

I will however provide my sample program if anyone is interested. The failure will/not occur as you comment out the below line:

mcSettings.LinqProvider = LinqProvider.V2;

I cannot upload the code so I will try to paste it all here, I hope it works…

using MongoDB.Bson.Serialization.Serializers;
using MongoDB.Bson.Serialization;
using MongoDB.Bson;
using MongoDB.Driver;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Linq.Expressions;
using System.Threading.Tasks;
using System.Reflection;
using static MyFirstCoreApp.ExpressionCombiner;
using System.Xml.Linq;
using Mongo219;
using MongoDB.Driver.Linq;

namespace MyFirstCoreApp
{
    public static class ExpressionCombiner
    {
        public static Expression<Func<T, bool>> And<T>(this Expression<Func<T, bool>> exp, Expression<Func<T, bool>> newExp)
        {
            var visitor = new ParameterUpdateVisitor(newExp.Parameters.First(), exp.Parameters.First());
            newExp = visitor.Visit(newExp) as Expression<Func<T, bool>>;
            var binExp = Expression.And(exp.Body, newExp.Body);
            return Expression.Lambda<Func<T, bool>>(binExp, newExp.Parameters);
        }

        public class CsharpLegacyGuidSerializationProvider : IBsonSerializationProvider
        {
            public IBsonSerializer GetSerializer(Type type)
            {
                if (type == typeof(Guid))
                    return new GuidSerializer(GuidRepresentation.Standard);

                return null;
            }
        }

        public class ParameterUpdateVisitor : System.Linq.Expressions.ExpressionVisitor
        {
            private ParameterExpression _oldParameter;
            private ParameterExpression _newParameter;

            public ParameterUpdateVisitor(ParameterExpression oldParameter, ParameterExpression newParameter)
            {
                _oldParameter = oldParameter;
                _newParameter = newParameter;
            }

            protected override Expression VisitParameter(ParameterExpression node)
            {
                if (object.ReferenceEquals(node, _oldParameter))
                    return _newParameter;

                return base.VisitParameter(node);
            }
        }
    }

    public class Program
    {
        public static void CreateClassMaps()
        {
            var types = Assembly.GetExecutingAssembly().GetTypes();

            foreach (var item in types)
            {
                try
                {
                    if (!item.IsInterface)
                    {
                        var classMap = new BsonClassMap(item);
                        classMap.AutoMap();
                        classMap.SetDiscriminator(item.FullName);

                        if (!BsonClassMap.IsClassMapRegistered(item))
                        {
                            BsonClassMap.RegisterClassMap(classMap);
                        }
                    }
                }
                catch (Exception)
                {
                    //unable to create specific class map
                }
            }
        }

        public static void Main(string[] args)
        {
            BsonSerializer.RegisterSerializationProvider(new CsharpLegacyGuidSerializationProvider());
            MongoClientSettings mcSettings = new MongoClientSettings();
            mcSettings.Server = new MongoServerAddress("localhost", 27017);
            mcSettings.LinqProvider = LinqProvider.V2;
            MongoClient client = new MongoClient(mcSettings);          
            IMongoDatabase mongoDB = client.GetDatabase("MongoTest");
            
            //clean it for fresh test each time
            mongoDB.GetCollection<AMongoThing>("Animals").DeleteMany(x => true);

            CreateClassMaps();
            AddSomeData(mongoDB);

            //just test we get all animals
            var getAllAnimals = GetThings<AAnimal>(
                mongoDB,
                filter: null);

            //should only get 1
            var getAnimalsBasedOnSomething = GetThings<AAnimal>(
                mongoDB,
                filter: x => (x as Pig).WillBeFood);
        }

        public static void AddSomeData(IMongoDatabase DB)
        {
            UpsertThing<Cat>(
                DB,
                filter: null,
                new Cat()
                {
                    ID = "63e169c103f81b89b23add99", // only setting this manually to prevent duplicates when re-running the program
                    IsDomesticated = true,
                    Age = 1,
                    Gender = "Male",
                    Name = "Fluffanutter"
                }
            );
            UpsertThing<Cat>(
                DB,
                filter: null,
                new Cat()
                {
                    ID = "63e169f4b42641ce7c5e85af", // only setting this manually to prevent duplicates when re-running the program
                    IsDomesticated = false,
                    Age = 2,
                    Gender = "Female",
                    Name = "Brown Cat"
                }
            );
            UpsertThing<Horse>(
                DB,
                filter: null,
                new Horse()
                {
                    ID = "63e169f73aad61eaad4a78aa", // only setting this manually to prevent duplicates when re-running the program
                    LivesOnFarm = true,
                    Age = 6,
                    Gender = "Male",
                    Name = "Neigh Neigh"
                }
            );
            UpsertThing<Horse>(
                DB,
                filter: null,
                new Horse()
                {
                    ID = "63e169fbbfb45bed8c515fe4", // only setting this manually to prevent duplicates when re-running the program
                    LivesOnFarm = false,
                    Age = 12,
                    Gender = "Male",
                    Name = "Mr. Ed"
                }
            );
            UpsertThing<Pig>(
                DB,
                filter: null,
                new Pig()
                {
                    ID = "63e169ffb57d09e93a93251c", // only setting this manually to prevent duplicates when re-running the program
                    WillBeFood = true,
                    Age = 3,
                    Gender = "Male",
                    Name = "Wilbur"
                }
            );
            UpsertThing<Pig>(
                DB,
                filter: null,
                new Pig()
                {
                    ID = "63e16a03db8e428dd6240b43", // only setting this manually to prevent duplicates when re-running the program
                    WillBeFood = false,
                    Age = 15,
                    Gender = "Female",
                    Name = "Sir Oinks"
                }
            );
        }

        public static T UpsertThing<T>(
            IMongoDatabase DB,
            Expression<Func<T, bool>> filter,
            T record)
        {
            var collectionName = (record as AMongoThing).StorageGrouping;
            var coll = DB.GetCollection<T>(collectionName);

            if ((record as AMongoThing).Created == null)
            {
                (record as AMongoThing).Created = DateTime.UtcNow;
            }

            (record as AMongoThing).LastModified = DateTime.UtcNow;

            if (string.IsNullOrEmpty((record as AMongoThing).ID))
            {
                coll.InsertOne(record);
                return record;
            }
            else
            {
                if (filter == null)
                {
                    filter = x => (x as AMongoThing).ID == (record as AMongoThing).ID;
                }
                else
                {
                    filter = filter.And<T>(x => (x as AMongoThing).ID == (record as AMongoThing).ID);
                }

                return coll.FindOneAndReplace(
                    filter, 
                    record, 
                    new FindOneAndReplaceOptions<T, T> { IsUpsert = true, ReturnDocument = ReturnDocument.After });
            }
        }

        public static List<T> GetThings<T>(
            IMongoDatabase DB,
            Expression<Func<T, bool>> filter)
        {
            var collectionName = "Unknown";

            if (typeof(T) == typeof(AMongoThing) || typeof(T).IsSubclassOf(typeof(AMongoThing)))
            {
                var temp = Activator.CreateInstance(typeof(T));
                collectionName = (temp as AMongoThing).StorageGrouping;
            }

            var coll = DB.GetCollection<T>(collectionName);
            var myCursor = coll.FindSync<T>(filter ?? FilterDefinition<T>.Empty);

            List<T> returnValue = new List<T>();
            while (myCursor.MoveNext())
            {
                returnValue.AddRange(myCursor.Current as List<T>);
            }

            return returnValue;
        }
    }
}

using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Bson;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Mongo219
{
    [BsonIgnoreExtraElements]
    public class AMongoThing
    {
        [BsonId]
        [BsonIgnoreIfDefault]
        [BsonRepresentation(BsonType.ObjectId)]
        public string? ID { get; set; }

        public string Name { get; set; } = "";

        public string StorageGrouping { get; set; }

        [BsonDateTimeOptions(Kind = DateTimeKind.Utc)]
        public DateTime? Created { get; set; }

        [BsonDateTimeOptions(Kind = DateTimeKind.Utc)]
        public DateTime? LastModified { get; set; }
    }

    [BsonIgnoreExtraElements]
    public class AAnimal : AMongoThing
    {
        public string Gender { get; set; }
        public int Age { get; set; }

        public AAnimal()
        {
            this.StorageGrouping = "Animals";
        }
    }

    [BsonIgnoreExtraElements]
    public class Cat : AAnimal
    {
        public bool IsDomesticated { get; set; }

        public Cat()
        {
            this.StorageGrouping = "Animals";
        }
    }

    [BsonIgnoreExtraElements]
    public class Horse : AAnimal
    {
        public bool LivesOnFarm { get; set; }

        public Horse()
        {
            this.StorageGrouping = "Animals";
        }
    }

    [BsonIgnoreExtraElements]
    public class Pig : AAnimal
    {
        public bool WillBeFood { get; set; }

        public Pig()
        {
            this.StorageGrouping = "Animals";
        }
    }
}

James_Kovacs · 2023-02-08T22:39:44.913Z

Hi, @Mark_Mann,

Thank you for your code example. I understand the problem that you’ve encountered.

You are creating a filter using x as AMongoThing, which LINQ3 attempts to convert into a server-side $convert operation. The server is not aware of C# class definitions and has no way to know how to cast an arbitrary object to AMongoThing and thus fails.

This worked by happenstance in LINQ2 because we blindly discarded cast operations that we didn’t understand. This is dangerous as the cast operation may be important to your logic.

Fortunately the fix in your code is straightforward. You can use a generic type constraint on your method, which is much safer than the cast.

public static T UpsertThing<T>(
        IMongoDatabase DB,
        Expression<Func<T, bool>> filter,
        T record) where T: AMongoThing

By annotating the method with where T: AMongoThing, you can safely eliminate all the as AMongoThing casts. Not only is the code clearer, but it is safer as the compiler prevents you from passing in objects that do not derive from AMongoThing. Previously you would have encountered a NullReferenceException at runtime if the object passed did not derive from AMongoThing.

I hope this resolves your problem.

Sincerely,
James

Mark_Mann · 2023-02-09T00:12:28.829Z

James,

Interesting and thank you for that feedback. I think the way I am using Mongo is extremely strange then, but it has been profoundly successful for us from a code management, expansion, and maintenance perspective.
We decided, for right or wrong I suppose, to put the responsibility on the developer to know what objects they have and derive from. You are correct that it would throw a null exception and we do catch that(and others) and deal with them accordingly.

In your above example it would me to specify a single “where T: ” at the end, but we actually use filters where there are multiple types assumed/used. So I cannot specify a single “AMongoThing” without sacrificing other aspects of my query

In my provided example I simplified things just to highlight the error I encountered, but we use mongoDB in some very interesting ways. I would be happy to show you what we’ve done if you were interested.

I will be using my sample project to attempt an upgrade to 2.19 as we encountered some other issues as well. Do you happen to know if/when LINQ2 will no longer be supported?

James_Kovacs · 2023-02-09T15:01:24.610Z

Hi, @Mark_Mann,

Thank you for the additional information and continued discussion. It was a design decision to be more rigorous about only removing casts (aka $convert) with LINQ3 as LINQ2 allows you to do strange things like cast a string to a bool - which will fail with LINQ-to-Objects but magically work server-side because the cast is simply stripped out of the expression.

In your use case, you use the casts to make the C# compiler happy, not to express server-side $convert expressions. While unusual, it is not as uncommon as we may have initially thought. I’m going to discuss this with the engineering team to see if and how we can support use cases such as yours.

It would be helpful to file a CSHARP ticket in JIRA along with a description of your use case, a repro, and any publicly available code so that we can review and triage it. Thank you in advance.

Removing LINQ2 support is a breaking change and will not be done until the next major version, 3.0.0. We do not have a timeline for the 3.0.0 release yet, but the soonest would be later this year or early next.

Sincerely,
James

Mark_Mann · 2023-02-09T15:47:27.228Z

James,

You are correct, the intention is not server-side conversion operations but rather to facilitate c# code LINQ type statements. I originally tried to do the (x is AThing) but of course MongoDB wouldn’t understand that and I’m not sure the discriminator is meant to work that way, my understanding was that the discriminator was more related to the de/serialization process.

I will file the bug accordingly. Thank you for responding to this with meaningful replies!

Ticket Made: https://jira.mongodb.org/browse/CSHARP-4522

If I did not provide information just let me know

system · 2023-02-14T15:47:32.267Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.