When using the $lookup builder in the Java API's com.mongodb.client.model.Aggregates, how do I reference values declared in "let" field when making expressions in the "pipeline" field?

odrilon · 2024-06-25T10:25:37.569Z

The builder com.mongodb.client.model.Aggregates::lookup is supposed to build the following statement :

$lookup: {
  from: "...",
  let: { <variables> },
  pipeline: [ <an aggregation pipeline> ],
  as: "..."
}

However, when building a $match stage in the pipeline field, I cannot find the correct Java API classes or functions to reference a variable declared in the from field.

I cannot reference the variable with $$notation when using the com.mongodb.client.model.Filters DSL. I also can’t find a way to create an MqlValue that references the variable.

I’ve already verified that:

The document field that the variable will be compared against has the correct value.
The variable is being set and has the correct value.

The sample code below shows what I’ve tried, including expressions that make the above verifications.

// exampleField has a value of "exampleValue"
final List<Variable<MqlValue>> letStatement = List.of(
  new Variable<>("exampleVariable", MqlValues.current().getString("exampleField"))
);

final List<Bson> pipelineStatement = List.of(
    Aggregates.match(Filters.and(
      // does not work:
//          Filters.eq("fieldToCompareTo", "$$exampleVariable"),
//          Filters.expr(MqlValues.current().getString("fieldToCompareTo").eq( <I cannot find a way to instantiate an MqlValue that references the exampleVariable> ))

      // the record is successfully retrieved when these conditions are present
      // this verifies that the field exists in the document and has the correct value
      Filters.eq("fieldToCompareTo", "exampleValue"),
      Filters.expr(MqlValues.current().getString("fieldToCompareTo").eq(MqlValues.of("exampleValue")))
    )),

    Aggregates.project(Projections.fields(
      // "exampleOutput" turns into "exampleValue"
      // this verifies that the variable is being set to the correct value
      Projections.computed("exampleOutput", "$$exampleVariable")
    ))
);

Aggregates.lookup("Example Collection", letStatement, pipelineStatement, "output");

Does the MongoDB Java API support references to variables or do I need to manually craft Bson statements?

steevej · 2024-06-25T13:36:14.868Z

What do you get as result?

You do not show how you use the result of

odrilon:

Aggregates.lookup("Example Collection", letStatement, pipelineStatement, "output");

The above does not aggregate the collection. It just create a document that can be used. But since you do not return it to a caller function or assign it to a variable, so how do you use it. The code that uses Aggregates.lookup() is missing. One way to know if Aggregates.lookup() creates the equivalent statement is to print it.

odrilon · 2024-06-25T17:26:14.112Z

My code example just returns the $lookup stage that I’m adding to a larger aggregate query. That larger query is irrelevant to my problem. I’m specifically asking about how to add a $match condition in this kind of $lookup stage.

In the actual output, I see something along the lines of

[
  { output: [ { <a record from Example Collection> } ] },
  { output: [ { <a record from Example Collection> } ] },
  { output: [ { <a record from Example Collection> } ] }...
]

Where output is the as value of the $lookup stage.

Basically, the aggregation pipeline works when I hardcode the expected value using Filters::eq or Filters.expr (and passing an MqlValue). My problem is outside of this toy example, I need to take the actual value from a variable ($$exampleVariable) instead of of hardcoding the value.

steevej · 2024-06-25T18:40:47.765Z

What do you mean by variable?

odrilon:

I need to take the actual value from a variable

A java variable? For a java variable there is nothing to do. Rather than the hard coded value exampleValue.

odrilon:

Filters.eq("fieldToCompareTo", "exampleValue"),

Simply do

Filters.eq("fieldToCompareTo",MqlValues.current().getString("exampleField"))

odrilon · 2024-06-26T00:40:47.741Z

A java variable? For a java variable there is nothing to do. Rather than the hard coded value exampleValue.

No, of course not. I meant a variable declared inside of the let field in the $lookup stage. In the example, it’s named “exampleVariable” and it’s reading “exampleField” from the parent aggregation pipeline containing the $lookup stage:

final List<Variable<MqlValue>> letStatement = List.of(
  new Variable<>("exampleVariable", MqlValues.current().getString("exampleField"))
);

So: how do I use the Java API to use $$exampleVariable within an expression?

The non-Java MongoDB documentation (https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup/#std-label-lookup-join-let) just says, “To reference variables in pipeline stages, use the “$$” syntax.” The problem is that it’s unclear which Java API method or object is supposed to accept the “$$” syntax. Filters::eq does not accept it, and neither does MqlString::eq (or any other ::eq value from MqlValue).

steevej · 2024-06-26T12:01:56.188Z

The link you to the documentation you provider also mention:

A $match stage requires the use of an $expr operator to access the variables. The $expr operator allows the use of aggregation expressions inside of the $match syntax.

As I am not very familiar with the builders I do no know what java class builds an $expr.

I am not familiar with the builders as I use the Document API directly so that my code looks a lot more like my JS code. Sometimes, I simply use Document.parse() on a JSON string. This way I do not need to know 2 API. I stay as close as possible to JSON because eventually it is what it is sent to the server.

steevej · 2024-07-06T13:10:01.451Z

@odrilon, any update on this? Is your issue resolved. Please share what worked so others on the forum knows what works or not.

odrilon · 2024-07-15T09:31:50.697Z

I examined the code of MqlExpression (the main implementation of MqlValue) and there really is a feature gap in the MongoDB Java API/library where it doesn’t support $variable (unless I misunderstood the code).

You can work around this by doing either of the following:

(1) Have a previous stage add the $variable as a field. I think this this is a peformance penalty if you’re are using the $variable in an early $match that filters out many records, because now MongoDB will need to process all the records before filtering them out.

(2) Manually create the expression by making a Document, as @steevej usually does. Thankfully, MqlValue instances get converted into Bson when you place them into Document instances, so you can do something like below if you’re using the MqlValue API in most of your code.

final MqlValue field = getThisFromSomewhere();

final Document document = new Document(Map.of(
  "$eq", List.of(field, "$$variable")
));