Update the aggregation output to two different collections

Santosh_Gonsalves · 2024-02-25T20:47:46.260Z

I have a 10 stage pipeline. After stage 5 I need to “$merge” it to collection "A: and then continue with the pipeline and after stage 10 I need to “$merge” it to collection “B”. Is this possible and what is the best way to achieve it

steevej · 2024-02-25T22:47:33.363Z

By the documentation about $merge you cannot as $merge as to be the last stage of a pipeline.

But nothing stops you from running one pipeline per $merge that you have to do.

steevej · 2024-02-26T13:22:05.404Z

Does the aggregation starts on A or B or a third one?

Santosh_Gonsalves · 2024-02-27T02:49:04.743Z

Yes, that can be done but the objective is to avoid querying the whole bunch of documents and processing them with the same logic again

Santosh_Gonsalves · 2024-02-27T02:51:24.243Z

You mean the second aggregation - it starts from the beginning. I want to avoid querying the documents and processing them up to stage 5 unnecessarily because I have already done it

steevej · 2024-02-28T02:39:17.878Z

Santosh_Gonsalves:

Yes, that can be done but the objective is to avoid querying the whole bunch of documents and processing them with the same logic again

That was clear from the beginning.

Santosh_Gonsalves:

You mean the second aggregation

No. I mean the first aggregation. You did not really shared the pipeline so I will try to illustrate my question.

db.FirstCollection.aggregate( [
stage-1 ,
...,
stage-5 ,
merge-into-A
stage-6 ,
... ,
stage-10
merge-into-B
] )

Is FirstCollection the same as A or B or is it a third one that is not A nor B? If it is a third one, then an idea that might work is that you create an on-demand materialized view from the first 5 stages. The you could $merge the materialized into A, and then start an aggregation on the materialized view with step-6 that you would merge into B.

This way step-1 to step-5 are executed only once when the materialized view is created.

steevej · 2024-03-15T12:42:22.619Z

@Santosh_Gonsalves, I have spent time working on your issue. I think I have provided valuable information. I would appreciate a followup and perhaps closure with one of my post marked as the solution.

Santosh_Gonsalves · 2024-03-15T14:20:25.527Z

@steevej This is resolved. Appreciate the solution!!!

system · 2024-03-20T14:20:56.619Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.