Mongo Source Connector Pipeline Caching new Date() function value

sanket_kokne · 2023-05-08T10:50:04.862Z

Hi Team,

I am working on one problem please refer to this ticket: [Mongo Kafka Connector poll.await.time.ms Not Working - Working with Data / Connectors & Integrations - MongoDB Developer Community Forums]
(Mongo Kafka Connector poll.await.time.ms Not Working)

So far we are able to poll in mongo DB via Kafka connector , but now in our change stream pipeline we are using new Date() function, so when connector runs and poll in every 30 min it always uses connector’s start time as return from new Date() function and it always returns same time, I think it is caching pipeline and reusing it every run, Please suggest me how to execute pipeline every time so that it will take current data and time when it executes on MongoDB server, Please let me know if I can provide any more information.

for reference I am using pipeline like this:'[{$set:{last30min: {$toDate: {$subtract: [new Date(), 1800000]}},
datenow: {$toDate: new Date()}}},{$match:{$expr:{$and:[{$lte: [$last30min,$fullDocument.datefield]},
{$gte: [$datenow,$fullDocument.datefield]}]}}},
{$unset:[‘last30min’, ‘datenow’]}] `

here if this pipeline executes in every 30 min, the new Date() function returns same date and time in every run.

I tried with $$NOW but no luck.

I would really appreciate any help here.

Zaro_N_A · 2024-11-11T04:20:04.661Z

Hey, did you find a solution to the caching problem? I have tried to implement a mongo source connector and in the pipeline that I use to output the documents from Mongo to Kafka, I try to add a field when this happened, but new Date() is cached.

“pipeline”: “[{$addFields: {"fullDocument.lastUpdatedOn": {$dateToString: {date: "$fullDocument.lastUpdatedOn"}}, "fullDocument.createdOn": {$dateToString: {date: "$fullDocument.createdOn"}}, "fullDocument.sourcedOn": {$dateToString: {date: new Date()}}}}]”,