Compass silently drops documents when importing from a civ

bruno_ronchetti · 2024-07-25T18:36:40.392Z

I am importing a fairly large csv file (230k documents) using Compass and end up with more than 6k missing. The logs seem fine.

I can open the same csv file in either Textmate or Apple Numbers and they both import the expected number of records.

I guess that their might be something in the data that is not acceptable to Compass, while being perfectly acceptable to Textmate and Numbers.

Any suggestions as to what I should be looking for in the data that causes this strange behavior?

Le_Roux_Bodenstein · 2024-07-26T08:55:54.744Z

The CSV parsing library used is papaparse (https://www.papaparse.com/). Maybe something in there?

Try smaller chunks of the file to see if it makes it easier to spot, see what happened to the docs immediately before or after the gaps.

bruno_ronchetti · 2024-07-31T03:29:17.512Z

Thanks.

I have bi-sected the initial file and now have a chunk of 15k initial docs which contain the 6k that are dropped. A bit more manageable, but I still do not see anything obvious in them.

I will take a look at the library before resuming the painstaking bisecting exercise.

bruno_ronchetti · 2024-07-31T05:24:46.467Z

In the end I used a workaround. Opened the cab file in Numbers, saved it as cab. The upload then worked. Apparently Numbers took care of the offending data.