$convert int to binData fails

ulyssesholmes_N_A · 2025-02-14T00:15:31.932Z

I am attempting to convert an integer field to hex by using $convert passing from int to binData and then from binData to string with format hex:

The first step - converting int to binData yields onError result.

I am using mongo 8.0.4

There is some discrepancy between the release notes for mongo 8 which suggest that only string to binData and vice versa are possible but in the documentation for $convert it shows int and string conversion prototypes. Additionally, the int to binData conversion doesn’t accept the argument ‘byteOrder’.

All of this leads me to suspect that this is featuring for 8.0.5 or perhaps 8.1 which I don’t currently have access to via the routes in which I pull my versioning. Wanted to see if I could get some clarification or perhaps some suggestions for alternate approaches.

Thanks

ulyssesholmes_N_A · 2025-02-20T21:40:46.259Z

bump for any thoughts

aneroid · 2025-03-19T14:16:32.449Z

Hi,

In order for us to be able to help:

Could you paste your aggregation pipeline (or just that one stage) as text so that we can actually edit/debug it. Typing out code from an image is not ideal.
Provide some example data (as text), and optionally, in Mongo Playground (even if it doesn’t work in v6)

That stage in the image appears to have three converts:

The innermost $convert → input → $convert → input has { $toInt: ... } (which is a shorthand for $convert: { to: "int" })
Then there’s the second-level $convert → input → $convert
The outermost $convert.

ulyssesholmes_N_A · 2025-04-02T17:53:11.169Z

Here is the code. Have currently updated to 8.0.5 still no change.

The objective is to convert an int field in the document and add it as a new hex representation ‘0x…’ field in the aggregation pipeline.

Here is the original implementation I tried:

        {
            "$addFields": {
                "hex_device_id": {
                    "$convert": {
                        "input": "$int_id",
                        "to": "string",
                        "format": "hex",
                        "onError": "Invalid hex.",
                        "onNull": "Missing value.",
                    }
                }
            }
        }

This populates the added field with just the string value of the int_id field. I.e. 256 → ‘256’

The more elaborate code shared above in the image:

        {
            "$addFields": {
                "hex_id": {
                    "$convert": {
                        "input": {
                            "$convert": {
                                "input": {"$toInt": "$int_id"},
                                "to": "binData",
                                "onError": "Invalid binary.",
                                "onNull": "Missing value.",
                            }
                        },
                        "to": "string",
                        "format": "hex",
                        "onError": "Invalid hex.",
                        "onNull": "Missing value.",
                    }
                }
            }
        }

The output just yields ‘Invalid binary’ for all documents for the added field ‘hex_id’ derived of course from the field ‘int_id’ which is an integer. I have just added the $toInt conversion for good measure.

From the documentation:

mongodb.com

$convert - MongoDB Atlas - MongoDB Docs

Learn how to use the $convert expression to provide schema enforcement for your streaming data

image812×483 62.5 KB

This reads as though a person can convert from anything to binary and from binary to anything so that is what I have done in the 2nd code segment. Int → binary → string but it fails converting int to binary and ultimately just passes the onError through to the string conversion which works naturally as it is then just the error string.

From the release notes:

mongodb.com

Release Notes for MongoDB 8.0 - Database Manual v8.0 - MongoDB Docs

It talks about binData to string conversions but this doesn’t help as the int to binData conversion is failing even using the mongoDB operator ‘$toInt’ as a fail-safe.

As mentioned above the first code converts the int to a string but it is just the string representation of the integer itself un-formatted as hex or otherwise, but serves to verify that the the ‘int_id’ field is in fact stored as an integer.

ulyssesholmes_N_A · 2025-04-02T18:12:08.361Z

Interestingly if I add the field ‘byteOrder’ to the aggregation I get an error saying the field doesn’t exist for the operator $convert.

        {
            "$addFields": {
                "hex_id": {
                    "$convert": {
                        "input": {
                            "$convert": {
                                "input": {"$toInt": "$int_id"},
                                "to": "binData",
                                 "byteOrder": "little",
                                "onError": "Invalid binary.",
                                "onNull": "Missing value.",
                            }
                        },
                        "to": "string",
                        "format": "hex",
                        "byteOrder": "little",
                        "onError": "Invalid hex.",
                        "onNull": "Missing value.",
                    }
                }
            }
        }

The Error:

pymongo.errors.OperationFailure: Invalid $addFields :: caused by :: $convert found an unknown argument: byteOrder, full error: {‘ok’: 0.0, ‘errmsg’: ‘Invalid $addFields :: caused by :: $convert found an unknown argument: byteOrder’, ‘code’: 9, ‘codeName’: ‘FailedToParse’, ‘$clusterTime’: {‘clusterTime’: Timestamp(1743617331, 1), ‘signature’: {‘hash’: b’ht\xf3\xbe\xca\x9f\xfd\x07|\x87\x8dkh\x94\x06\xdbf\x9e\xe6\x00’, ‘keyId’: 7488712192850657286}}, ‘operationTime’: Timestamp(1743617331, 1)}

This is unexpected as this field is clearly documented for the $convert operator. I have upgraded to pymongo latest and still get the same error and this error also triggers regardless of which $convert the byteOrder field is added to.

aneroid · 2025-04-02T20:48:47.016Z

1. When the inner convert has a missing vale, the result would be "Missing value." Then the outer $convert tries to convert that string to a string. So even with "format": "hex", you don’t get an error since strings aren’t re-converted, regardless of format.

2. No example data yet

aneroid:

Provide some example data (as text)

3. Wrt byteOrder:

ulyssesholmes_N_A:

Interestingly if I add the field ‘byteOrder’ to the aggregation I get an error saying the field doesn’t exist for the operator $convert.

ulyssesholmes_N_A:

This is unexpected as this field is clearly documented for the $convert operator

The $convert docs you linked to are for MongoDB Atlas Stream Processing; that’s not for the Aggregations on Atlas, Enterprise, or Community editions. I ran it with the Docker image for Atlas locally and got the same error for the byteOrder field.

4. From the $convert Aggregation examples for “Convert to BinData” docs (Atlas, Enterprise, Community editions), int can’t be converted to binData:

{
  input: 123,
  to: { type: "binData", subtype: 0 }
}

Unsupported conversion from int to binData in $convert with no onError value

ulyssesholmes_N_A · 2025-04-03T17:40:48.610Z

Thank you for the replies and clarifications.

Apologies about my struggles to communicate effectively.

So, in conclusion - if I understand correctly there is no way to do what I am doing with $convert unless I am using Atlas Stream Processing.

In other words, there is no way to take an integer and convert it to its hex via conversions between string and binData. I.e. The only outcome is a hex representation of the string representation of the integer not the hex representation of the integer itself.

aneroid · 2025-04-04T10:41:50.770Z

ulyssesholmes_N_A:

if I understand correctly there is no way to do what I am doing with $convert unless I am using Atlas Stream Processing.

Correct.

ulyssesholmes_N_A:

The only outcome is a hex representation of the string representation of the integer not the hex representation of the integer itself.

No, that’s not possible either. For the final conversion of a String to BinData, that can be done but it’s base64:

So using these example documents I made up:

[
  { "int_id": 10 },
  { "int_id": "200" },
  { "int_id": null },
  { "_id": "no int_id field" }
]

Using this aggregation pipeline:

db.collection.aggregate([
  { $set: { asStr: { $toString: "$int_id" } } },
  {
    $set: {
      asHex: {
        $convert: {
          input: "$asStr",
          to: {
            type: "binData",
            subtype: 0
          },
          format: "utf8",
          onError: "asHexError"
        }
      }
    }
  }
])

Results:

[
  {
    "int_id": 10,
    "asStr": "10",
    "asHex": { "$binary": { "base64": "MTA=", "subType": "00" } }
  },
  {
    "int_id": "200",
    "asStr": "200",
    "asHex": { "$binary": { "base64": "MjAw", "subType": "00" } }
  },
  {
    "int_id": null,
    "asStr": null,
    "asHex": null
  },
  {
    "_id": "no int_id field",
    "asStr": null,
    "asHex": null
  }
]