Solved the issue
i was doing a map on rdd
result = df.rdd.map(lambda x: parse(x))
and then creating a spark dataframe out of the result above with a custom schema
the schema had a datatype issue for one of the fields, basically instead of using TimestampType for a date field i was using a StringType.