Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 12231

I want pyspark to throw error, not fill column with null

$
0
0

Say one is reading a csv file.

It has a column, UnitPrice, that has decimals.In Pyspark, code was constructed in error with

StructField("UnitPrice", IntegerType())

The data frame is loading the file when using the schema with NULL values for column UnitPrice.

df=spark.read.format("csv").schema(ordersSchema).load

I want an error to be thrown as it is not loading this column's data correctly from the CSV file.

NOTE: There may exist row with null values for UnitPrice.


Viewing all articles
Browse latest Browse all 12231

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>