I am currently setting up PyLint to check control my code base.
The idea is to install PyLint and run it while PySpark is disabled.Then I will develop or use some open source librairies to run "PySpark checkers on my code"
Unfortunately, on the following code:
# Import sectionfrom pyspark.sql.functions import col# Listing columnscolumns = [ col("A"), col("B"),]
When I do run PyLint with the options:
--generated-members=pyspark--extension-pkg-whitelist=pyspark--ignored-modules=pyspark--ignored-classes=pyspark
I am actually filtering a lot of issues regarding pyspark "dynamic" class members, but I am still having undefined variable "col" errors flagged.
How can I properly handle it?Is there a correct setup for Python/PySpark code?
FYI: I will take care about pyspark lints in a second part, with my own checkers, so I don't need any pyspark lints flags, I only want to ensure the quality of my Python code grammar