Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 12111

How to properly setup PyLint to check pyspark code

$
0
0

I am currently setting up PyLint to check control my code base.

The idea is to install PyLint and run it while PySpark is disabled.Then I will develop or use some open source librairies to run "PySpark checkers on my code"

Unfortunately, on the following code:

# Import sectionfrom pyspark.sql.functions import col# Listing columnscolumns = [    col("A"),    col("B"),]

When I do run PyLint with the options:

--generated-members=pyspark--extension-pkg-whitelist=pyspark--ignored-modules=pyspark--ignored-classes=pyspark

I am actually filtering a lot of issues regarding pyspark "dynamic" class members, but I am still having undefined variable "col" errors flagged.

How can I properly handle it?Is there a correct setup for Python/PySpark code?

FYI: I will take care about pyspark lints in a second part, with my own checkers, so I don't need any pyspark lints flags, I only want to ensure the quality of my Python code grammar


Viewing all articles
Browse latest Browse all 12111

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>