Testing pyspark with pytest

Posted on Sun 15 January 2023 in Python • Tagged with spark, pyspark, pytest, testing

Using pytest to test your pyspark code

drake meme

In my past couple of roles I have been developing on the pyspark platform and needed to to add tests to the code as part of the CI pipeline. I couldn't really find much out there that describes the process neatly, mostly a …

Continue reading