TIL: Installing Spark On Windows

I’m going through Justin Pihony’s Pluralsight course on Spark, and he covers installing.  I’m glad that he did because there are a couple nuances involved.

First, you can download Spark without having Hadoop installed, but if you do that, you’ll want to follow Sangeet Chourey’s instructions.  If you’re running 64-bit Windows, you’ll want to get his version of winutils (which is 106 KB); if you’re running 32-bit Windows, you’ll need a different executable which is something like 43 or 46 KB.

If you do not follow Sangeet’s instructions, you’ll end up with an error described in SPARK-10528, but because it’s technically not a Spark error, they can’t do much to help out here.

I also recommend going into your Spark\conf\ directory and open log4j.properties.template.  Change INFO and WARN to ERROR and save the file as log4j.properties.  That will turn down logging to errors only, which gets rid of a lot of the mess of verbiage Spark inundates you with when you hit the console.