I’m going through Justin Pihony’s Pluralsight course on Spark, and he covers installing.  I’m glad that he did because there are a couple nuances involved.

First, you can download Spark without having Hadoop installed, but if you do that, you’ll want to follow Sangeet Chourey’s instructions.  If you’re running 64-bit Windows, you’ll want to get his version of winutils (which is 106 KB); if you’re running 32-bit Windows, you’ll need a different executable which is something like 43 or 46 KB.

If you do not follow Sangeet’s instructions, you’ll end up with an error described in SPARK-10528, but because it’s technically not a Spark error, they can’t do much to help out here.

I also recommend going into your Spark\conf\ directory and open log4j.properties.template.  Change INFO and WARN to ERROR and save the file as log4j.properties.  That will turn down logging to errors only, which gets rid of a lot of the mess of verbiage Spark inundates you with when you hit the console.

2 thoughts on “TIL: Installing Spark On Windows

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s