Vote For SPARK-27006

In lieu of a PolyBase Revealed post today, I’m going to ask a favor of you. Yes, you, the person wearing the thing, right.

Vote For Duke

There is a Jira ticket for the Apache Spark project, SPARK-27006. The gist of this ticket is to bring .NET support to Spark, specifically by supporting DataFrames in C# (and hopefully F#). No support for Datasets or RDDs is included in here, but giving .NET developers DataFrame access would make it easy for us to write code which interacts with Spark SQL and a good chunk of the SparkSession object.

One of the more interesting parts of this is the support of C# UDFs and UDAFs, allowing us to write potentially complicated functions in .NET and execute them directly against data in Spark.

If you like the idea of being able to write C# (or hopefully F#) code against Spark clusters, I have a request to make of you: vote for this Jira ticket. If you do not already have a Jira account for Apache projects, you will need to create one, but that’s a couple of minutes’ work. Then click the “Vote for this issue” link on the right-hand side. Having a large number of votes for this issue will show the Spark community that there’s value in bringing .NET support in as a first-class citizen. It also tells Microsoft that there’s a community interested in the topic, so they should dedicate time and get it right.