PASS Summit Follow-Up

This was quite a busy Summit...  My original plan was to write one PASS Summit-related post per day (hence the Wednesday through Friday gap last week), but too much time and not enough to do. The good news is that PASS Summit just defined the next three months for me. Polybase I already have several…

Advertisement

Restoring An Azure SQL Database

Not too long ago, I had a chance to restore a database in Azure SQL Database, as someone had accidentally run an UPDATE statement without a WHERE clause.  That happens to the best of us, and it's one of many reasons why we keep database backups around.  To correct the mistake, I had to restore a backup…

SQL Saturday Raleigh

SQL Saturday Raleigh 2017 is official.  The event will be Saturday, March 11th at William Peace University in Raleigh, North Carolina.  SQL Saturdays are free, one-day training events for database administrators, database developers, and anybody in the broader data platform space.  Attendance is free, and there is an optional $15 charge for lunch.  Our call for…

F# Zip Doesn’t Sort First

Here's a problem I ran into as I worked through my Kafka code.  Near the end of the consumer project, I used the Seq.zip3 function to zip together three separate tuples and turn them into a quadruple.  This is a great function to flatten out a set of tuples, but I kept getting results that…

Thinking About Kafka Performance

Wrapping up my series on Kafka, I want to spend a few moments thinking about how my Kafka application has performed.  When I started this journey, I wanted to put together an easy-to-understand application which performs well.  I think I've solved the first problem well enough, but still have the second problem.  What follows are some…

Resetting Kafka Topics

Let's say you're working on a program to load a Kafka topic and you mess up and want to start over.  There are two good ways of doing this.  Both of these methods involve connecting to the name node and running shell scripts in /usr/hdp/[version]/kafka/bin (for the Hortonworks Data Platform; for some other distro, I…

Console App 3: The Consumer

We've looked at two console applications thus far, one which reads data from a text file into Kafka; and one which reads from one Kafka topic, enriches the data with some information coming out of SQL Server, and puts the results into another topic.  Today's post will create a consumer which answers the questions we had at the…

Console App 2: The Enricher

Yesterday's post introduced our first console application, which pushed data onto a Kafka topic entitled Flights.  Today's console application will pull data from the Flights topic, give us a bit more information, and then push the final product onto an EnrichedFlights topic. What Is This Enrichment Of Which You Speak? Data enrichment is extremely common…

Console App 1: The Producer

Today's post will look at the first of three console applications.  This app is going to read in flight data from an external source (in my case, a flat file) and push messages out to a Kafka topic. A Sidebar:  F# If you look on the left-hand side, you'll see the three console projects that we are going…

Kafka Shell Scripts

Today's Kafka post will be a relatively simple one, where we use the built-in shell scripts to create a new topic, add some records, and view those records.  We'll wrap it up by creating the topics I need for the rest of the process. I'm going to use the Hortonworks Data Platform 2.4 sandbox for this.…