Last night, I went to a local .NET User Group meetup and got my first taste of Docker.
In my case, I ended up running on Elementary OS rather than Windows, but the experience was a good one, going through the tutorials. In the end, I installed Solr and was able to load a document for indexing. Installing nginx was also easy.
If you are interested in installing Docker in a Windows environment, start with Mano Marks’s article on the topic. You can also look at the Docker Tools for Visual Studio.
I had the privilege to sit down with a couple Hortonworks solution engineers and discuss a potential Hadoop solution in our environment. During that time, I learned an interesting strategy for handling data in Kafka.
Our environment uses MSMQ for queueing. What we do is add items to a queue, and then consumers pop items off of the queue and consume them. The advantage to this is that you can easily see how many items are currently on the queue and multiple consumer threads can interact, playing nice with the queue. Queue items last for a certain amount of time—in our case, we typically expire them after one hour.
With Kafka, however, queues are handled a bit differently. The queue manager does not know or care about which consumers read what data when (making it impossible for a consumer to tell how many items are left on the queue at a certain point in time), and the consumers have no ability to pop items off of the queue. Instead, queue items fall off after they expire.
Our particular scenario has some web servers which need to handle incoming clicks. Ideally, we want to handle that click and dump it immediately onto a queue, freeing up the web server thread to handle the next request. Once data gets into the queue, we want it to live until our internal systems have a chance to process that data—if we have a catastrophe and items fall off of this queue, we lose revenue.
The strategy in this case is to take advantage of multiple queues and multiple stages. I had thought of “a” queue, into which the web server puts click data and out of which the next step pulls clicks for processing. Instead of that, a better strategy (given what we do and our requirements) is to immediately put the data into a queue and then have consumers pull from the queue, perform some internal “enrichment” processes, and finally put the enriched data back onto a new queue. That new queue will collect data and an occasional batch job pulls it off to write to HDFS. This way, you don’t take the hit of streaming rows into HDFS. As far as maintaining data goes, we’d need to set our TTL to last long enough that we can deal with an internal processing engine catastrophe but not so long that we run out of disk space holding messages. That’s a fine line trade-off we’ll need to figure out as we go along.
Summing things up, Kafka is quite a different product than MSMQ, and a workable architecture is going to look different depending upon which queue product you use.
Last night, Jamie Dixon (whose book you should buy) talked about his experience at Build this year. His main takeaway is that Microsoft is pushing their Bot Framework pretty hard. Jamie showed how to create a stock lookup bot and deploy it to Slack using F# code and an Azure account.
About a month ago, I went to F8 and my main takeaway from that conference is that Facebook is pushing bots on their Messenger platform pretty hard.
Based on my (extremely) limited knowledge of both, it seems that the Facebook bot platform is a bit easier to deal with, as there’s a user interface for writing messages and tokenizing, but right now, Microsoft’s platform is a bit more customizable and developer-friendly. What’s particularly interesting about both of these is that the analytics and intelligence engines are both closed-source—neither company will let you see the wizard behind the curtain.
I’m giving a presentation on monitoring this Monday. As part of that, I want to firm up some thoughts on the differences between auditing, monitoring, and alerting. All three of these are vital for an organization, but they serve entirely different functions and have different requirements. I’ll hit a bunch of bullet points for each.
Auditing is all about understanding a process and what went on. Ideally, you would audit every business-relevant action, in order, and be able to “replay” that business action. Let’s say we have a process which grabs a flat file from an FTP server somewhere, dumps data into a staging table, and then performs ETL and puts rows into transactional tables. Our auditing process should be able to show what happened, when. We want to log each activity (grab flat file, insert rows into staging table, process ETL) down to its most granular level. If we make an external API call for each row as part of the ETL process, we should log the exact call. If we throw away a row, we should note that. If we modify attributes, we should note that.
Of course, this is a huge amount of data and depending upon processing requirements and available storage space, you probably have to live with something much less thorough. So here are some thoughts:
Monitoring is all about seeing what’s going on in your system “right now.“ You want nice visualizations which give you relevant information about currently-running processes, and I put “right now” in quotation marks because you can be monitoring a process which only updates once every X minutes.
There are a couple of important things to consider with monitoring:
Alerting is all about sending messages and getting your attention. This is because an alert is telling you something that you (as a trained operator) need to act upon. I think alerting is the hardest thing on the list to get right because there are several important considerations here:
Auditing, monitoring, and alerting solve three different sets of problems. They also have three different sets of requirements for what kind of data to use, how frequently to refresh this data, and how people interact with them. It’s important to keep these clearly delineated for that reason.
During this, I’m also working on some toy monitoring stuff, so I hope that’ll be tomorrow’s TIL.
I recently got through some difficulties installing Jupyter and incorporating R support, so I wanted to write up a quick installation post for a Linux installation.
First, install Jupyter through anaconda. Notes:
When running Jupyter, I started by following this guide. Notes:
As for installing R support, Andrie de Vries has a nice post on the topic. Notes:
One step closer to my dream goal: getting Management Studio on Linux.
I have a reasonably powerful laptop that I use for demos. I also use VMware Player because it’s free, because I have experience with VMware, and because I want virtual machines for demos. That way, I can keep my demonstration environment fairly well controlled. I don’t upgrade my demo machines that often, and when I do, I’m reasonably careful about it. This also allows me to repeat my demonstrations without too much bother, and it also means that my futzing about with other work doesn’t affect my ability to demo presentations.
Just about two months ago, I had disaster befall me at SQL Saturday Pittsburgh, when my laptop and the provided projector absolutely would not play nice. I had a tablet with me, but there’s no way my little tablet would be able to power SQL Server, even if I had it installed. What that tablet can do, though, is run a VM. I also have an Azure subscription, so I decided that one of my many safety measures would be to migrate my demo VM up to Azure so that I could spin up a VM in the event of future disaster.
There are two approaches that will work: the Microsoft Virtual Machine Converter and manually uploading VMs. I’ll walk through each.
There are a few good resources on how to get MVMC working. I started with Carsten Lemm’s blog post on the topic because I wanted to migrate a VMware VM into Azure and I could afford to spend 30 minutes on the task. Sadly, my experience took well over 30 minutes…
After downloading and installing the MVMC executable, I followed Carsten’s instructions and made sure that my VM obtains DNS and IP addresses automatically rather than hard-coding addresses. I also turned on Remote Desktop. At that point, Carsten’s blog post is a bit out of date, as he references an executable which no longer exists. But that’s okay, because the MVMC 3.0 executable now has a nice wizard. The route is pretty simple: on the first screen, we want to select the “Virtual machine conversion” radio button. Then, select the “Migrate to Microsoft Azure” option and hit Next again. The next tab asks you for a Subscription ID and Certificate Thumbprint.
To get the Subscription ID, you can go to the old Azure portal and click the Settings tab on the left-hand navigation bar. That will give you a GUID which represents your subscription ID. Copy and paste that into the Subscription ID section and you will find a bug: this page has an off-by-one error. If you paste in your subscription ID, you’ll see that the last character is missing and the app does not allow you to type in that last character. Even if you delete characters, you’re still stuck. The only way I was able to get past this is to type in my GUID manually.
As for the Certificate Thumbprint, the same applies: I needed to type it in manually or else the app would cut off part of the thumbprint. Don’t type in any of the spaces and you’ll be fine. If you don’t know how to create a certificate, check out the Additional Resources section.
From here, I’m going to cut out because the next screen ended up being my downfall: they want me to put in my VCenter, ESX, or ESXi server name. I don’t have one of those; I’m just using VMware player and want to convert my VMDK files to VHDs so Azure can use them. I realized at this point that MVMC was not the trick. The only reason I’m including this section is to point out the bug above, just in case anybody gets errors like The certificate with thumbprint [thumbprint] was not found in the personal certificate store. If you know that you’re copy-pasting correctly but it’s still giving you that error, type the thumbprint out and see how that goes.
From here, I decided to cut my losses and start over without MVMC. The first step is that you need to run sysprep on the VM. If you don’t do this, the image will fail to provision and you might not be able to use the Azure VM you create. Sysprep is available in the Windows\System32\Sysprep directory, and has a GUI. In my case, I decided to copy my VM folder to ensure that it didn’t mess up the local copy of my demo box VM. The last thing I want is for my Azure copy to mess up my local copy. Anyhow, run sysprep.exe to begin sysprep.
Once sysprep’s GUI appears, select the “Enter System Out-of-Box Experience (OOBE)” drop-down option and check the “Generalize” checkbox, and then select the “Shutdown” drop-down menu item from the Shutdown Options list. Let sysprep do its thing, and then it’ll shut down your VM.
Once sysprep was done, I needed to find a way to get the VMDK files converted to VHDs. A blog post turned me on to StarWind Software’s V2V Converter. It’s a free tool which allows you to convert virtual hard drive files from one format to another. Installing this tool let me turn my set of VMDKs into one 45GB VHD. One note is that, at least on my machine, I needed to run the V2V Converter from a command prompt; executing the app directly from the Start menu would cause the app to appear for a moment and then disappear, as though some error killed the program. The tool installs by default in “%programfiles(x86)%\StarWind Software\StarWind V2V Image Converter\StarV2V.exe” From there, I just needed to get that big image into Azure.
Make sure that you have the Microsoft Azure Powershell cmdlets. Then, follow these instructions to connect your Azure subscription to the local machine and upload your VHD using the Add-AzureVhd cmdlet. Make sure that you have a Storage object and a Blob container, as that’s where you’re going to store the VHD file from which you’ll make a Virtual Machine image.
Once you have that image uploaded, you can create a new Azure VM from an image. Select the “MY IMAGES” option and you can pick your demo image. It’ll take a while for the VM to be provisioned. Also, don’t forget that you’re going to be charged for that VM as long as it’s running, so if you’re not using it, turn that sucker off.
I wanted a nice and easy GUI that let me tell an app where my VMDK files were and let it do all of the preparation, conversion, uploading, image creating, and provisioning. You aren’t going to get that. For these types of one-off scenarios, I accept (but am not happy with) the second approach listed. If I were doing this in an enterprise environment, there’d be a lot more Powershell.
