Scheduling Azure ML Compute Instances

Not too long ago, I mentioned that compute instances in Azure Machine Learning are lacking one critical thing: auto-stop functionality.

The single biggest problem I have with compute instances is that there is no auto-stop functionality to them. This is really frustrating because you’re paying for that virtual machine like you would any other, so if you forget to turn it off when you go home for the weekend, it’ll cost you. I wish there were a built-in option to shut off a compute instance after a certain amount of inactivity. Instead, you’ll need to start and stop them manually.

It turns out that you can and so I wanted to write a post to correct the record. I have a fairly standard compute instance called CSCompute.

One compute instance, ah-ah-ah.

If I select that compute instance, I can see resource properties. This includes a section called Schedules.

In fairness, it was way over on the right-hand side, so how was I to know?

Select the pencil icon and you get the ability to set a startup and shutdown schedule for this compute instance.

This seems like a reasonable idea.

From there, you can mark whether you’d like to start or stop the compute instance. In my case, I just want to have a shutdown schedule, not a startup schedule, although if this Azure ML compute instance were tied to my full-time job, I’d like to have the compute instance start up 5-10 minutes before I normally start to work, so that I don’t have to wait for it to begin. In my case, I just want to make sure that it shuts down. I’m in the eastern United States, so 5 AM UTC is either 1 AM or midnight, depending on whether it’s Daylight Savings Time or Normal Time for Normal People Who Are Normal. I’m rarely working on Azure ML stuff that late, so it’s a pretty safe shutdown time in the event that I forgot to turn it off during the day.

Always Be Closing.

If you’re creating a new compute instance, you’ll have the ability to add a schedule, although you do need to select the Next: Advanced Settings option instead of going straight to Create from the required settings page.

Start with a schedule.

If you want to build a compute instance via ARM template, schedule creation is pretty easy. Here are examples of stop and start schedules.

"schedules": {
  "value": {
	"computeStartStop": [
	  {
		"action": "Stop",
		"triggerType": "Cron",
		"cron": {
		  "startTime": "2022-01-10T02:34:23",
		  "timeZone": "UTC",
		  "expression": "00 20 * * 1,2,3,4,5"
		}
	  },
	  {
		"action": "Start",
		"triggerType": "Cron",
		"cron": {
		  "startTime": "2022-01-10T02:34:26",
		  "timeZone": "UTC",
		  "expression": "00 08 * * 1,2,3,4,5"
		}
	  }
	]
  }
}

You can then reference it in the schedules tag of the ML compute workspace resources like so:

"resources": [
    {
      "type": "Microsoft.MachineLearningServices/workspaces/computes",
      "apiVersion": "2021-07-01",
      "name": "[concat(parameters('workspaceName'), '/', parameters('computeInstanceName'))]",
      "location": "[parameters('location')]",
      "properties": {
        "computeType": "ComputeInstance",
        "disableLocalAuth": "[parameters('disableLocalAuth')]",
        "properties": {
          "VMSize": "[parameters('vmSize')]",
          "applicationSharingPolicy": "Personal",
          "sshSettings": {
            "sshPublicAccess": "[parameters('sshAccess')]"
          },
          "schedules": "[parameters('schedules')]"
        }
      }
    }
  ]

Conclusion

You can and should schedule your compute instances to shut down. Doing so will save you a lot of money over time, especially if you tend to be as forgetful as I am about these sorts of things.