I’ve recently had the good fortune of being asked to develop a Data Science Architecture based on Azure Machine Learning, Databricks, and Azure AI Services.
I won’t pretend to be a data scientist, but I do know infrastructure, and how to connect the right people and the right systems together.
That said, networking in Azure Machine Learning has several configuration options available, and understanding the options in full enables us to design the best possible architecture based upon the environment we are deploying into.
Workspace Managed Outbound Access
So where does the uncertainty stem from? Networking in Azure Machine Learning is concerned with 3 concepts
- Inbound access to the workspace itself
- Outbound access from the workspace to Azure resources, such as keyvaults.
- Outbound access from the Compute resources (Instances/clusters)
I will save a post about securing access to and from the workspace (1 and 2) for another day as that requires a broader view of Azure networking and DNS specifically.
Why do we need to be concerned about this? The compute nodes need legitimate access to
- The data you want them to process
- Any software packages which are required to do the processing
Where the data is publicly available, or your security policies permit the use of public endpoints, we can perhaps enable unrestricted Internet access. When dealing with sensitive or confidential data, or any corporate environment, the data should be accessible over private networks only.
I will discuss package security considerations and hosting in another post.
Security threats in AI
On that note, how about security threats? Data Exfiltration is the most obvious example. These compute nodes are processing potentially Terabytes of highly sensitive/confidential data. If one were to be compromised and that data freely uploaded due to no outbound restrictions the consequences could be severe. If corporate policy requires all traffic be inspected by a central firewall, then allowing Microsoft to manage the outbound access will not meet policy.
When looking at an existing workspace and the network settings we have 3 options available. It is important to note that you can only move up from Disabled > Allow Internet Outbound > Allow Only Approved Outbound. You cannot disable managed outbound access once you have enabled it.
Disabled vs Enabled – Impact
What’s the difference between selected Disabled vs Allow Internet Outbound/Allow Only Approved Outbound? This is seen when creating Compute within the workspace.
Disabled
When managed outbound access is disabled we are given the option to select a virtual network (within the same subscription and region) when we are creating compute in the workspace.
When the nodes are deployed they will pick up their network configuration from the vNet as usual, including route tables, and will appear as a connected device in the virtual network. We can now control traffic from those compute nodes in exactly the same way we do for all corporate traffic.
What if I don’t choose a network?
If we choose not to select a network and leave this option disabled when creating compute then nodes are provided access by the default outbound access for Azure VMs. It is likely Azure Machine Learning compute nodes will be impacted by the September 2025 deadline.
Enabled – Allow Internet Outbound
If we select to enabled outbound access, you can no longer select a vNet when creating the compute
But we do have other options made available in the Azure Portal which allow us to enable outbound access from the compute to any Private Endpoint within the environment. Note that this allows you to create rules which are targeted at any resource which can have a private endpoint, but the endpoint does need to be created for the rule to be set as “active”
In this configuration the compute nodes will have full, unrestricted, access to any Internet based resources, and will only be able to access internal secured resources which have a private endpoint configured and a specific outbound rule set at the workspace level. We can still connect to private resources if those resources have the public endpoint enabled.
Enabled – Allow Only Approved Outbound
Moving all the way to the right and selected to Allow Only Approved Outbound will block the default Internet access granted to the Compute resources, but it also provides us with two new options for our outbound rules – Service Tag and FQDN
Selecting Serice tag allows us to select any of the standard Azure Service Tags, allowing us to talk directly to any of them with no additional cost. But it is ALL of them, even those which are not part of your tenant.
Selecting FQDN provides us with a new information box which is warning us that we will now incur charges if we proceed
With this setting we can control precisely which FQDNs our compute nodes can talk to, but this is enabled by Azure spinning up, and charging us for, a managed Azure Firewall. Whilst this would be an improvement over the Service Tags mentioned above, as we can permit to our specific storage account only, it could be very very costly if TBs of data are then ingested via the Firewall.
What does this all mean?
As discussed earlier, there are 2 data flows we are predominantly concerned with from the managed compute nodes – Acces to Data, and Access to Packages.
For the highest security both Data and Packages should be available from internal/private resources, accessible from your own vNets. You will still need to permit traffic from the hosts out to various Azure services for the service to run (as per this link) via Service Tags, but these rules will be configured and monitored on your central firewall, and those service tags are likely already permitted for use by security policies.
If you have
- Well established processes for managing your Azure network
- You are planning on deploying Azure Machine Learning at scale for multiple projects
- You are handling sensitive information
- You work in a highly secure environment
Then you should continue to make use of those processes to manage this network traffic, and route it via your central firewall.
This means choosing to deploy the compute nodes into our own vNets using the “Disabled” option. This way we can connect to our storage accounts via their private endpoints directly, while also restricting Internet bound north/south traffic under the usual corporate policies and controls.
If, on the other hand, you have
- Limited platform level capabilities to manage Azure networking
- You are handling sensitive information
- You do not need the highest levels of security
Then the option for “Allow Internet Outbound” coupled with workspace outbound rules to permit direct access to specific private endpoint will help to keep your data secure, whilst allowing your data scientists access to public repositories to download packages to improve their productivity.
Finally, if you have
- Limited platform level capailities to manage Azure networking
- You are not handling sensitive information and the data is available publicly
- You are working in a low security environment
Then I would suggest looking at using the “Allow Internet Outbound” option with no rules required to enable access to privately held data.
Feel free to drop any comments/questions below and I’ll do my best to answer!
Thanks for reading, I hope it was useful