Networking-Blog

My WordPress Blog

Secure Azure Databricks Deployment

Prerequisites

Please take a note of Azure Databricks control plane endpoints for your workspace from here (map it based on region of your workspace). We’ll need these details to configure Azure Firewall rules later.

NameSourceDestinationProtocol:PortPurpose
databricks-webappAzure Databricks workspace subnetsRegion specific Webapp Endpointhttps:443Communication with Azure Databricks webapp
databricks-webappAzure Databricks workspace subnetsRegion specific Webapp Endpointhttps:443Communication with Azure Databricks webapp
databricks-observability-eventhubAzure Databricks workspace subnetsRegion specific Observability Event Hub Endpointhttps:9093Transit for Azure Databricks on-cluster service specific telemetry
databricks-artifact-blob-storageAzure Databricks workspace subnetsRegion specific Artifact Blob Storage Endpointhttps:443Stores Databricks Runtime images to be deployed on cluster nodes
databricks-dbfsAzure Databricks workspace subnetsDBFS Blob Storage Endpointhttps:443Azure Databricks workspace root storage
databricks-sql-metastore
(OPTIONAL – please see Step 3 for External Hive Metastore below)
Azure Databricks workspace subnetsRegion specific SQL Metastore Endpointtcp:3306Stores metadata for databases and child objects in a Azure Databricks workspace
Configure Azure Firewall Rules

With Azure Firewall, you can configure:

    • Application rules that define fully qualified domain names (FQDNs) that can be accessed from a subnet.
    • Network rules that define source address, protocol, destination port, and destination address.
    • Network traffic is subjected to the configured firewall rules when you route your network traffic to the firewall as the subnet default gateway.
Configure Application Rule

We first need to configure application rules to allow outbound access to Log Blob Storage and Artifact Blob Storage endpoints in the Azure Databricks control plane plus the DBFS Root Blob Storage for the workspace.

    • Go to the resource group, and select the firewall.
    • On the firewall page, under Settings, select Rules.
    • Select the Application rule collection tab.
    • Select Add application rule collection.
    • For Name, type databricks-control-plane-services.
    • For Priority, type 200.
    • For Action, select Allow.
    • Configure the following in Rules -> Target FQDNs
NameSource typeSourceProtocol
:Port
Target
FQDNs
databricks-spark-log
-blob-storage
IP AddressAzure
Databricks workspace subnets
https:443Refer notes
from Prerequisites
above
(for Central US)
databricks-audit-log-
blob-storage
IP AddressAzure
Databricks workspace subnets
https:443Refer notes
from Prerequisites
above
(for Central US)
This is separate
log storage only
for US regions today
databricks-artifact-
blob-storage
IP AddressAzure
Databricks workspace subnets
https:443Refer notes
from Prerequisites
above
(for Central US)
databricks-dbfsIP AddressAzure
Databricks workspace subnets
https:443Refer notes
from Prerequisites
above
Public Repositories for
Python and R Libraries(OPTIONAL –
if workspace users are
allowed to install libraries
from public repos)
IP AddressAzure
Databricks workspace subnets
https:443*pypi.org,*pythonhosted.
org,
cran.r-project.org
Add any other
public repos as
desired
Used by Ganglia UIIP AddressAzure
Databricks workspace subnets
https:443cdnjs.com or cdnjs.cloudflare.com
Configure Network Rule

Some endpoints can’t be configured as application rules using FQDNs. So we’ll set those up as network rules, namely the Observability Event Hub and Webapp.

    • Open the resource group adblabs-rg, and select the firewall.
    • On the firewall page, under Settings, select Rules.
    • Select the Network rule collection tab.
    • Select Add network rule collection.
    • For Name, type databricks-control-plane-services.
    • For Priority, type 200.
    • For Action, select Allow.
    • Configure the following in Rules -> IP Addresses.
NameProtocolSource typeSourceDest
type
Dest
Address
Dest Ports
databricks-
webapp
TCPIP AddressAzure
Databricks workspace
subnets
IP AddressRefer notes
from Prerequisites
above (for Central US)
443
databricks-
observability-
eventhub
TCPIP AddressAzure
Databricks workspace
subnets
IP AddressRefer notes
from
Prerequisites
above
(for Central US)
9093
databricks-sql-
metastore
(OPTIONAL –
please see
Step 3 for External Hive
Metastore
above)
TCPIP AddressAzure
Databricks workspace subnets
IP AddressRefer notes
from
Prerequisites
above
(for Central US)
3306


Below is a terraform script to add rules to the Azure Firewall.

# Priority range -14150 – 14159
resource “azurerm_firewall_policy_rule_collection_group” “data-archive” {
count = var.ENVIRONMENT == “npd” ? 1 : 0
name = “${module.names-group-data-office.standard[“afw-policy-group”]}-data-archive”
firewall_policy_id = data.azurerm_firewall_policy.this.id
priority = 14150

# Col1
application_rule_collection {
name = “${module.names-data-archive.standard[“afw-rule-collection”]}-perimeter”
priority = 3200
action = “Allow”
# Rule 1
rule {
name = “${module.names-data-archive.standard[“afw-rule”]}-allow-ado-agents-https-outbound”
protocols {
type = “Https”
port = 443
}
terminate_tls = true
source_addresses = var.customers.data-archive.vmss_subnets
destination_urls = [
“raw.githubusercontent.com/databricks/“, # required for partner terraform provider download (databricks) “github.com/databricks/“, # required for partner terraform provider download (databricks)
“objects.githubusercontent.com/github-production-release-asset” #github redirects to this. there is no way to make a specific rule as the rest is a SAS token which changes every time ] } # Rule 2 rule { # Allow databricks subnets access to databricks APIs name = “${module.names-data-archive.standard[“afw-rule”]}-allow-agent-databricks-api-calls” protocols { type = “Https” port = 443 } terminate_tls = true source_addresses = var.customers.data-archive.databrick_subnets destination_fqdns = [ “.azuredatabricks.net” #Calling databricks API for terraform creation of databricks objects
]
}
# Rule 3
rule {
# Allow databricks subnets access to Maven repo URLs – called by Databricks to install Java libraries needed by Spark
name = “${module.names-data-archive.standard[“afw-rule”]}-allow-databricks-maven-calls”
protocols {
type = “Https”
port = 443
}
terminate_tls = true
source_addresses = var.customers.data-archive.databrick_subnets
destination_urls = [
“maven-central.storage-download.googleapis.com/maven2/“, “repo1.maven.org/maven2/“,
“repos.spark-packages.org/*”
]
}
}

# Col 2
network_rule_collection {
# needed for dbricks to work – step 4 https://databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html
# https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/udr
name = “${module.names-data-archive.standard[“afw-rule”]}-dataarchive-databricks-net”
priority = 3100
action = “Allow”
# Rule 2
rule {
# Allow databricks subnets to access the databricks webapp
# Tried to use application rule but it failed complaining that it needed Target Fqdns,Target Urls, FqdnTags or WebCategories.
name = “${module.names-data-archive.standard[“afw-rule”]}-allow-dbricks-webapp”
protocols = [“TCP”]
source_addresses = var.customers.data-archive.databrick_subnets
destination_addresses = [“51.140.204.4/32”]
destination_ports = [“443”]
}
# Rule 1
rule {
# Allow databricks subnets to access the observability hub
name = “${module.names-data-archive.standard[“afw-rule”]}-allow-dbricks-observability”
protocols = [“TCP”]
source_addresses = var.customers.data-archive.databrick_subnets
destination_fqdns = [“prod-ukwest-observabilityeventhubs.servicebus.windows.net”] # obervability address for uksouth for databricks
destination_ports = [“9093”]
}
}

lifecycle {
create_before_destroy = true
}
}

Create User Defined Routes (UDRs)

At this point, the majority of the infrastructure setup for a secure, locked-down deployment has been completed. We now need to route appropriate traffic from Azure Databricks workspace subnets to the Control Plane SCC Relay IP (see FAQ below) and Azure Firewall setup earlier.

    • On the Azure portal menu, select All services and search for Route Tables. Go to that section.
    • Select Add
    • For Name, type firewall-route.
    • For Subscription, select your subscription.
    • For the Resource group, select adblabs-rg.
    • For Location, select the same location that you used previously i.e. Central US
    • Select Create.
    • Select Refresh, and then select the firewall-route-table route table.
    • Select Routes and then select Add.
    • For Route name, add to-firewall.
    • For Address prefix, add 0.0.0.0/0.
    • For Next hop type, select Virtual appliance.
    • For the Next hop address, add the Private IP address for the Azure Firewall that you noted earlier.
    • Select OK.

Now add one more route for Azure Databricks SCC Relay IP.

    • Select Routes and then select Add.
    • For Route name, add to-central-us-databricks-SCC-relay-ip.
    • For Address prefix, add the Control Plane SCC relay service IP address for Central US from here. Please note that there could be more than one ip addresses for relay service and in that case add additional rules on the UDR accordingly. In order to get SCC relay IP, please run nslookup on the relay service endpoint e.g.,
    • For Next hop type, select Internet, although it says Internet, traffic between Azure Databricks data plane and Azure Databricks SCC relay service IP stays on Azure Network and does not travel over public internet, for more details please refer to this guide).
      .
    • Select OK.

The route table needs to be associated with both of the Azure Databricks workspace subnets.

    • Go to the firewall-route-table.
    • Select Subnets and then select Associate.
    • Select Virtual network > azuredatabricks-spoke-vnet.
    • For Subnet, select both workspace subnets.
    • Select OK.

Below if the terraform code:

   "routeTable": {
      "disableBgpRoutePropagation": true,
      "routes": [
        {
          "name": "default-via-fw",
          "addressPrefix": "0.0.0.0/0",
          "nextHopIpAddress": "10.196.0.4",
          "nextHopType": "VirtualAppliance"
        },
        {
          "name": "to-uk-south-databricks-webapp",
          "addressPrefix": "51.140.204.4/32",
          "nextHopIpAddress": "",
          "nextHopType": "Internet"
        },
        {
          "name": "to-uk-south-databricks-scc-relay",
          "addressPrefix": "51.141.103.193/32",
          "nextHopIpAddress": "",
          "nextHopType": "Internet"
        },
        {
          "name": "to-uk-south-databricks-control-plane",
          "addressPrefix": "51.140.203.27/32",
          "nextHopIpAddress": "",
          "nextHopType": "Internet"
        },
        {
          "name": "to-uk-south-databricks-extended-infrastructure",
          "addressPrefix": "51.141.64.128/28",
          "nextHopIpAddress": "",
          "nextHopType": "Internet"
        }
      ]
    }

Azure Create Application Gateway that hosts multiple web sites using Azure PowerShell

You can use Azure Powershell to configure the hosting of multiple web sites when you create an application gateway. In this article, you define backend address pools using virtual machines scale sets. You then configure listeners and rules based on domains that you own to make sure web traffic arrives at the appropriate servers in the pools. This article assumes that you own multiple domains and uses examples of www.contoso.com and www.fabrikam.com.

In this article, you learn how to:

  • Set up the network
  • Create an application gateway
  • Create backend listeners
  • Create routing rules
  • Create virtual machine scale sets with the backend pools
  • Create a CNAME record in your domain
Multi-site routing example

If you don’t have an Azure subscription, create a free account before you begin.

 Note

This article has been updated to use the new Azure PowerShell Az module. You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. For Az module installation instructions, see Install Azure PowerShell.

Use Azure Cloud Shell

Azure hosts Azure Cloud Shell, an interactive shell environment that you can use through your browser. You can use either Bash or PowerShell with Cloud Shell to work with Azure services. You can use the Cloud Shell preinstalled commands to run the code in this article without having to install anything on your local environment.

To start Azure Cloud Shell:

OptionExample/Link
Select Try It in the upper-right corner of a code block. Selecting Try It doesn’t automatically copy the code to Cloud Shell.Example of Try It for Azure Cloud Shell
Go to https://shell.azure.com, or select the Launch Cloud Shell button to open Cloud Shell in your browser.
Select the Cloud Shell button on the menu bar at the upper right in the Azure portal.Cloud Shell button in the Azure portal

To run the code in this article in Azure Cloud Shell:

  1. Start Cloud Shell.
  2. Select the Copy button on a code block to copy the code.
  3. Paste the code into the Cloud Shell session by selecting Ctrl+Shift+V on Windows and Linux or by selecting Cmd+Shift+V on macOS.
  4. Select Enter to run the code.

If you choose to install and use the PowerShell locally, this article requires the Azure PowerShell module version 1.0.0 or later. To find the version, run Get-Module -ListAvailable Az . If you need to upgrade, see Install Azure PowerShell module. If you’re running PowerShell locally, you also need to run Login-AzAccount to create a connection with Azure.

Create a resource group

A resource group is a logical container into which Azure resources are deployed and managed. Create an Azure resource group using New-AzResourceGroup.Azure PowerShellCopyTry It

New-AzResourceGroup -Name myResourceGroupAG -Location eastus

Create network resources

Create the subnet configurations using New-AzVirtualNetworkSubnetConfig. Create the virtual network using New-AzVirtualNetwork with the subnet configurations. And finally, create the public IP address using New-AzPublicIpAddress. These resources are used to provide network connectivity to the application gateway and its associated resources.Azure PowerShellCopyTry It

$backendSubnetConfig = New-AzVirtualNetworkSubnetConfig `
  -Name myBackendSubnet `
  -AddressPrefix 10.0.1.0/24

$agSubnetConfig = New-AzVirtualNetworkSubnetConfig `
  -Name myAGSubnet `
  -AddressPrefix 10.0.2.0/24

$vnet = New-AzVirtualNetwork `
  -ResourceGroupName myResourceGroupAG `
  -Location eastus `
  -Name myVNet `
  -AddressPrefix 10.0.0.0/16 `
  -Subnet $backendSubnetConfig, $agSubnetConfig

$pip = New-AzPublicIpAddress `
  -ResourceGroupName myResourceGroupAG `
  -Location eastus `
  -Name myAGPublicIPAddress `
  -AllocationMethod Dynamic

Create an application gateway

Create the IP configurations and frontend port

Associate the subnet that you previously created to the application gateway using New-AzApplicationGatewayIPConfiguration. Assign the public IP address to the application gateway using New-AzApplicationGatewayFrontendIPConfig.Azure PowerShellCopyTry It

$vnet = Get-AzVirtualNetwork `
  -ResourceGroupName myResourceGroupAG `
  -Name myVNet

$subnet=$vnet.Subnets[0]

$gipconfig = New-AzApplicationGatewayIPConfiguration `
  -Name myAGIPConfig `
  -Subnet $subnet

$fipconfig = New-AzApplicationGatewayFrontendIPConfig `
  -Name myAGFrontendIPConfig `
  -PublicIPAddress $pip

$frontendport = New-AzApplicationGatewayFrontendPort `
  -Name myFrontendPort `
  -Port 80

Create the backend pools and settings

Create the first backend address pool for the application gateway using New-AzApplicationGatewayBackendAddressPool. Configure the settings for the pool using New-AzApplicationGatewayBackendHttpSettings.Azure PowerShellCopyTry It

$contosoPool = New-AzApplicationGatewayBackendAddressPool `
  -Name contosoPool

$fabrikamPool = New-AzApplicationGatewayBackendAddressPool `
  -Name fabrikamPool

$poolSettings = New-AzApplicationGatewayBackendHttpSettings `
  -Name myPoolSettings `
  -Port 80 `
  -Protocol Http `
  -CookieBasedAffinity Enabled `
  -RequestTimeout 120

Create the listeners and rules

Listeners are required to enable the application gateway to route traffic appropriately to the backend address pools. In this article, you create two listeners for your two domains. Listeners are created for the contoso.com and fabrikam.com domains.

Create the first listener using New-AzApplicationGatewayHttpListener with the frontend configuration and frontend port that you previously created. A rule is required for the listener to know which backend pool to use for incoming traffic. Create a basic rule named contosoRule using New-AzApplicationGatewayRequestRoutingRule.Azure PowerShellCopyTry It

$contosolistener = New-AzApplicationGatewayHttpListener `
  -Name contosoListener `
  -Protocol Http `
  -FrontendIPConfiguration $fipconfig `
  -FrontendPort $frontendport `
  -HostName "www.contoso.com"

$fabrikamlistener = New-AzApplicationGatewayHttpListener `
  -Name fabrikamListener `
  -Protocol Http `
  -FrontendIPConfiguration $fipconfig `
  -FrontendPort $frontendport `
  -HostName "www.fabrikam.com"

$contosoRule = New-AzApplicationGatewayRequestRoutingRule `
  -Name contosoRule `
  -RuleType Basic `
  -HttpListener $contosoListener `
  -BackendAddressPool $contosoPool `
  -BackendHttpSettings $poolSettings

$fabrikamRule = New-AzApplicationGatewayRequestRoutingRule `
  -Name fabrikamRule `
  -RuleType Basic `
  -HttpListener $fabrikamListener `
  -BackendAddressPool $fabrikamPool `
  -BackendHttpSettings $poolSettings

Create the application gateway

Now that you created the necessary supporting resources, specify parameters for the application gateway using New-AzApplicationGatewaySku, and then create it using New-AzApplicationGateway.Azure PowerShellCopyTry It

$sku = New-AzApplicationGatewaySku `
  -Name Standard_Medium `
  -Tier Standard `
  -Capacity 2

$appgw = New-AzApplicationGateway `
  -Name myAppGateway `
  -ResourceGroupName myResourceGroupAG `
  -Location eastus `
  -BackendAddressPools $contosoPool, $fabrikamPool `
  -BackendHttpSettingsCollection $poolSettings `
  -FrontendIpConfigurations $fipconfig `
  -GatewayIpConfigurations $gipconfig `
  -FrontendPorts $frontendport `
  -HttpListeners $contosoListener, $fabrikamListener `
  -RequestRoutingRules $contosoRule, $fabrikamRule `
  -Sku $sku

Create virtual machine scale sets

In this example, you create two virtual machine scale sets that support the two backend pools that you created. The scale sets that you create are named myvmss1 and myvmss2. Each scale set contains two virtual machine instances on which you install IIS. You assign the scale set to the backend pool when you configure the IP settings.Azure PowerShellCopyTry It

$vnet = Get-AzVirtualNetwork `
  -ResourceGroupName myResourceGroupAG `
  -Name myVNet

$appgw = Get-AzApplicationGateway `
  -ResourceGroupName myResourceGroupAG `
  -Name myAppGateway

$contosoPool = Get-AzApplicationGatewayBackendAddressPool `
  -Name contosoPool `
  -ApplicationGateway $appgw

$fabrikamPool = Get-AzApplicationGatewayBackendAddressPool `
  -Name fabrikamPool `
  -ApplicationGateway $appgw

for ($i=1; $i -le 2; $i++)
{
  if ($i -eq 1) 
  {
    $poolId = $contosoPool.Id
  }
  if ($i -eq 2)
  {
    $poolId = $fabrikamPool.Id
  }

  $ipConfig = New-AzVmssIpConfig `
    -Name myVmssIPConfig$i `
    -SubnetId $vnet.Subnets[1].Id `
    -ApplicationGatewayBackendAddressPoolsId $poolId

  $vmssConfig = New-AzVmssConfig `
    -Location eastus `
    -SkuCapacity 2 `
    -SkuName Standard_DS2 `
    -UpgradePolicyMode Automatic

  Set-AzVmssStorageProfile $vmssConfig `
    -ImageReferencePublisher MicrosoftWindowsServer `
    -ImageReferenceOffer WindowsServer `
    -ImageReferenceSku 2016-Datacenter `
    -ImageReferenceVersion latest `
    -OsDiskCreateOption FromImage

  Set-AzVmssOsProfile $vmssConfig `
    -AdminUsername azureuser `
    -AdminPassword "Azure123456!" `
    -ComputerNamePrefix myvmss$i

  Add-AzVmssNetworkInterfaceConfiguration `
    -VirtualMachineScaleSet $vmssConfig `
    -Name myVmssNetConfig$i `
    -Primary $true `
    -IPConfiguration $ipConfig

  New-AzVmss `
    -ResourceGroupName myResourceGroupAG `
    -Name myvmss$i `
    -VirtualMachineScaleSet $vmssConfig
}

Install IIS

Azure PowerShellCopyTry It

$publicSettings = @{ "fileUris" = (,"https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1"); 
  "commandToExecute" = "powershell -ExecutionPolicy Unrestricted -File appgatewayurl.ps1" }

for ($i=1; $i -le 2; $i++)
{
  $vmss = Get-AzVmss `
    -ResourceGroupName myResourceGroupAG `
    -VMScaleSetName myvmss$i

  Add-AzVmssExtension -VirtualMachineScaleSet $vmss `
    -Name "customScript" `
    -Publisher "Microsoft.Compute" `
    -Type "CustomScriptExtension" `
    -TypeHandlerVersion 1.8 `
    -Setting $publicSettings

  Update-AzVmss `
    -ResourceGroupName myResourceGroupAG `
    -Name myvmss$i `
    -VirtualMachineScaleSet $vmss
}

Create CNAME record in your domain

After the application gateway is created with its public IP address, you can get the DNS address and use it to create a CNAME record in your domain. You can use Get-AzPublicIPAddress to get the DNS address of the application gateway. Copy the fqdn value of the DNSSettings and use it as the value of the CNAME record that you create. Using A-records isn’t recommended because the VIP may change when the application gateway is restarted in the V1 SKU.Azure PowerShellCopyTry It

Get-AzPublicIPAddress -ResourceGroupName myResourceGroupAG -Name myAGPublicIPAddress

Test the application gateway

Enter your domain name into the address bar of your browser. Such as, http://www.contoso.com.

Test contoso site in application gateway

Change the address to your other domain and you should see something like the following example:

Test fabrikam site in application gateway
Clean up resources
When no longer needed, remove the resource group, application gateway, and all related resources using Remove-AzResourceGroup.

Azure PowerShell

Copy

Try It
Remove-AzResourceGroup -Name myResourceGroupAG

AzCopy large files to Azure Data Lake Storage

There are many ways to upload files to Azure Data Lake Storage (ADLS) Gen2.In this article, we will compare two popular ways for an organization to upload files to ADLS. Mainly, we will compare below parameters to identify which one suits best for your need:

  1. Performance: Speed by which the file is being uploaded
  2. Ease: How easy it is to set up and use
  3. Automation: Where there will be or not any manual intervention after operationalization

In this tutorial we will rate each of the above three parameters with the range from 1 to 5, with 1 being lowest and 5 being highest supported/achievable parameter.

We will be testing these two approaches to upload 10 GB test file:

  1. AzCopy
  2. Azure Storage Explorer

Without further ado, let’s get started.

AzCopy

In order to transfer file using AzCopy, you will need AzCopy, which you can download from here.

Once the AzCopy is downloaded, lets create a ADLS Gen2 for our tutorial. I have already created one for the purpose of this tutorial and a container for the AzCopy test:

Screenshot example of AzCopy ADLS Gen2 setup

In order to use AzCopy to transfer file to container, we need a SAS token, lets generate our SAS token:

Click on “Shared access signature” under “Security + networking” within Storage account

Screenshot example of SAS token generation

I have selected 1-day range for the SAS token to be active. Then click the “Generate SAS and Connection string” button

Screenshot example range for SAS token

Copy the SAS token from the available URLs, it should look like below:


?sv=2020-08-04&ss=bfqt&srt=sco&sp=rwdlacupx&se=2021-09-30T04:15:53Z&st=2021-09-28T20:15:53Z&spr=https&sig=w%2Bn8Jl%2BsnOe1Xk5A2CfjM6%2BDuHz8kb4ZFQugmgtOcfQ%3D

Now that we have SAS token lets generate our AzCopy command, below is the syntax for AzCopy command-line:


azcopy copy '' 'https://..core.windows.net//'

// TIP This example encloses path arguments with single quotes (''). 
// Use single quotes in all command shells except for the Windows Command Shell (cmd.exe). 
// If you're using a Windows Command Shell (cmd.exe), enclose path arguments with double quotes ("") instead of single quotes ('').

I have placed the sample file and AzCopy application both in the root folder on C drive called “AzCopy”. Replacing the values in above syntax, below is what we get:


azcopy copy “c:\AzCopy\10GB.bin” “https://tuttransfer.blob.core.windows.net/azcopy/10GB.bin?sv=2020-08-04&ss=bfqt&srt=sco&sp=rwdlacupx&se=2021-09-30T04:15:53Z&st=2021-09-28T20:15:53Z&spr=https&sig=w%2Bn8Jl%2BsnOe1Xk5A2CfjM6%2BDuHz8kb4ZFQugmgtOcfQ%3D”

AzCopy being command-line utility, we will have to open command prompt and execute above command, at the end of the execution, AzCopy will provide a summary of transfer. Let’s execute:

Screenshot example command line summary of ADLS transfer with AzCopy and SAS token authorization

So, it took 16.34 mins to transfer 10 GB file using AzCopy and SAS token authorization.

Now let’s try using Storage Explorer.

Storage Explorer

Azure Storage Explorer has graphical user interface to interact with storage account on Azure, which makes it more user friendly, but here we are trying to compare which one of these two options is best for transferring big files to ADLS Gen 2. Let’s download the Storage Explorer from here. Execute the executable file downloaded and follow the installation wizard.

Once installed, the storage explorer should automatically start with the setup screen to connect to Azure Storage, select “ADLS Gen2 container or directory” option:

Screenshot example Azure Storage Explorer to ADLS Gen2 setuo

Now that we have the SAS token we generated during AzCopy test, lets select SAS URL option on next screen and then click “Next”:

Screenshot example Azure Storage Explorer select connection method

Input the friendly name in Display name and paste the SAS URL and click next:

Screenshot example Azure Storage Explorer connection info

Click “Connect” on the Summary screen:

Screenshot example summary screen

This step will connect to your storage container, the next step is simply using GUI to click “Upload”, select the file and start uploading:

Screenshot example Storage Explorer upload

We can monitor the transfer on the bottom pane:

Screenshot example Storage Explorer monitor transfer

The total time taken for Storage explorer was 18.03 minutes.

Conclusion

Let’s use our parameters to identify which approach is best between above two approaches to transfer big files to Azure Storage

Performance: Speed by which the file is being uploaded

I would rate performance of the AzCopy better than Storage Explorer, it took ~3 mins less than storage explorer:

  • AzCopy – 5 points
  • Storage Explorer – 3 points

Ease: How easy it is to set up and use

Storage Explorer is much easier to use than AzCopy because of the user-friendly GUI, while AzCopy is command-line utility and requires command formation to be executed

  • AzCopy – 3 points
  • Storage Explorer – 5 points

Automation: Where there will be or not any manual intervention after operationalization

AzCopy can be automated using Windows Task, PowerShell, and other orchestration and parameterized executions, while Storage Explorer is completely manual, we cannot automate jobs or tasks using Storage explorer GUI

  • AzCopy – 4 points
  • Storage Explorer – 1 point
AzCopy vs Storage Explorer comparison table

Though both of the options have pros and cons, AzCopy can cover all scenarios of Storage Explorer transferring to/from ADLS Gen2, while some use cases might not be achievable using Storage Explorer, especially automation.

Additional Notes:

Azure AD PIM Request to enable contributor access to Azure resources.

Cmd command:

azcopy login – To authenticate to Azure AD

Azure AD:
azcopy copy “C:\AzCopy/FujitsuOB/textfile.txt” “https://account.blob.core.windows.net/textfile.txt

Azure Storage Access Token:
azcopy copy “C:\AzCopy/FujitsuOB/textfile.txt” “https://account.blob.core.windows.net/mycontainer1/?sv=2018-03-28&ss=bjqt&srt=sco&sp=rwddgcup&se=2019-05-01T05:01:17Z&st=2019-04-30T21:01:17Z&spr=https&sig=MGCXiyEzbtttkr3ewJIh2AR8KrghSy1DGM9ovN734bQF4%3D” –recursive=true

Azure AppGW WAF Inspect Request Body

For Application gateway with WAF_V2 SKU, it is now in public Preview to increase the request body size to 2000KB(2MB) as long as we are using OWASP 3.2 in the WAF.

For Application gateway with WAF_V2 SKU, it is now in public Preview to increase the request body size to 2000KB(2MB) as long as we are using OWASP 3.2 in the WAF.

Recommended size is 128k.

Configure Azure Application Gateway (WAF) to preserve client ip address

HTTP requests often pass through one or more proxy servers before they reach the endpoint web server, which changes the source IP address for the request. As a result, endpoint web servers cannot rely on the source IP from the network connection (socket) to be the IP address of the original request. For this reason, you may want to use one of two options to preserve the original client IP address: X-Forwarded-For (XFF), or transparent proxy.

The backend server sees requests via its logs as coming from the Application Gateway (via its private IP Address) and not on the requesting “public” IP Address.

How do I configure WAF so that it will preserve/pass the requests as coming from public P address (not the private IP address)?

We are unable to preserve the client IP because the Application gateway is a proxy. It will replace the original client IP with the Application gateway instance IP and forward requests to the backend server. However, Application gateway inserts extra headers to all requests before it forwards the requests to the backend. It includes the x-forwarded-for header which has the original client IP information.

You can configure Application gateway to modify request and response headers and URL by using Rewrite HTTP headers and URL or to modify the URI path by using a path-override setting. However, unless configured to do so, all incoming requests are proxied to the backend.
You can use header rewrite to remove the port information from the X-Forwarded-For header to only keep the IP addresses.

Designing a DMZ for Azure Virtual Machines

This article will show you three designs, each building on the other, for a demilitarised zone (DMZ) or perimeter network for Internet facing n-tier applications based on Azure virtual machines and networking.

The DMZ

The concept of a DMZ or perimeter network is not new; it’s a classic design that uses a layered network security approach to minimize the attack footprint of an application.

In a simple design:

  1. Web servers are placed in one VLAN, with just TCP 80 and TCP 443 accessible from the Internet.
  2. Application servers are in another VLAN. Web servers can communicate with the application servers using just the application protocol. The Internet has no access to this VLAN.
  3. Database servers are in a third VLAN. Application servers can communicate with the database servers using only the database communications protocol. Web servers and the Internet have no access to this VLAN.

You can modify this design in many ways, including:

  • Adding additional application layer security.
  • Including reverse proxies.
  • Using logical implementations of multiple VLANs by using other methods of network isolation, such as network security groups (NSGs) in Azure.

The concept of a DMZ with n-tier applications

So how do you recreate this concept in Azure for virtual machines? I’ll present you with three designs from Microsoft, each of which builds on the concepts of the previous ones.

Network Security Groups


The first and simplest way to build a DMZ in Azure is to use network security groups (NSGs). An NSG is a five-tuple rule that will allow or block TCP or UDP traffic between designated addresses on a virtual network.

You can deploy an n-tier solution into a single virtual network that is split into two or more subnets; each subnet plays the role of a VLAN, as shown above. NSG rules are then created to restrict network traffic. In the below diagram, NSGs will:

  • Allow web traffic into the FrontEnd subnet.
  • All application traffic to flow from the FrontEnd subnet to the BackEnd subnet.
  • Block all other traffic.


A DMZ using Azure network security groups 


The benefit of this design is that it is very simple. The drawback of this design is that it assumes that your potential hackers are stuck in the 1990s; a modern attack tries to compromise the application layer. A port scan of the above from an external point will reveal that TCP 80/443 are open, so an attacker will try to attack those ports. A simple five-tuple rule will not block that traffic, so the hacker can either flood the target with a DDOS attack or compromise application vulnerabilities.


NSGs and a Firewall


Modern edge network devices can protect and enhance hosted applications with applications layer scanning and/or reverse proxy services. The Azure Marketplace allows you to deploy these kinds of devices from multiple vendors into your Azure virtual networks.


The following design below uses a virtual network appliance to protect an application from threats; this offers more than just simple protocol filtering because the appliance understands the allowed traffic and can identify encapsulated risks.

Using a firewall virtual appliance with NSGs to create a DMZ

NSGS are deployed to enforce that all communications from the Internet must flow through the virtual appliance. NSGs will also control the protocols and ports that are allowed for internal communications between the subnets.

Ideally, we’d like to have all communications inside of the virtual network to flow through the virtual appliance, but the default routing rules of the network will prevent this from happening.

User Defined Routes, NSGs, and a Firewall

We can override the default routes of a virtual network using user-defined routes (UDRs). The following design uses one subnet in a single virtual network for each layer of the n-tier application. An additional subnet is created just for the virtual firewall appliance, which will secure the application.

UDRs are created to override the default routes between the subnets, forcing all traffic between subnets to route via the virtual firewall appliance. NSGs are created to enforce this routing and block traffic via the default routes.

An Azure DMZ made from user-defined routes, a virtual appliance firewall and NSGs (Image Credit: Microsoft)

The result is a DMZ where the virtual appliance controls all traffic to/from the Internet and between the subnets.

Tip: Try to use a next generation firewall and compliment this with defence with additional security products that will work with the Azure Security Center so that you have a single view of all trends and risks.

Azure Create Application Gateway that hosts multiple web sites using Azure PowerShell

You can use Azure Powershell to configure the hosting of multiple web sites when you create an application gateway. In this article, you define backend address pools using virtual machines scale sets. You then configure listeners and rules based on domains that you own to make sure web traffic arrives at the appropriate servers in the pools. This article assumes that you own multiple domains and uses examples of www.contoso.com and www.fabrikam.com.

In this article, you learn how to:

  • Set up the network
  • Create an application gateway
  • Create backend listeners
  • Create routing rules
  • Create virtual machine scale sets with the backend pools
  • Create a CNAME record in your domain
Multi-site routing example

If you don’t have an Azure subscription, create a free account before you begin.

 Note

This article has been updated to use the new Azure PowerShell Az module. You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. For Az module installation instructions, see Install Azure PowerShell.

Use Azure Cloud Shell

Azure hosts Azure Cloud Shell, an interactive shell environment that you can use through your browser. You can use either Bash or PowerShell with Cloud Shell to work with Azure services. You can use the Cloud Shell preinstalled commands to run the code in this article without having to install anything on your local environment.

To start Azure Cloud Shell:

OptionExample/Link
Select Try It in the upper-right corner of a code block. Selecting Try It doesn’t automatically copy the code to Cloud Shell.Example of Try It for Azure Cloud Shell
Go to https://shell.azure.com, or select the Launch Cloud Shell button to open Cloud Shell in your browser.
Select the Cloud Shell button on the menu bar at the upper right in the Azure portal.Cloud Shell button in the Azure portal

To run the code in this article in Azure Cloud Shell:

  1. Start Cloud Shell.
  2. Select the Copy button on a code block to copy the code.
  3. Paste the code into the Cloud Shell session by selecting Ctrl+Shift+V on Windows and Linux or by selecting Cmd+Shift+V on macOS.
  4. Select Enter to run the code.

If you choose to install and use the PowerShell locally, this article requires the Azure PowerShell module version 1.0.0 or later. To find the version, run Get-Module -ListAvailable Az . If you need to upgrade, see Install Azure PowerShell module. If you’re running PowerShell locally, you also need to run Login-AzAccount to create a connection with Azure.

Create a resource group

A resource group is a logical container into which Azure resources are deployed and managed. Create an Azure resource group using New-AzResourceGroup.Azure PowerShellCopyTry It

New-AzResourceGroup -Name myResourceGroupAG -Location eastus

Create network resources

Create the subnet configurations using New-AzVirtualNetworkSubnetConfig. Create the virtual network using New-AzVirtualNetwork with the subnet configurations. And finally, create the public IP address using New-AzPublicIpAddress. These resources are used to provide network connectivity to the application gateway and its associated resources.Azure PowerShellCopyTry It

$backendSubnetConfig = New-AzVirtualNetworkSubnetConfig `
  -Name myBackendSubnet `
  -AddressPrefix 10.0.1.0/24

$agSubnetConfig = New-AzVirtualNetworkSubnetConfig `
  -Name myAGSubnet `
  -AddressPrefix 10.0.2.0/24

$vnet = New-AzVirtualNetwork `
  -ResourceGroupName myResourceGroupAG `
  -Location eastus `
  -Name myVNet `
  -AddressPrefix 10.0.0.0/16 `
  -Subnet $backendSubnetConfig, $agSubnetConfig

$pip = New-AzPublicIpAddress `
  -ResourceGroupName myResourceGroupAG `
  -Location eastus `
  -Name myAGPublicIPAddress `
  -AllocationMethod Dynamic

Create an application gateway

Create the IP configurations and frontend port

Associate the subnet that you previously created to the application gateway using New-AzApplicationGatewayIPConfiguration. Assign the public IP address to the application gateway using New-AzApplicationGatewayFrontendIPConfig.Azure PowerShellCopyTry It

$vnet = Get-AzVirtualNetwork `
  -ResourceGroupName myResourceGroupAG `
  -Name myVNet

$subnet=$vnet.Subnets[0]

$gipconfig = New-AzApplicationGatewayIPConfiguration `
  -Name myAGIPConfig `
  -Subnet $subnet

$fipconfig = New-AzApplicationGatewayFrontendIPConfig `
  -Name myAGFrontendIPConfig `
  -PublicIPAddress $pip

$frontendport = New-AzApplicationGatewayFrontendPort `
  -Name myFrontendPort `
  -Port 80

Create the backend pools and settings

Create the first backend address pool for the application gateway using New-AzApplicationGatewayBackendAddressPool. Configure the settings for the pool using New-AzApplicationGatewayBackendHttpSettings.Azure PowerShellCopyTry It

$contosoPool = New-AzApplicationGatewayBackendAddressPool `
  -Name contosoPool

$fabrikamPool = New-AzApplicationGatewayBackendAddressPool `
  -Name fabrikamPool

$poolSettings = New-AzApplicationGatewayBackendHttpSettings `
  -Name myPoolSettings `
  -Port 80 `
  -Protocol Http `
  -CookieBasedAffinity Enabled `
  -RequestTimeout 120

Create the listeners and rules

Listeners are required to enable the application gateway to route traffic appropriately to the backend address pools. In this article, you create two listeners for your two domains. Listeners are created for the contoso.com and fabrikam.com domains.

Create the first listener using New-AzApplicationGatewayHttpListener with the frontend configuration and frontend port that you previously created. A rule is required for the listener to know which backend pool to use for incoming traffic. Create a basic rule named contosoRule using New-AzApplicationGatewayRequestRoutingRule.Azure PowerShellCopyTry It

$contosolistener = New-AzApplicationGatewayHttpListener `
  -Name contosoListener `
  -Protocol Http `
  -FrontendIPConfiguration $fipconfig `
  -FrontendPort $frontendport `
  -HostName "www.contoso.com"

$fabrikamlistener = New-AzApplicationGatewayHttpListener `
  -Name fabrikamListener `
  -Protocol Http `
  -FrontendIPConfiguration $fipconfig `
  -FrontendPort $frontendport `
  -HostName "www.fabrikam.com"

$contosoRule = New-AzApplicationGatewayRequestRoutingRule `
  -Name contosoRule `
  -RuleType Basic `
  -HttpListener $contosoListener `
  -BackendAddressPool $contosoPool `
  -BackendHttpSettings $poolSettings

$fabrikamRule = New-AzApplicationGatewayRequestRoutingRule `
  -Name fabrikamRule `
  -RuleType Basic `
  -HttpListener $fabrikamListener `
  -BackendAddressPool $fabrikamPool `
  -BackendHttpSettings $poolSettings

Create the application gateway

Now that you created the necessary supporting resources, specify parameters for the application gateway using New-AzApplicationGatewaySku, and then create it using New-AzApplicationGateway.Azure PowerShellCopyTry It

$sku = New-AzApplicationGatewaySku `
  -Name Standard_Medium `
  -Tier Standard `
  -Capacity 2

$appgw = New-AzApplicationGateway `
  -Name myAppGateway `
  -ResourceGroupName myResourceGroupAG `
  -Location eastus `
  -BackendAddressPools $contosoPool, $fabrikamPool `
  -BackendHttpSettingsCollection $poolSettings `
  -FrontendIpConfigurations $fipconfig `
  -GatewayIpConfigurations $gipconfig `
  -FrontendPorts $frontendport `
  -HttpListeners $contosoListener, $fabrikamListener `
  -RequestRoutingRules $contosoRule, $fabrikamRule `
  -Sku $sku

Create virtual machine scale sets

In this example, you create two virtual machine scale sets that support the two backend pools that you created. The scale sets that you create are named myvmss1 and myvmss2. Each scale set contains two virtual machine instances on which you install IIS. You assign the scale set to the backend pool when you configure the IP settings.Azure PowerShellCopyTry It

$vnet = Get-AzVirtualNetwork `
  -ResourceGroupName myResourceGroupAG `
  -Name myVNet

$appgw = Get-AzApplicationGateway `
  -ResourceGroupName myResourceGroupAG `
  -Name myAppGateway

$contosoPool = Get-AzApplicationGatewayBackendAddressPool `
  -Name contosoPool `
  -ApplicationGateway $appgw

$fabrikamPool = Get-AzApplicationGatewayBackendAddressPool `
  -Name fabrikamPool `
  -ApplicationGateway $appgw

for ($i=1; $i -le 2; $i++)
{
  if ($i -eq 1) 
  {
    $poolId = $contosoPool.Id
  }
  if ($i -eq 2)
  {
    $poolId = $fabrikamPool.Id
  }

  $ipConfig = New-AzVmssIpConfig `
    -Name myVmssIPConfig$i `
    -SubnetId $vnet.Subnets[1].Id `
    -ApplicationGatewayBackendAddressPoolsId $poolId

  $vmssConfig = New-AzVmssConfig `
    -Location eastus `
    -SkuCapacity 2 `
    -SkuName Standard_DS2 `
    -UpgradePolicyMode Automatic

  Set-AzVmssStorageProfile $vmssConfig `
    -ImageReferencePublisher MicrosoftWindowsServer `
    -ImageReferenceOffer WindowsServer `
    -ImageReferenceSku 2016-Datacenter `
    -ImageReferenceVersion latest `
    -OsDiskCreateOption FromImage

  Set-AzVmssOsProfile $vmssConfig `
    -AdminUsername azureuser `
    -AdminPassword "Azure123456!" `
    -ComputerNamePrefix myvmss$i

  Add-AzVmssNetworkInterfaceConfiguration `
    -VirtualMachineScaleSet $vmssConfig `
    -Name myVmssNetConfig$i `
    -Primary $true `
    -IPConfiguration $ipConfig

  New-AzVmss `
    -ResourceGroupName myResourceGroupAG `
    -Name myvmss$i `
    -VirtualMachineScaleSet $vmssConfig
}

Install IIS

Azure PowerShellCopyTry It

$publicSettings = @{ "fileUris" = (,"https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1"); 
  "commandToExecute" = "powershell -ExecutionPolicy Unrestricted -File appgatewayurl.ps1" }

for ($i=1; $i -le 2; $i++)
{
  $vmss = Get-AzVmss `
    -ResourceGroupName myResourceGroupAG `
    -VMScaleSetName myvmss$i

  Add-AzVmssExtension -VirtualMachineScaleSet $vmss `
    -Name "customScript" `
    -Publisher "Microsoft.Compute" `
    -Type "CustomScriptExtension" `
    -TypeHandlerVersion 1.8 `
    -Setting $publicSettings

  Update-AzVmss `
    -ResourceGroupName myResourceGroupAG `
    -Name myvmss$i `
    -VirtualMachineScaleSet $vmss
}

Create CNAME record in your domain

After the application gateway is created with its public IP address, you can get the DNS address and use it to create a CNAME record in your domain. You can use Get-AzPublicIPAddress to get the DNS address of the application gateway. Copy the fqdn value of the DNSSettings and use it as the value of the CNAME record that you create. Using A-records isn’t recommended because the VIP may change when the application gateway is restarted in the V1 SKU.Azure PowerShellCopyTry It

Get-AzPublicIPAddress -ResourceGroupName myResourceGroupAG -Name myAGPublicIPAddress

Test the application gateway

Enter your domain name into the address bar of your browser. Such as, http://www.contoso.com.

Test contoso site in application gateway

Change the address to your other domain and you should see something like the following example:

Test fabrikam site in application gateway

Clean up resources

When no longer needed, remove the resource group, application gateway, and all related resources using Remove-AzResourceGroup.Azure PowerShellCopyTry It

Remove-AzResourceGroup -Name myResourceGroupAG

Azure – Create, Configure Application Gateway to host multiple web sites using the Azure portal

You can use the Azure portal to configure the hosting of multiple web sites when you create an application gateway. In this tutorial, you define backend address pools using virtual machines. You then configure listeners and rules based on domains that you own to make sure web traffic arrives at the appropriate servers in the pools. This tutorial assumes that you own multiple domains and uses examples of www.contoso.com and www.fabrikam.com.

In this tutorial, you learn how to:

  • Create an application gateway
  • Create virtual machines for backend servers
  • Create backend pools with the backend servers
  • Create backend listeners
  • Create routing rules
  • Create a CNAME record in your domain
Multi-site routing example

If you don’t have an Azure subscription, create a free account before you begin.

Sign in to Azure

Sign in to the Azure portal at https://portal.azure.com

Create an application gateway

  1. Select Create a resource on the left menu of the Azure portal. The New window appears.
  2. Select Networking and then select Application Gateway in the Featured list.

Basics tab

  1. On the Basics tab, enter these values for the following application gateway settings:
    • Resource group: Select myResourceGroupAG for the resource group. If it doesn’t exist, select Create new to create it.
    • Application gateway name: Enter myAppGateway for the name of the application gateway.Create new application gateway: Basics
  2. For Azure to communicate between the resources that you create, it needs a virtual network. You can either create a new virtual network or use an existing one. In this example, you’ll create a new virtual network at the same time that you create the application gateway. Application Gateway instances are created in separate subnets. You create two subnets in this example: one for the application gateway, and another for the backend servers.Under Configure virtual network, select Create new to create a new virtual network . In the Create virtual network window that opens, enter the following values to create the virtual network and two subnets:
    • Name: Enter myVNet for the name of the virtual network.
    • Subnet name (Application Gateway subnet): The Subnets grid will show a subnet named Default. Change the name of this subnet to myAGSubnet.
      The application gateway subnet can contain only application gateways. No other resources are allowed.
    • Subnet name (backend server subnet): In the second row of the Subnets grid, enter myBackendSubnet in the Subnet name column.
    • Address range (backend server subnet): In the second row of the Subnets Grid, enter an address range that doesn’t overlap with the address range of myAGSubnet. For example, if the address range of myAGSubnet is 10.0.0.0/24, enter 10.0.1.0/24 for the address range of myBackendSubnet.
    Select OK to close the Create virtual network window and save the virtual network settings.Create new application gateway: virtual network
  3. On the Basics tab, accept the default values for the other settings and then select Next: Frontends.

Frontends tab

  1. On the Frontends tab, verify Frontend IP address type is set to Public.
    You can configure the Frontend IP to be Public or Private as per your use case. In this example, you’ll choose a Public Frontend IP. NoteFor the Application Gateway v2 SKU, you can only choose Public frontend IP configuration. Private frontend IP configuration is currently not enabled for this v2 SKU.
  2. Choose Create new for the Public IP address and enter myAGPublicIPAddress for the public IP address name, and then select OK.Create new application gateway: frontends
  3. Select Next: Backends.

Backends tab

The backend pool is used to route requests to the backend servers that serve the request. Backend pools can be NICs, virtual machine scale sets, public IPs, internal IPs, fully qualified domain names (FQDN), and multi-tenant back-ends like Azure App Service. In this example, you’ll create an empty backend pool with your application gateway and then add backend targets to the backend pool.

  1. On the Backends tab, select +Add a backend pool.
  2. In the Add a backend pool window that opens, enter the following values to create an empty backend pool:
    • Name: Enter contosoPool for the name of the backend pool.
    • Add backend pool without targets: Select Yes to create a backend pool with no targets. You’ll add backend targets after creating the application gateway.
  3. In the Add a backend pool window, select Add to save the backend pool configuration and return to the Backends tab.
  4. Now add another backend pool called fabrikamPool.Create new application gateway: backends
  5. On the Backends tab, select Next: Configuration.

Configuration tab

On the Configuration tab, you’ll connect the frontend and backend pools you created using a routing rule.

  1. Select Add a rule in the Routing rules column.
  2. In the Add a routing rule window that opens, enter contosoRule for the Rule name.
  3. A routing rule requires a listener. On the Listener tab within the Add a routing rule window, enter the following values for the listener:
    • Listener name: Enter contosoListener for the name of the listener.
    • Frontend IP: Select Public to choose the public IP you created for the frontend.
    Under Additional settings:Accept the default values for the other settings on the Listener tab, then select the Backend targets tab to configure the rest of the routing rule.Create new application gateway: listener
  4. On the Backend targets tab, select contosoPool for the Backend target.
  5. For the HTTP setting, select Create new to create a new HTTP setting. The HTTP setting will determine the behavior of the routing rule. In the Add an HTTP setting window that opens, enter contosoHTTPSetting for the HTTP setting name. Accept the default values for the other settings in the Add an HTTP setting window, then select Add to return to the Add a routing rule window.
  6. On the Add a routing rule window, select Add to save the routing rule and return to the Configuration tab.
  7. Select Add a rule and add a similar rule, listener, backend target, and HTTP setting for Fabrikam.Create new application gateway: routing rule
  8. Select Next: Tags and then Next: Review + create.

Review + create tab

Review the settings on the Review + create tab, and then select Create to create the virtual network, the public IP address, and the application gateway. It may take several minutes for Azure to create the application gateway.

Wait until the deployment finishes successfully before moving on to the next section.

Add backend targets

In this example, you’ll use virtual machines as the target backend. You can either use existing virtual machines or create new ones. You’ll create two virtual machines that Azure uses as backend servers for the application gateway.

To add backend targets, you’ll:

  1. Create two new VMs, contosoVM and fabrikamVM, to be used as backend servers.
  2. Install IIS on the virtual machines to verify that the application gateway was created successfully.
  3. Add the backend servers to the backend pools.

Create a virtual machine

  1. On the Azure portal, select Create a resource. The New window appears.
  2. Select Compute and then select Windows Server 2016 Datacenter in the Popular list. The Create a virtual machine page appears.
    Application Gateway can route traffic to any type of virtual machine used in its backend pool. In this example, you use a Windows Server 2016 Datacenter.
  3. Enter these values in the Basics tab for the following virtual machine settings:
    • Resource group: Select myResourceGroupAG for the resource group name.
    • Virtual machine name: Enter contosoVM for the name of the virtual machine.
    • Username: Enter azureuser for the administrator user name.
    • Password: Enter Azure123456! for the administrator password.
  4. Accept the other defaults and then select Next: Disks.
  5. Accept the Disks tab defaults and then select Next: Networking.
  6. On the Networking tab, verify that myVNet is selected for the Virtual network and the Subnet is set to myBackendSubnet. Accept the other defaults and then select Next: Management.
    Application Gateway can communicate with instances outside of the virtual network that it is in, but you need to ensure there’s IP connectivity.
  7. On the Management tab, set Boot diagnostics to Off. Accept the other defaults and then select Review + create.
  8. On the Review + create tab, review the settings, correct any validation errors, and then select Create.
  9. Wait for the virtual machine creation to complete before continuing.

Install IIS for testing

In this example, you install IIS on the virtual machines only to verify Azure created the application gateway successfully.

  1. Open Azure PowerShell. To do so, select Cloud Shell from the top navigation bar of the Azure portal and then select PowerShell from the drop-down list.Install custom extension
  2. Run the following command to install IIS on the virtual machine:Azure PowerShellCopyTry ItSet-AzVMExtension ` -ResourceGroupName myResourceGroupAG ` -ExtensionName IIS ` -VMName contosoVM ` -Publisher Microsoft.Compute ` -ExtensionType CustomScriptExtension ` -TypeHandlerVersion 1.4 ` -SettingString '{"commandToExecute":"powershell Add-WindowsFeature Web-Server; powershell Add-Content -Path \"C:\\inetpub\\wwwroot\\Default.htm\" -Value $($env:computername)"}' ` -Location EastUS
  3. Create a second virtual machine and install IIS using the steps that you previously completed. Use fabrikamVM for the virtual machine name and for the VMName setting of the Set-AzVMExtension cmdlet.

Add backend servers to backend pools

  1. Select All resources, and then select myAppGateway.
  2. Select Backend pools from the left menu.
  3. Select contosoPool.
  4. Under Targets, select Virtual machine from the drop-down list.
  5. Under VIRTUAL MACHINE and NETWORK INTERFACES, select the contosoVM virtual machine and it’s associated network interface from the drop-down lists.
  6. Add backend servers
  7. Select Save.
  8. Repeat to add the fabrikamVM and interface to the fabrikamPool.

Wait for the deployment to complete before proceeding to the next step.

Create a www A record in your domains

After the application gateway is created with its public IP address, you can get the IP address and use it to create an A record in your domains.

  1. Click All resources, and then click myAGPublicIPAddress.Record application gateway DNS address
  2. Copy the IP address and use it as the value for a new www A record in your domains.

Test the application gateway

  1. Enter your domain name into the address bar of your browser. Such as, http://www.contoso.com.Test contoso site in application gateway
  2. Change the address to your other domain and you should see something like the following example:Test fabrikam site in application gateway

Clean up resources

When you no longer need the resources that you created with the application gateway, remove the resource group. When you remove the resource group, you also remove the application gateway and all its related resources.

To remove the resource group:

  1. On the left menu of the Azure portal, select Resource groups.
  2. On the Resource groups page, search for myResourceGroupAG in the list, then select it.
  3. On the Resource group page, select Delete resource group.
  4. Enter myResourceGroupAG for TYPE THE RESOURCE GROUP NAME and then select Delete.

Working with NSG access and Azure Bastion

When working with Azure Bastion, you can use network security groups (NSGs).

In this diagram:

  • The Bastion host is deployed to the virtual network.
  • The user connects to the Azure portal using any HTML5 browser.
  • The user navigates to the Azure virtual machine to RDP/SSH.
  • Connect Integration – Single-click RDP/SSH session inside the browser
  • No public IP is required on the Azure VM.

Network security groups

This section shows you the network traffic between the user and Azure Bastion, and through to target VMs in your virtual network:

AzureBastionSubnet

Azure Bastion is deployed specifically to AzureBastionSubnet.

  • Ingress Traffic:
    • Ingress Traffic from public internet: The Azure Bastion will create a public IP that needs port 443 enabled on the public IP for ingress traffic. Port 3389/22 are NOT required to be opened on the AzureBastionSubnet.
    • Ingress Traffic from Azure Bastion control plane: For control plane connectivity, enable port 443 inbound from GatewayManager service tag. This enables the control plane, that is, Gateway Manager to be able to talk to Azure Bastion.
  • Egress Traffic:
    • Egress Traffic to target VMs: Azure Bastion will reach the target VMs over private IP. The NSGs need to allow egress traffic to other target VM subnets for port 3389 and 22.
    • Egress Traffic to other public endpoints in Azure: Azure Bastion needs to be able to connect to various public endpoints within Azure (for example, for storing diagnostics logs and metering logs). For this reason, Azure Bastion needs outbound to 443 to AzureCloud service tag.

Target VM Subnet

This is the subnet that contains the target virtual machine that you want to RDP/SSH to.

  • Ingress Traffic from Azure Bastion: Azure Bastion will reach to the target VM over private IP. RDP/SSH ports (ports 3389/22 respectively) need to be opened on the target VM side over private IP. As a best practice, you can add the Azure Bastion Subnet IP address range in this rule to allow only Bastion to be able to open these ports on the target VMs in your target VM subnet.

The overall NSG would look like this: