How to optimize Microsoft Azure services usage

    If you are building services on Microsoft Azure technology, optimizing usage of the Azure platform is an important step for effective cost control, performance scalability and resource usage optimization.

    Azure Analysis Services (SSAS) is a fully managed platform as a service (PaaS) that provides enterprise-grade data models in the cloud. Use advanced mashup and modeling features to combine data from multiple data sources, define metrics, and secure your data in a single, trusted tabular semantic data model. The data model provides an easier and faster way for users to perform ad hoc data analysis using tools like Power BI and Excel.

    Azure Analysis Services is available in DeveloperBasic, and Standard tiers. Within each tier, plan costs vary according to processing power, Query Processing Units (QPUs), and memory size. When you create a server, you select a plan within a tier. You can change plans up or down within the same tier, or upgrade to a higher tier, but you can’t downgrade from a higher tier to a lower tier.

    Scale up, down, or pause your server. The total cost depends on a number of factors. For example, your chosen region, tier, query replicas, and pause/resume. You only pay for what you use.

    The main concept to keep the cost as low as you can, is the following:

    1. Implement a background job which monitors the SSAS usage
      1. If there is no usage, then scale down to the lowest in the current tier
      2. Pause the server to stop money burning
    2. Implement a trigger logic which starts the SSAS before first usage if it’s in paused status.

    Azure SQL Database with serverless model

    Azure SQL Database is a fully managed platform as a service (PaaS) database engine that handles most of the database management functions such as upgrading, patching, backups, and monitoring without user involvement. Azure SQL Database is always running on the latest stable version of the SQL Server database engine and patched OS with 99.99% availability

    SQL Database enables you to easily define and scale performance within two different purchasing models: a vCore-based purchasing model and a DTU-based purchasing model.

    The serverless model automatically scales compute based on workload demand, and bills for the amount of compute used per second. The serverless compute tier also automatically pauses databases during inactive periods when only storage is billed, and automatically resumes databases when activity returns.

    The serverless compute tier for single databases in Azure SQL Database is parameterized by a compute autoscaling range and an auto-pause delay. The configuration of these parameters shapes the database performance experience and compute cost.

    This is a good option when you need database but not for the whole day and during inactivity you do not want to be billed.
    What you need to pay attention to when implementing a retry logic for Database operations is to avoid the application interruption until it waits for the database startup.

    Azure Container Registry(ACR)

    When using Azure ML without custom image the execution will prepare a Docker image and upload it to the Azure Container Registry (ACR), then use this image for the current experiment processing. If you generate a lot of runs, the Basic 10GiB or Standard 100GiB storage will run out of space in the ACR and because billing logic – which counts the number of images as well – it can generate unwanted billing.

    To avoid the space problem and billing problem you can do the following:

    1. Delete the unused images manually or in a scheduled way.
    2. Build and use custom images for as many cases as you can

    Azure ML

    Azure Machine Learning can be used for any kind of machine learning, from classical ML to deep learning, supervised, and unsupervised learning. Whether you prefer to write Python or R code with the SDK or work with no-code/low-code options in the studio, you can build, train, and track machine learning and deep-learning models in an Azure Machine Learning Workspace.

    Automated machine learning, also referred to as automated ML or AutoML, is the process of automating the time consuming, iterative tasks of machine learning model development. It allows data scientists, analysts, and developers to build ML models with high scale, efficiency, and productivity, all while sustaining model quality.
    Traditional machine learning model development is resource-intensive, requiring significant domain knowledge and time to produce and compare dozens of models. With automated machine learning, you’ll accelerate the time it takes to get production-ready ML models with great ease and efficiency.

    When you are familiar with Azure ML and know the tasks well, which you can implement, then it is recommended to build a custom Docker image. Within this image, you can manage the data preprocessing, model finding, registered model usage used by the Python programming language.
    In many cases one Azure Container Instance(ACI) – 16GiB memory, 4CPU – is enough to run your custom image which contains full business flow to decrease the costs.

     

    Leave a Reply