According to AWS Machine Learning, concurrency sweeps help optimize your generative AI endpoint by identifying the instance type and service configuration that maximizes price/performance within acceptable latency. Without a systematic approach, optimization requires manual deployment, load testing, tweaking, and iteration, which can result in inefficient use of idle GPU budget. If you select too many instances, you can waste your budget, and if you select too few, requests will be queued, latency will increase, and users will experience degraded service. Concurrency sweeps solve this problem. A parallelism sweep is a systematic benchmarking approach that sends controlled, increasing levels of parallel traffic to your Amazon SageMaker AI endpoint and analyzes its performance. Concurrency sweeps are built into Amazon SageMaker AI recommendations, so there is no need to build or maintain custom load testing infrastructure. In this article we will look at deploying the NVIDIA Nemotron-3 Nano 30B model, launching an automated
Bathroom parallelism sweeps and use the results to make informed power decisions. In the end, you'll learn how many parallel requests your endpoint can handle before the latency becomes unacceptable, and how to automate this process. A concurrency sweep sends a controlled number of simultaneous requests to your SageMaker AI endpoint and measures two metrics at each level: throughput (the number of tokens per second your endpoint produces) and latency (the time it takes to process each request). By gradually increasing parallelism (for example, from 64 to 256 to 1024 concurrent requests), you can trace a curve that shows the saturation point of your endpoint. This is the point where adding more concurrent traffic stops improving throughput and worsens latency. A concurrency sweep provides three key metrics for production planning: ideal balance—the level of concurrency that maximizes throughput while allowing latency; breaking point - the level at which latency exceeds your service level agreement (SLA) threshold; fact
p optimization - the number of instances required to cover peak traffic, taking into account the capacity per instance. The parallelism sweep workflow consists of four steps: deploying the model to a SageMaker AI endpoint using the vLLM container, setting up a load profile (number of input and output tokens, streaming mode), running a parallelism sweep using the CreateAIBenchmarkJob API, and analyzing the results to determine the optimal level of parallelism and optimize your instance pool. Before you begin, make sure you have an AWS account with access to Amazon SageMaker AI and an IAM role with rights for SageMaker AI and Amazon Simple Storage Service (Amazon S3). After deploying the model, the endpoint configuration, and the endpoint itself, we check the deployment using validation before moving on to benchmarking. Setting up a load profile requires determining what traffic will be simulated. We create a workload profile that reflects a realistic generative AI output pattern using CreateAIWorkloadConfig. After determining the load profile, we can start the sweep. Creating a benchmark job using the CreateAIBenchmarkJob API allows us to specify a pair
meters for sweep. The benchmark engine (AIPerf) runs each level of parallelism sequentially within a single job, sending a total number of requests defined for each level. Once the job completes, the results are written to Amazon S3 in a tarball format containing tier-by-tier metrics in JSON format. Once the job is completed, we can download and analyze the results. Four metrics are most important when reading the results: throughput (output tokens per second), p99 latency, latency spread from p50 to p99, and time to first token (TTFT), which is critical for streaming user interfaces. When you plot throughput versus latency at different levels of parallelism, the saturation point is visible as a "knee" in the curve: throughput levels off while p99 latency increases sharply. With the benchmarking completed, we can clean up the resources created during the process. To avoid recurring costs, delete your SageMaker endpoints, endpoint configurations, and models after testing. Concurrency sweeps replace guesswork with data in the capacity planning process for generative AI endpoints. Instead of redundant
Whether you're deploying accurately or identifying bottlenecks in production, you can systematically evaluate the performance of your endpoint and make informed decisions about instance pool size before the first user request arrives. The material was prepared by a neural network based on the original publication. AI may make mistakes in details (numbers, names, dates, quotes) - for important information, check with the original source. Empowering the Public Sector with Agentforce and AWS Trane Improves HVAC Diagnostics by 60x with Amazon Bedrock AWS Machine Learning Unveils New Agent Skills Assessment System Using Strands Evals and Amazon Bedrock. AWS Machine Learning shows how to combine data automation with Agentforce. AI infrastructure may be more valuable than the AI applications themselves.
