#Intel MLPerf

[govindhtech]

Intel MLPerf: Benchmarking Hardware For Machine Learning(ML)

Overview

This briefing describes Intel MLPerf, a popular and rapidly growing benchmark suite for machine learning (ML) hardware, software, and services. Intel MLPerf, formed by a wide coalition of academic, scientific, and industry organisations, compares ML systems impartially to accelerate innovation. MLPerf's definition, operation, aims, and relevance in artificial intelligence will be discussed in this article.

What's MLPerf?

When combined, “ML” for machine learning and “Perf” for performance create “MLPerf.” MLPerf is a series of benchmarks that evaluates ML systems in different tasks and conditions.

Intel MLPerf, an industry benchmark, measures ML hardware and software performance. It standardises machine learning system evaluation and progress tracking.

MLPerf emphasises real-world application settings rather than vendor-specific criteria to level the playing field for machine learning performance assessment. Developers, researchers, and consumers may pick the best hardware and software for their machine learning needs.

How MLPerf Works

MLPerf's rigorous and transparent process involves several key elements:

Benchmark Suites: Intel MLPerf has several benchmark suites for specific ML problems. Over time, these suites evolve with the field. Edge computing, inference, and training examples are given.

Machine learning concerns including recommendation systems, object recognition, picture classification, and NLP are addressed.

Open Participation: The Intel MLPerf collaboration welcomes cloud service providers, software developers, hardware manufacturers, and educational organisations. This coordinated approach ensures benchmark applicability and credibility.

Standardised Rules and indicators: MLPerf sets strict benchmarking standards and performance metrics to ensure fair comparisons. The rules cover allowed optimisations, model accuracy targets, and data preparation.

Benchmarks include strict requirements to provide fair system comparisons.

After participants submit their performance results, the MLPerf website posts complete software stacks and system specifications for public review. This transparency encourages healthy rivalry and clear comparisons. Leaderboards are crucial for tracking progress:

Users may see how different systems perform on different machine learning workloads because the findings are publically available.

Focus on Practical Tasks: Intel MLPerf benchmarks simulate genuine ML applications using representative or public datasets. It ensures that performance indicators apply to real-world use cases.

The Value of MLPerf

The Intel article emphasises many aims and MLPerf's role in the AI ecosystem:

Objective Comparisons: MLPerf simplifies machine learning system comparisons by standardising methods and metrics. This lets customers make data-driven choices.

MLPerf sets defined performance objectives and makes hardware and software innovations public to motivate vendors to improve results. Competition increases growth.

Open submission and comprehensive reporting standards make ML performance claims transparent. Users may view software stacks and settings used to achieve goals.

Influencing Purchase Decisions: Intel MLPerf findings assist organisations adopt ML solutions by revealing the performance capabilities of different hardware and software alternatives for specific workloads.

Monitoring Development in the Field: MLPerf results indicate how new algorithms, software optimisations, and architectural upgrades affect ML system performance over time.

It tracks ML technology advancement.

MLPerf benchmarks training and inference at many levels of ML. This provides a complete system performance view.

The Changes and Impact of MLPerf

Remember that MLPerf is a dynamic project that extends beyond description and operation.

New ML tasks, models, and application areas are introduced to benchmark suites often to keep current. Long-term effects depend on adaptability.

The quest for MLPerf benchmark perfection affects hardware and software design, including CPUs, GPUs, memory systems, interconnects, and software frameworks. To meet these standards, companies actively optimise their products.

Community-Driven Development: MLPerf's strength is its community participation. The consortium's transparent and cooperative structure ensures that benchmarks reflect machine learning community concerns.

Addressing Emerging Trends: MLPerf is assessing edge computing, personalised recommendation systems, and massive language models to keep up with AI application changes.

In conclusion

The primary machine learning system effectiveness benchmark is Intel MLPerf. A standardised, transparent, and community-driven evaluation strategy empowers users, stimulates innovation, and facilitates informed decision-making in the fast-growing field of artificial intelligence. MLPerf's development and use are crucial for tracking progress and understanding AI technology potential.

[ourwitching]

NVIDIA's MLPerf Training V4.0 is out. It is mostly NVIDIA H100 and H200 so if you are looking to com...

[hackernewsrobot]

MLPerf training tests put Nvidia ahead, Intel close, and Google well behind

https://spectrum.ieee.org/generative-ai-training

[ai-news]

Intel, Nvidia and Google have made significant strides in recent months that enable faster LLM training results in the MLPerf Training 3.1 benchmarks. #AI #ML #Automation

[strategictech]

MLPerf Releases Latest Inference Results and New Storage Benchmark

MLCommons this week issued the results of its latest MLPerf Inference (v3.1) benchmark exercise. Nvidia was again the top performing accelerator, but Intel (Xeon CPU) and Habana (Gaudi1 and 2) performed well. Google provided a peak at its new TPU (v5e) performance. MLCommons also debuted a new MLPerf Storage (v0.5) benchmark intended to measure storage performance under ML training workloads. Submitters in the first Storage run included: Argonne National Laboratory (ANL), DDN, Micron, Nutanix, and Weka.

@tonyshan #techinnovation https://twitter.com/intent/follow?screen_name=tonyshan https://www.linkedin.com/comm/mynetwork/discovery-see-all?usecase=PEOPLE_FOLLOWS&followMember=tonyshan

[dailystraitsdotcom]

Intel's AI Shines Competitively

Tech: Intel's AI Shines Competitively.

In the latest MLPerf Training 3.0 benchmark, both Habana® Gaudi®2 deep learning accelerator and 4th Gen Intel® Xeon® Scalable processor showcased outstanding training results, as announced by MLCommons. These results challenge the prevailing industry narrative that generative AI and large language models (LLMs) can only run on Nvidia GPUs. Intel’s AI solutions offer competitive alternatives,…

[dailytechnologynews]

MLPerf 0.7 Training Results When Winners Only Apply | Nvidia, Google, Intel, Huawei | ServeTheHome https://ift.tt/2X6HTwT

[govindhtech]

MLPerf Inference v4.1 For AMD Instinct MI300X Accelerators

Engineering Insights: Introducing AMD Instinct MI300X Accelerators’ MLPerf Results. The full-stack AMD inference platform demonstrated its prowess with the remarkable results AMD Instinct MI300X GPUs, powered by one of the most recent iterations of open-source ROCm, obtained in the MLPerf Inference v4.1 round.

LLaMA2-70B

The first submission concentrated on the well-known LLaMA2-70B type, which is renowned for its excellent performance and adaptability. By outperforming the NVIDIA H100 in Gen  AI inference, it established a high standard for what AMD Instinct MI300X accelerators are capable of.

MLPerf Inference

Comprehending MLPerf and Its Relevance to the Industry

Efficient and economical performance is becoming more and more important for inference and training as large language models (LLMs) continue to grow in size and complexity. Robust parallel processing and an optimal software stack are necessary to achieve high-performance LLMs.

This is where the best benchmarking package in the business, MLPerf, comes into play. The open-source AI benchmarks known as MLPerf Inference, which were created by the cross-industry cooperation MLCommons, of which AMD is a founding member, include Gen AI, LLMs, and other models that give exacting, peer-reviewed criteria. Businesses are able to assess the efficacy of AI technology and software by using these benchmarks.

A major accomplishment for AMD, excelling in MLPerf Inference v4.1 demonstrates their dedication to openness and providing standardized data that enables businesses to make wise choices.

An Extensive Analysis of the LLaMA2-70B Benchmark

The AMD LLaMA2-70B model was utilized in their first MLPerf Inference. A major development in LLMs, the LLaMA2-70B model is essential for practical uses such as large-scale inference and natural language processing. A Q&A scenario using 24,576 samples from the OpenORCA dataset, each with up to 1,024 input and output tokens, was included in the MLPerf benchmarking test. Two situations were analyzed by the benchmark to assess inference performance:

In an offline scenario, queries are processed in batches to increase throughput in tokens per second.

 Server Scenario: This model tests the hardware’s capacity to provide quick, responsive performance for low-latency workloads by simulating real-time queries with stringent latency limitations (TTFT* < 2s, TPOT* ≤ 200ms).

Performance of AMD Instinct MI300X in MLPerf

With four important entries for the LLaMA2-70B model, the AMD Instinct MI300X demonstrated remarkable performance in its first MLPerf Inference utilizing the Supermicro AS-8125GS-TNMR2 machine. These findings are especially noteworthy since they provide an apples-to-apples comparison with rival AI accelerators, are repeatable, vetted by peer review, and grounded in use cases that are relevant to the industry.

Combination Performance of CPU and GPU

Submission ID 4.1-0002: Two AMD EPYC 9374F (Genoa) CPUs paired with eight AMD Instinct MI300X accelerators in the Available category.

This setup demonstrated the potent synergy between 4th Gen EPYC CPUs (previously codenamed “Genoa”) and AMD Instinct MI300X GPU accelerators for AI workloads, providing performance within 2-3% of NVIDIA DGX H100 with 4th Gen Intel Xeon CPUs in both server and offline environments at FP8 precision.

Previewing Next-Generation CPU Performance

Submission ID 4.1-0070: Two AMD EPYC “Turin” CPUs and eight AMD Instinct MI300X CPUs in the Preview category.

It showcased the performance increases from the next AMD EPYC “Turin” 5th generation CPU when paired with AMD Instinct MI300X GPU accelerators. In the server scenario, it outperformed the NVIDIA DGX H100 with Intel Xeon by a small margin, and it maintained a similar level of performance even offline at FP8 precision.

LLaMA2-70B GPU

Efficiency of a Single GPU

Submission ID 4.1-0001: In the Available category, AMD Instinct MI300X accelerator with AMD EPYC 9374F 4th Gen CPUs (Genoa).

This submission emphasized the AMD Instinct MI300X’s enormous 192 GB memory, which allowed a single GPU to effectively execute the whole LLaMA2-70B model without requiring the network cost that comes with dividing the model over many GPUs at FP8 precision.

The AMD Instinct MI300X has 192 GB of HBM3 memory and a peak memory bandwidth of 5.3 TB/s thanks to its AMD CDNA 3 architecture. The AMD Instinct MI300X can execute and host a whole 70 billion parameter model, such as LLaMA2-70B, on a single GPU with ease because to its large capacity.

The findings in Figure 2 show that the scaling efficiency with the ROCm software stack is almost linear from 1x AMD Instinct MI300X (TP1) to 8x AMD Instinct MI300X (8x TP1), indicating that AMD Instinct MI300X can handle the biggest MLPerf inference model to date.

Outstanding Dell Server Architecture Outcomes Using AMD Instinct MI300X Processors

Submission ID 4.1-0022: Two Intel Xeon Platinum 8460Y+ processors and eight AMD Instinct MI300X accelerators in the Available category.

Along with AMD submissions, Dell used their PowerEdge XE9680 server and LLaMA2-70B to submit their findings, validating the platform-level performance of AMD Instinct accelerators on an 8x AMD Instinct MI300X arrangement. This proposal demonstrates their collaboration and emphasizes how strong it ecosystem is, making them a great option for deployments including both data centers and edge inference. Further information on such outcomes is available here.

Performance Of Engineering Insights

The AMD Instinct MI300X accelerators exhibit great competitive performance due to their high computational power, huge memory capacity with rapid bandwidth, and optimized ROCm software stack. The latter enables effective processing of large  AI models such as LLaMA2-70B. A few important elements were pivotal:

Big GPU Memory Capacity

The AMD Instinct MI300X has the most GPU memory that is currently on the market, which enables the whole LLaMA2-70B model to fit into memory while still supporting KV cache. By avoiding model splitting among GPUs, this maximizes inference speed while avoiding network cost.

Batch Sizes: They set the max_num_seqs parameter to 2048 in the offline scenario to optimize throughput, and to 768 in the server scenario to achieve latency requirements. These values are much greater than the 256 default value used in vLLM.

Effective KV cache management is made possible by the vLLM’s paged attention support, which helps prevent memory fragmentation brought on by huge memory AMD Instinct MI300X accelerators.

FP8 Precision

AMD expanded support for the FP8 numerical format throughout the whole inference software stack, using the AMD Instinct MI300X accelerator hardware. They quantized the LLaMA2-70B model weights to FP8 using Quark while maintaining the 99.9% accuracy needed by MLPerf. To further improve speed, it improved the hipBLASLt library, introduced FP8 support to vLLM, and implemented FP8 KV caching.

Software Enhancements

Kernel Optimization: AMD Composable Kernels (CK) based prefill attention, FP8 decode paged attention, and fused kernels such residual add RMS Norm, SwiGLU with FP8 output scaling were among the many profiles and optimizations to carried out.

vLLM Enhancements: The scheduler was improved to optimize both offline and server use cases, allowing for quicker decoding scheduling and better prefill batching.

CPU Enhancement

While GPUs handle the majority of the AI task processing, CPU speed is still quite important. CPUs with fewer cores and higher peak frequencies such as the 32-core EPYC 9374F offer the best performance, particularly in server applications. Performance improvements over the 4th generation EPYC CPUs which were submitted as a preview were seen during testing with the forthcoming “Turin” generation of EPYC CPUs.

LLaMa 3.1 405B

Establishing a Standard for the Biggest Model

The AMD Instinct MI300X GPU accelerators have shown their performance in MLPerf Inference with LLaMA2-70B, and the positive outcomes set a solid precedent for their future efficacy with even bigger models, such as Llama 3.1. They are pleased to provide Day 0 support for AMD Instinct MI300X accelerators with Meta’s new LLaMa 3.1 405B parameter model.

Only a server driven by eight AMD Instinct MI300X GPU accelerators can fit the whole LLaMa 3.1 model, with 405 billion parameters, on a single server utilizing FP16 datatype MI300-7A, owing to the industry-leading memory capacities of the AMD Instinct MI300X platform MI300-25. This lowers expenses and lowers server use. The most ideal way to power the biggest open models on the market right now is with AMD Instinct MI300X accelerators.

Read more on govindhtech.com

[ourwitching]

NVIDIA's MLPerf Training V4.0 is out. It is mostly NVIDIA H100 and H200 so if you are looking to com...

[whats-new-guys-blog]

The first benchmark results from the MLPerf consortium have been released and Nvidia is a clear winner for inference performance.

For those unaware, inference takes a deep learning model and processes incoming data however it’s been trained to.

MLPerf is a consortium which aims to provide “fair and useful” standardised benchmarks for inference performance. MLPerf can be thought of as doing for inference what SPEC does for benchmarking CPUs and general system performance.

The consortium has released its first benchmarking results, a painstaking effort involving over 30 companies and over 200 engineers and practitioners. MLPerf’s first call for submissions led to over 600 measurements spanning 14 companies and 44 systems.

However, for datacentre inference, only four of the processors are commercially-available:

Intel Xeon P9282

Habana Goya

Google TPUv3

Nvidia Turing

[componentplanet]

Nvidia in Advanced Talks to Buy ARM, Upend Silicon Industry

Those rumors about Nvidia being in talks with SoftBank about purchasing ARM have been upgraded to “advanced talks.” (Does that make these “advanced rumors?”)

Even if SoftBank can come to an agreement with Nvidia over selling ARM, which it bought for $32B, the regulatory scrutiny from various nations would be enormous, as Bloomberg reports. Apple, Qualcomm, AMD, and Intel all have architecture licenses from ARM, allowing them to design their own CPUs that are compatible with ARM’s instruction sets but that otherwise contain custom IP. Dozens more companies depend on ARM’s extensive hard-IP licenses for various CPU solutions. Given ARM’s ubiquitous position in smartphones, and its burgeoning presence in HPC and servers, everyone from Ampere to MediaTek is going to be concerned about ARM being owned by any single silicon company.

What’s the Advantage of Ownership?

In my previous story, I stated that buying ARM would give Nvidia an easy path to return to desktop and laptop computing with an integrated ARM/Nvidia SoC. What I should’ve addressed then — and didn’t — is how this would be different from Nvidia taking out an architectural license (which it already has), in the first place. After all, Nvidia already builds chips like Project Denver and its successor, Carmel, on an ARM architecture. Owning ARM doesn’t change that.

What owning ARM would do is give Nvidia control over how the entire ARM IP stack evolves in the future. If it wanted to pour development into ARM’s Neoverse server concept and develop new SIMD extensions that would speed its own HPC workloads, it could do so. Instead of being limited to an Nvidia-specific implementation, ARM could design said extensions directly into the standard.

Running multiple Docker container-based demos on Nvidia Jetson Xavier NX.

There are other potential advantages for Nvidia as well. The company could design a low-level GPU as a replacement for ARM’s own efforts, then extend the IP across its core families as well, giving the GeForce brand significant reach across the mobile ecosystem.

Regulatory issues, however, could still scuttle the deal. Historically, Nvidia has always preferred a very closed development model. The company doesn’t license CUDA to anyone and it typically prefers to develop its own value-added software and hardware capabilities as opposed to creating cross-vendor ecosystems. So long as Nvidia is just one ARM licensee among many, this presents no problem. If Nvidia were to buy ARM itself, however, the numerous firms that rely on ARM licenses would demand guarantees that their access to future products or licenses wouldn’t be impeded by anti-competitive measures. If the deal gets to this point, Nvidia will undoubtedly make a number of concessions and guarantees to avoid the appearance of favoritism.

What Nvidia would be buying, with ARM, isn’t just the ability to take out an architectural license. It has one already. What it would be buying, ultimately, is the ability to influence how ARM SoCs evolve in the future at multiple price points and markets. If Nvidia thought it would be useful to their own position to implement CUDA for mobile GPUs, they’d be able to do so. If they wanted to introduce a high-end hard-IP GPU core under the GeForce brand and position the SoC as a gaming solution, they could do that as well.

Just How Shelved Is AMD K12?

One thing I’d love to know is just how far AMD got with K12 before they shelved it and whether the chip might ever see the light of day. According to AMD contacts I spoke to when the company decided to pivot towards Ryzen, the K12 design wasn’t scrapped — AMD just decided that the ecosystem wasn’t mature enough to justify bringing the product to market. The scuttlebutt around K12 always suggested it was similar to Ryzen, with a number of shared design elements between the cores. While ARM and x86 are two different CPU architectures, it would be much easier to cross-leverage IP between ARM and x86 then between, say, x86 and Itanium. There’s no evidence that AMD finished the design or continued to evolve it in the background, but they wouldn’t have thrown the chip away, either. If ARM starts chewing into x86’s market share, I expect AMD might dust off K12, update it for the modern era, and bring it to market.

AMD’s K12 slide. This is most of what we know about the one-time product. AMD has never said how much of the work it completed before shelving the CPU.

Right now, the CPU market is more dynamic than it’s been in decades. A new ARM owner could send major ripples through the company’s long-term trajectory. Intel is struggling with manufacturing issues. AMD is gaining market share. Heck, even open-source efforts like RISC-V continue to drive engagement and interest. Any Nvidia effort to buy ARM can likely be read as an intention to push into x86’s turf in one market or another.

Feature image is Nvidia’s Orin, a self-driving car module with onboard ARM cores and an Ampere-based GPU.

Now Read:

Nvidia Crushes New MLPerf Tests, but Google’s Future Looks Promising

Nvidia Could Bring Ampere to Gamers for Just $5 a Month

x86 Beware: Nvidia May Be Eyeing an ARM Takeover From Soft Bank

from ExtremeTechExtremeTech https://www.extremetech.com/computing/313405-nvidia-in-advanced-talks-to-buy-arm-upend-silicon-industry from Blogger http://componentplanet.blogspot.com/2020/07/nvidia-in-advanced-talks-to-buy-arm.html

[strategictech]

Intel and Nvidia Square Off in GPT-3 Time Trials

MLPerf provides LLM testbed for Nvidia’s H100 and top Intel chipsets 

[tech-battery]

Centaur Creates First x86 SoC with Integrated AI Co-Processor

People typically think of x86 processors as coming from Intel and AMD, but there is a third architectural license holder: VIA. This week, an Austin, Texas-based subsidiary of the Taipei-headquartered company announced it's demonstrating an x86 processor that comes with an integrated artificial intelligence (AI) co-processor.

VIA's Centaur Technology is a small CPU design company. The unnamed processor it's developing is built on the 16nm fabrication process and manufactured by TSMC. It's a complete system-on-chip (SoC) with eight cores, 16MB of L3cache and am AI co-processor. In total, it has a die size of 195 square millimeters, which isn’t all that big. For comparison, a Ryzen 5 chip with one CCD and one IOD has a die measuring 199 square millimeters.

Centaur’s new chip isn't meant to land in consumer PCs. Rather, the end goal is to land in enterprise systems aimed at deep learning and other industrial applications.

Centaur is developing this processor to tackle the challenge of x86 processors needing external inference acceleration (such as a GPU with Nvidia’s Tensor cores). It wants to integrate this feature into one chip and, consequently, reduce power consumption for deep learning tasks.

Next to its eight x86 cores, 16MB of L3 cache and 20 TOPS AI co-processor, Centaur’s chip comes with a total of 44 PCIe lanes and four DDR4 memory channels. Therefore, if a user wanted to further improve a supporting system’s inference performance they'd also be able to add GPUs into the mix.

Currently, the reference platform runs at 2.5 GHz. It also comes with the AVX-512 instruction set, which, thus far, has only been implemented in very select few processors.

The 16MB of memory that the AI co-processors also have access to enable them to communicate at up to 20 TBps, which has led to the lowest latency for image classification within just 330 microseconds in the MLPerf benchmark.

“We set out to design an AI co-processor with 50 times the inference performance of a general-purpose CPU. We achieved that goal. Now we are working to enhance the hardware for both high-performance and low-power systems, and we are disclosing some of our technology details to encourage feedback from potential customers and technology partners," Glenn Hendry, Centaur’s Chief Architect of the AI co-processor, said in a statement.

Centaur’s x86 CPU with its AI co-processor isn’t ready for prime time yet but will be demonstrated at ISC East on November 20 and 21, with technical details to be published on December 2.

[lippyawards]

MLPerf Releases First Results From AI Inferencing Benchmark

MLPerf Releases First Results From AI Inferencing Benchmark

AI is everywhere these days. SoC vendors are falling over themselves to bake these capabilities into their products. From Intel and Nvidia at the top of the market to Qualcomm, Google, and Tesla, everyone is talking about building new chips to handle various workloads related to artificial intelligence and machine learning.

While these companies have shown their own products racking up…