#C++engine

[govindhtech]

Watsonx.data Presto C++ With Intel Sapphire Rapids On AWS

Using Watsonx.data Presto C++ with the Intel Sapphire Rapid Processor on AWS to speed up query performance

Over the past 25 years, there have been notable improvements in database speed due to IBM and Intel’s long-standing cooperation. The most recent generation of Intel Xeon Scalable processors, when paired with Intel software, can potentially improve IBM Watsonx.data performance, according to internal research conducted by IBM.

A hybrid, managed data lake house, IBM Watsonx.data is tailored for workloads including data, analytics, and artificial intelligence. Using engines like Presto and Spark to drive corporate analytics is one of the highlights. Watsonx.data also offers a single view of your data across hybrid cloud environments and a customizable approach.

Presto C++

The next edition of Presto, called Presto C++, was released by IBM in June. It was created by open-source community members from Meta, IBM, Uber, and other companies. The Velox, an open-source C++ native acceleration library made to be compatible with various compute engines, was used in the development of this query engine in partnership with Intel. In order to further improve query performance through efficient query rewrite, IBM also accompanied the release of the Presto C++ engine with a query optimizer built on decades of experience.

Summary

A C++ drop-in replacement for Presto workers built on the Velox library, Presto C++ is also known as the development name Prestissimo. It uses the Proxygen C++ HTTP framework to implement the same RESTful endpoints as Java workers. Presto C++ does not use JNI and does not require a JVM on worker nodes because it exclusively uses REST endpoints for communication with the Java coordinator and amongst workers.

Inspiration and Goals

Presto wants to be the best data lake system available. The native Java-based version of the Presto evaluation engine is being replaced by a new C++ implementation using Velox in order to accomplish this goal.

In order to allow the Presto community to concentrate on more features and improved connectivity with table formats and other data warehousing systems, the evaluation engine has been moved to a library.

Accepted Use Cases

The Presto C++ evaluation engine supports just certain connectors.

Reads and writes via the Hive connection, including CTAS, are supported.

Only reads are supported for iceberg tables.

Both V1 and V2 tables, including tables with delete files, are supported by the Iceberg connector.

TPCH.naming=standard catalog property for the TPCH connector.

Features of Presto C++

Task management: Users can monitor and manage tasks using the HTTP endpoints included in Presto C++. This tool facilitates tracking ongoing procedures and improves operational oversight.

Data processing across a network of nodes can be made more effective by enabling the execution of functions on distant nodes, which improves scalability and distributed processing capabilities.

For secure internal communication between nodes, authentication makes use of JSON Web Tokens (JWT), guaranteeing that data is safe and impenetrable while being transmitted.

Asynchronous data caching with prefetching capabilities is implemented. By anticipating data demands and caching it beforehand, this maximizes processing speed and data retrieval.

Performance Tuning: Provides a range of session parameters, such as compression and spill threshold adjustments, for performance tuning. This guarantees optimal performance of data processing operations by enabling users to adjust performance parameters in accordance with their unique requirements.

Limitations of Presto C++

There are some drawbacks to the C++ evaluation engine:

Not every built-in function is available in C++. A query failure occurs when an attempt is made to use unimplemented functions. See Function Coverage for a list of supported functions.

C++ does not implement all built-in types. A query failure will occur if unimplemented types are attempted to be used.

With the exception of CHAR, TIME, and TIME WITH TIMEZONE, all basic and structured types in Data Types are supported. VARCHAR, TIMESTAMP, and TIMESTAMP WITH TIMEZONE are subsumptions of these.

The length n in varchar[n] is not honored by Presto C++; it only supports the limitless length VARCHAR.

IPADDRESS, IPPREFIX, UUID, KHYPERLOGLOG, P4HYPERLOGLOG, QDIGEST, TDIGEST, GEOMETRY, and BINGTILE are among the types that are not supported.

The C++ evaluation engine does not use all of the plugin SPI. Specifically, several plugin types are either fully or partially unsupported, and C++ workers will not load any plugins from the plugins directory.

The C++ evaluation engine does not support PageSourceProvider, RecordSetProvider, or PageSinkProvider.

Block encodings, parametric types, functions, and types specified by the user are not supported.

At the split level, the event listener plugin is not functional.

See Remote Function Execution for information on how user-defined functions differ from one another.

The C++ evaluation engine has a distinct memory management system. Specifically:

There is no support for the OOM killer.

There is no support for the reserved pool.

Generally speaking, queries may utilize more memory than memory arbitration permits. Refer to Memory Management.

Functions

reduce_agg

Reduce_agg is not allowed to return null in the inputFunction or the combineFunction of C++-based Presto. This is acceptable but ill-defined behavior in Presto (Java). See reduce_agg for more details about reduce_agg in Presto.

Amazon Elastic Compute Cloud (EC2) R7iz instances are high-performance CPU instances that are designed for memory. With a sustained all-core turbo frequency of 3.9 GHz, they are the fastest 4th Generation Intel Xeon Scalable-based (Sapphire Rapids) instances available in the cloud. R7iz instances can lower the total cost of ownership (TCO) and provide performance improvements of up to 20% over Z1d instances of the preceding generation. They come with integrated accelerators such as Intel Advanced Matrix Extensions (Intel AMX), which provide a much-needed substitute for clients with increasing demands for AI workloads.

R7iz instances are well-suited for front-end Electronic Design Automation (EDA), relational database workloads with high per-core licensing prices, and workloads including financial, actuarial, and data analytics simulations due to their high CPU performance and large memory footprint.

IBM and Intel have collaborated extensively to offer open-source software optimizations to Watsonx.data, Presto, and Presto C++. In addition to the hardware enhancements, Intel 4th Gen Xeon has produced positive Watsonx.data outcomes.

Based on publicly available 100TB TPC-DS Query benchmarks, IBM Watsonx.data with Presto C++ v0.286 and query optimizer on AWS ROSA, running on Intel processors (4th generation), demonstrated superior price performance over Databrick’s Photon engine, with better query runtime at comparable cost.

Read more on Govindhtech.com

[booart5]

Reblogging this drawing with the Timelapse because I finally figured out how to add those and I think it’s cool

Took me about 60 hours which is CRAZY,, I call this one The Quark Family when I’m talking about it (which seems apt) and I’ve literally crammed so many references and details and hours into this my life feels so empty without it. Tumblr is butchering the image quality but that’s just what it is ig,,, I’ve tried and I can’t seem to fix it

Progress shots and closeups under the cut (and a full list of the references and stuff I crammed in in the tags bc I have to know that they’re all noticed)

[fizzy-dizz]

All Aboard The Starlight Express 🚂💫🪐

[nostalgebraist]

Trying to picture how myself from early 2015 would react if I told him that 10 years later, he'd be freaking out and doomscrolling the news because

"Elon Musk and President Donald Trump, acting through the new federal organization 'DOGE,' are trying to implement Mencius Moldbug's plan to 'reboot the government' by mass-retiring government employees"

[wolfram-but-art]

a comic i made a couple days ago

[carsthatnevermadeitetc]

Aston Martin Jaguar Type C Roadster, 1959. The Aston Martin Type C was a racing car fielded by the company from 1938 and produced in limited numbers. This one-off Type C was repowered with a Jaguar XK150 210 bhp, 3.4-litre straight 6 engine

[yes-no-maybe-soo]

To All Love and Deepspace Hunters

Happy 1 year anniversary to loml 🫶🏻 🐦‍⬛exactly 1 year ago today Love and Deepspace released this teaser, which was Sylus first ever official appearance. Before that fans could only speculate what he would look like (some believed him to be the guy at the Nest in the end chapter of Under Deepspace)... or look at the leaks. The latter was what prompted the Devs to release this teaser ahead of schedule. Mind you, only the teaser/trailer was dropped prematurely not LAR or Sylus as an LI; they were always planned to be released in July of 2024.

Anyway, just wanted to remind y'all that 1 year ago today the world got its first official glimpse of the most gorgeous man inside and out ever created ❤️. (Not me though. My ass didn't see him until October 🥲💔 the downside of barely being on social media at all...)

[leclercskiesahead]

Rained in and can’t go out for quali therefore become kids playing in the school computer lab

[letsraisealittlehelltrds]

Whistling's okay, if it’s Pearl and Rusty! <3

Kayna Montecillo and Jeevan Braich as Pearl and Rusty (Starlight Express - Troubadour Wembley Theatre)

🎥 : @mttztrading

2nd Pearl and Rusty set recently…. I’m okay i swear-

[cheriecelestial]

Jacob Black’s Self Saving System and His Shenanigans™️

[venusiinfurss]

i truly think sunny is going to end with either macdennis kiss or macdennis full penetration (no in-between) and then dennis has an existential crisis walks outside gets hit by a car and dies. cut to black end of series

[lynnemariehedvig]

TECHNOLOGIES

United States of America

[bleue-flora]

Ok, I recently wrote an essay [here] talking about the definition and duties of civil engineering as well as the ethics because of the brain rot @swordfright gave me with calling Dream Sam’s ultimate engineering project. So, because I actually am a civil engineer I took it upon myself to design the title and summary of quantities sheets just like I do at work for roads but with Dream as the project instead. And in honor of angst day sponsored by @sixteenth-day-event, I figured I’d share it because I feel like it kinda works for the prison of the mind prompt.

“Sam’s “ultimate engineering project” he deemed too damaged like a bumpy road or crumbling building that wasn’t worthy of patching and filling in the cracks or reinforcing, that’s too eroded to be fixed and preserved. So, Sam strived to tear him down to the bedrock so he could remake, remold, and reengineer Dream according to his design for the common safety, public health and well-fair.”

{These are very similar to the actual sheets I make day to day, which I shall not share for the sake of doxing my location, but yea pretty much everything has a significance. Some of it doesn’t necessarily make sense but that was because I was more so taking inventory of what we see in lore (so you know I counted ;) lol)}

[wackywatchdotcom]

sometimes i think about the idea that the digital circus is JUST caine. its all what hes made. i mean he named himself, whose to say the circus isnt his own creation, an extension of himself

[nvoc]

[shih-coulda-had-it]

i could make a one for all racing team AU. Formula 1 but not Formula 1… Kudou is team principal; Bruce is in the mechanics’ garage with Hikage; Yoichi is bankrolling… Banjo is En’s race engineer, Nana was racing but exited just in time to slot her protégé Toshinori up from the One for All racing academy… the AU would absolutely smash the timeline together but that would make it so much funnier.