#CloudQuery

[cleverhottubmiracle]

[ad_1] Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more Gaining visibility — and, ultimately, insights — into enterprise cloud assets is growing ever more challenging.  Cloud estates are sprawling and fragmented, and inventory capabilities in existing tools can be narrow and unintuitive, separating elements like cost and security data into disconnected platforms with limited flexibility.  Cloud governance company CloudQuery is positioning itself to address this problem by centralizing cloud assets, security metadata and cost in one place, and making it accessible through easy, built-in SQL queries and reports. The company is taking a developer-first approach to cloud governance, pulling data from 60-plus sources — including AWS, GCP, Azure, Okta and Wiz — into a single, queryable data warehouse.  The company is now announcing a $16 million funding round led by Partech to further scale its approach to cloud visibility. “The biggest challenge with existing tools is that they’re siloed — one for security, one for cost, one for asset inventory — making it hard to get a unified view across domains,” CQ founder Yevgeny Pats told VentureBeat. “Even simple questions like ‘What EBS volume is attached to an EC2 that is turned off? are hard to answer without stitching together multiple tools.”  CloudQuery under the hood CloudQuery uses two key technologies under the hood: Data warehouse and open-source database ClickHouse and the Apache Arrow framework for developing data analytics applications.  This high-performance plugin architecture built in Go connects directly to APIs like AWS, Azure, Google Cloud Platform (GCP) and many other platforms pulling in configuration, security, and cost metadata. The platform continuously syncs data from dozens of cloud providers and services into a normalized, centralized asset inventory.  “We place a strong emphasis on data accuracy and freshness, syncing at high frequency to ensure teams are working with the most reliable, up-to-date information,” said Pats.  That data, he explained, is structured relationally to power CloudQuery’s SQL engine and built-in reports, so that teams can have full flexibility without relying on black-box tools.  The company also “selectively” uses large language models (LLMs) for natural language querying, SQL generation and recommendations, “but always on top of a foundation of accurate, transparent data,” said Pats. He pointed out that because AI understands SQL well, tools like Claude and OpenAI can create customized reports and analysis in plain English. Taking a developer-first approach is critical, said Pats, because developers are ultimately the ones building, operating and securing today’s cloud infrastructure. Still, many cloud visibility tools were built for top-down governance, not for the people actually in the trenches. “When you put developers first, with accessible data, flexible APIs and native language like SQL, you empower them to move faster, catch issues earlier and build more securely,” he said. Customers are finding ways to use CloudQuery beyond asset inventory. “Many start with visibility, then quickly grow into use cases like compliance monitoring, security posture management, cost optimization, all from the same core platform,” said Pats.  How Hexagon built a serverless data lake for all its cloud stores One enterprise already seeing results is Hexagon. The software company’s cloud center of excellence (CCoE) team had a goal to build a fully serverless data lake that could collect data from all of its cloud accounts and store it in a single data lake.  They also wanted the ability to query this data using SQL and visualize it with tools they were familiar with (such as AWS QuickSight), and explore the history of their cloud configuration over time.  The team built a serverless data pipeline using CloudQuery to collect data from all accounts and store it in S3. AWS Glue then ingests data into Glue DB in a format that Amazon Athena can query, which Athena then does, then visualises in QuickSight. “Having a fully serverless solution was an important requirement,” Hexagon cloud governance and FinOps expert Peter Figueiredo and CloudQuery director of engineering Herman Schaaf wrote in a blog post. “This decision brought lots of benefits since there is no need for time-consuming updates and virtually zero maintenance.” They did have to overcome some challenges, particularly with Amazon S3 support plugins. The CCoE team was one of the first to try out CloudQuery features in the S3 destination and offered insights leading to new features. These include:  Parquet support: The CloudQuery file destination initially only supported CSV and JSON data formats. Errors in JSON interpretations led CloudQuery to add Parquet support.  Data partitioning: A CloudQuery file destination plugin now allows partitioning on initial write (which previously wasn’t available, resulting in extra unnecessary steps).  Resource view for Athena: CloudQuery initially only offered a resources view for AWS compatible with Postgres. But Athena didn’t support this, so CloudQuery added a function that can retrieve a list of all tables to build or update a resources view.  Figueiredo’s team used CloudQuery to replace AWS’s VPC IP address manager (IPAM) — which he called expensive and limited in that it does not cover other cloud providers.  Ultimately, his team runs CloudQuery in ‘data lake’ mode using “ultra cheap infrastructure” including AWS S3, ECS, Glue, Athena and Lambda,” Figueiredo told VentureBeat. This keeps costs low and allows Hexagon to merge all its IP addresses across different cloud providers. “We can quickly query any IP across the board and find who the owners are,” said Figueiredo. “We are now able to collect all we need at a very low cost with near zero maintenance. This is the holy grail for our team.” Daily insights on business use cases with VB Daily If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI. Read our Privacy Policy Thanks for subscribing. Check out more VB newsletters here. An error occured. [ad_2] Source link

[someengineering]

Streamlining Cloud Infrastructure Data Processing!

Unlock the potential of your cloud infrastructure data with an ELT framework and CloudQuery integration. Simplify data processing, enhance scalability, and make informed decisions effortlessly.

[hackgit]

[Media] ​​CloudQuery

​​CloudQuery The open source high performance data integration platform designed for security and infrastructure teams. CloudQuery extracts, transforms, and loads your cloud assets. CloudQuery enables you to assess, audit, and monitor the configurations of your cloud assets. CloudQuery key use-cases and features: ▫️ Search: Use standard SQL to find any asset based on any configuration or relation to other assets. ▫️ Visualize: Connect CloudQuery standard PostgreSQL database to your favorite BI/Visualization tool such as Grafana, QuickSight, etc. ▫️ Policy-as-Code: Codify your security & compliance rules with SQL as the query engine. https://github.com/cloudquery/cloudquery

[wolfcloudlv]

AWS / Cloud / SecDevOps: Know Your Environment Part III

AWS / Cloud / SecDevOps: Know Your Environment Part III

A quick look at the three most promising open source cloud inventory tools on the market today, part III “Steampipe”. https://github.com/turbot/steampipe Steampipe is by far the easiest solution to start with, within a few minutes you have some visualisation to show. Where as the other 2 solutions, CloudQuery and Resoto, need some more configuration and tweaking and more resources like…

[techdirectarchive]

Cloud Asset Management: How to manage your Cloud Assets in AWS with CloudQuery

Cloud Asset Management: How to manage your Cloud Assets in AWS with CloudQuery

A software program that manages data on an organization’s assets is called a cloud asset management software. Cloud-based asset management enables centralized data storage, which may help organizations get easy access to their cloud estates from anywhere with ease as any device that supports an internet connection can view this info. The main advantage is that it offers data security. The good…

[g33kxinc]

How to tame cloud infrastructure sprawl with open source CloudQuery

How to tame cloud infrastructure sprawl with open source CloudQuery

Commentary: The cloud makes infrastructure sprawl easier and worse than ever. Here’s an open source tool to help you keep it in control.Commentary: The cloud makes infrastructure sprawl easier and worse than ever. Here’s an open source tool to help you keep it in control.

[rosdiablatiff01]

Easily query, monitor and analyze your cloud infrastructure | CloudQuery Hub

https://hub.cloudquery.io/

[hackernewsrobot]

Show HN: CloudQuery – Turn any website to serverless API with SPA support

https://github.com/cloudfetch/cloudquery Comments

[macronimous]

cloudquery - Turn any website to serverless #API https://t.co/S86i5uUPYS https://t.co/WuwoByDizW

cloudquery - Turn any website to serverless #API https://t.co/S86i5uUPYS pic.twitter.com/WuwoByDizW

— Macronimous.com (@macronimous) January 31, 2019

from Twitter https://twitter.com/macronimous January 31, 2019 at 11:20PM via IFTTT

[topicprinter]

Article URL: https://github.com/cloudfetch/cloudquery

Comments URL: https://news.ycombinator.com/item?id=18987102

Points: 11

# Comments: 2

[someengineering]

Episode 16: ELT for Cloud Infrastructure Data | Some Engineering Podcast

ELT describes the process of extracting raw data from a source, loading it into a destination, and then transforming the data for analytics purposes. ELT has become mainstream with the rise of cloud warehouses and data lakes, in a shift away from ETL.

[technteacher]

Show HN: CloudQuery – Turn any website to serverless API (with SPA support)

Show HN: CloudQuery – Turn any website to serverless API (with SPA support) 2 by timqian | from Blogger http://bit.ly/2B0b4q9

[wolfcloudlv]

AWS / Cloud: Know Your Environment Part II

AWS / Cloud: Know Your Environment Part II

A quick look at the three most promising open source cloud inventory tools on the market, part II “CloudQuery” osx: brew install cloudquery/tap/cloudquery [upgrade: brew upgrade cloudquery] terminal: cloudquery init aws docker: https://www.docker.com/products/docker-desktop/ docker: docker run --name cloudquery_postgres -p 5432:5432 -e PASSWORD =pass -d postgres aws: configure aws profiles in…

[rosdiablatiff01]

Easily query, monitor and analyze your cloud infrastructure | CloudQuery Hub

OrganizationIdentity