#data pipeline data science

[analyticspursuit]

What is a Data Pipeline? | Data Pipeline Explained in 60 Seconds

If you've been curious about data pipelines but don't know what they are, this video is for you! Data pipelines are a powerful way to manage and process data, and in this video, we'll explain them in 60 seconds.

If you're looking to learn more about data pipelines, or want to know what they are used for, then this video is for you! We'll walk you through the data pipeline architecture and share some of the uses cases for data pipelines.

By the end of this video, you'll have a better understanding of what a data pipeline is and how it can help you with your data management needs!

[cbirt]

Metagenomics and Metatranscriptomics: New Insights and Pipelines to Better Navigate Data Analysis

Scientists at the Institute of Parasitology and Biomedicine and the University of Granada, Spain, along with collaborators, developed two pipelines that could automate and optimize metagenomics and metatranscriptomics data analysis. These pipelines could be adapted for 16S, shotgun, and RNA-Seq data. Its performance was validated through three studies by assessing its taxonomy classification ability.

When Anton van Leeuwenhoek first opened the doors to the unseen world of microorganisms in 1673 through his self-made single-lens microscope, it couldn’t have been possible to imagine the explosion of discoveries that were to follow in its wake. The paradoxical world of microbes is a source of infinite curiosity to many scientists around the world. Thus, it was a no-brainer that with the advent of NGS, the microbes would get their very own niche within it—Metagenomics.

Continue Reading

[godhasforsnakenme]

NASA Cassini-Huygens

Commonly called Cassini, was a space-research mission by NASA, the European Space Agency (ESA), and the Italian Space Agency (ASI) to send a space probe to study the planet Saturn and its system, including its rings and natural satellites. The Flagship-class robotic spacecraft comprised both NASA's Cassini space probe and ESA's Huygens lander, which landed on Saturn's largest moon, Titan. It was the fourth space probe to visit Saturn and the first to enter its orbit, where it stayed from 2004 to 2017. The two craft took their names from the astronomers Giovanni Cassini and Christiaan Huygens.

[rajaniesh]

Unveiling the Power of Delta Lake in Microsoft Fabric

Discover how Microsoft Fabric and Delta Lake can revolutionize your data management and analytics. Learn to optimize data ingestion with Spark and unlock the full potential of your data for smarter decision-making.

In today’s digital era, data is the new gold. Companies are constantly searching for ways to efficiently manage and analyze vast amounts of information to drive decision-making and innovation. However, with the growing volume and variety of data, traditional data processing methods often fall short. This is where Microsoft Fabric, Apache Spark and Delta Lake come into play. These powerful…

[technicalfika]

What is the difference between Data Scientist and Data Engineers ?

In today’s data-driven world, organizations harness the power of data to gain valuable insights, make informed decisions, and drive innovation. Two key players in this data-centric landscape are data scientists and data engineers. Although their roles are closely related, each possesses unique skills and responsibilities that contribute to the successful extraction and utilization of data. In…

[qwikskills]

Resources for Professionals looking to build their knowledge and skills in the field of data science

Data science is a rapidly growing field that requires professionals to have an understanding of various disciplines such as mathematics, statistics, programming, and machine learning. For professionals looking to build their knowledge and skills in the field of data science, there are a variety of resources available. These include online courses, tutorials, books, conferences, and bootcamps that can help individuals gain the necessary skills to become successful data scientists. Additionally, networking with experienced data scientists can provide valuable insight into the industry and help professionals stay up-to-date with the latest trends in data science. With the right resources and dedication to learning, anyone can become a successful data scientist.

Dataiku certification is a valuable resource for professionals who are looking to build their knowledge and skills in the field of data science. This certification program provides an opportunity for individuals to gain expertise in the Dataiku platform, which is used by many organizations around the world. The certification program covers topics such as data preparation, machine learning, natural language processing, and more. With this certification, individuals can demonstrate their proficiency in using Dataiku to create powerful data-driven solutions that drive business success.

[knowledgehound]

Why You Need Search-Driven Analytics? — KnowledgeHound

As the world’s only search-driven survey data analysis solution, Knowledgehound enables Fortune 500 companies to unlock the potential hidden in their survey research. We exist to help uncover and amplify the critical insights that drive greater audiences, happier customers, and the wins over your competitors that insights teams exist to create.

Below are a few of the ways we most commonly help our clients:

1. There’s not enough time in the day

Fear not, KnowledgeHound helps teams reduce the time spent analyzing data by up to 75%. Study up on our $1B pharma client case study.

2. My data is going to waste

We know that all too well… maybe we can help! It’s likely because most of your data is inaccessible to your key stakeholders. Check out this success story on how Fandango transformed into insights explorers.

3. I can’t find the answers I need

Oh no! Help is on the way….. 🏃KnowledgeHound layers the power of search onto an analytics platform built specifically for your survey research. Learn more about how search-driven analytics can unlock powerful outcomes for your business here.

Whether you’re often stuck in Excel building custom tables for a marketing colleague or spending hours crafting a Powerpoint presentation or even answering calls from your agency partners around target messaging, KnowledgeHound makes it insanely easy to find the right insight at the right time to make you look like the rockstar that you are!

Built to ingest, house, and analyze all kinds of respondent-level data, KnowledgeHound is the platform of choice for insights teams that have mountains of consumer data, but need to move quickly when answering critical business questions. Simply connect your favorite DIY survey tools to your account or have your vendor send us your data files, and off to the races you go!

“I immediately saw KnowledgeHound and thought it would make my own life 1000x easier. This stuff takes me hours, days and weeks. It’s a solution that would also help improve synergies across teams. I like the ease-of-use, the interface is very visually appealing and easy to understand. ” — New International CPG Client…..Read more

Also Read: How Danone Changed The Direction Of A $1.4 Billion Line

[cephalopodvictorious]

@convertingslowking

There are (in my experience) three kinds of conspiracy theories

1) the government really did that and we have proof (MK Ultra, the Tuskegee experiments, a lot of CIA actions around the world, Operation Paperclip, etc)

2) this is definitely a real natural phenomenon, trust me bro (Nessie, Moth Man, swamp lights, most kinds of cryptid, etc)

3) The Jews Did It

Category 3 is uhhhh, unfairly common. Anything that ends with (((they))) don’t want you to know boils down to it. Who is hiding knowledge about giants? Who planted dinosaur bones? Who is hiding things behind the ice wall of our flat earth? Who put flouride in the water to do mind control on us? Who are the lizard people drinking child blood and running the world?

Like I said in my tags on that post - miniminuteman does a fantastic job of explaining the pseudo-archaeology to right wing bigot pipeline in his talk here:

Like look, I love silly stories about our world too, they can be really fun! And you can build some REALLY cool stories about the world around you! They can be fantastic jumping off points for original fiction

But for the most part, once you start disbelieving real science and real data and start vibing with "feelings" its very easy for someone to come to you and go "your feeling if discontent is valid" (it may well be, the world is Not Good) and then "let me give you a target"

And that's pretty universally us Jews.

[kp777]

By Brett Wilkins

Common Dreams

Sept. 13, 2023

"It's clear that the crisis is spiraling out of control and the policies of your administration with regard to fossil fuels fail to align with what the science tells us must happen to avert calamity."

In an open letter published Wednesday, around 400 scientists implored U.S. President Joe Biden to endorse the demands of this weekend's March to End Fossil Fuels in New York—which include halting new fossil fuel projects, ending oil and gas drilling on public lands, and declaring a climate emergency.

Noting that "on your first day in office, you issued an executive order pledging that it is 'the policy of my administration to listen to the science' in tackling the climate crisis," the letter's signers lamented that "more than two years later, it's clear that the crisis is spiraling out of control and the policies of your administration with regard to fossil fuels fail to align with what the science tells us must happen to avert calamity."

"With the climate crisis raging all around us—in the form of fires, floods, hurricanes, drought, heatwaves, crop failures, and more—we call on you directly, clearly, and unequivocally to stop enacting policies contrary to science and do what is needed to address the crisis," the signatories added.

The scientists called on Biden to:

Stop federal approval for new fossil fuel projects and repeal permits for climate bombs like the Willow project and the Mountain Valley Pipeline;

Phase out fossil drilling on our public lands and waters;

Declare a climate emergency to halt fossil fuel exports and investments abroad, and turbocharge the buildout of more just, resilient distributed energy (like rooftop and community solar); and

Provide a just transition to a renewable energy future that generates millions of jobs while supporting workers' and community rights, job security, and employment equity.

"We scientists heard the president loud and clear when he pledged two years ago to 'listen to the science' on climate. Yet now we're watching our nation's greenhouse gas emissions spiral out of control while White House policy becomes increasingly unaligned with reality," Sandra Steingraber—an initial signatory of the letter and a senior scientist at the Science and Environmental Health Network"—said in a statement.

"Science says we need to ratchet down fossil fuel extraction—the White House is doubling down," she added. " Scientists are here to say that our data support the demands of this march."

"Given how bad global heating has now gotten, it's simply insane that President Biden still refuses to declare a climate emergency."

Peter Kalmus of NASA's Jet Propulsion Laboratory—another initial signer—said that "given how bad global heating has now gotten, it's simply insane that President Biden still refuses to declare a climate emergency, and indeed, continues to make everything worse by expanding fossil fuels."

"Nothing takes away my hope for humanity's collective future more than Biden's choice to stand with the fossil fuel industry," Kalmus added. "He must pivot and become the climate leader the planet needs, or else he'll continue locking in higher temperatures and ever more irreversible damage to Earth's habitability."

Nearly 800 international, national, and local organizations have endorsed Sunday's March to End Fossil Fuels, which comes ahead of United Nations Secretary-General António Guterres' Climate Ambition Summit and this fall's U.N. Climate Change Conference—also known as COP28—in Dubai. More than 400 marches, rallies, and other climate mobilizations are slated for this weekend.

Our work is licensed under Creative Commons (CC BY-NC-ND 3.0). Feel free to republish and share widely.

[mariacallous]

America’s first large-scale offshore wind farms began sending power to the Northeast in early 2024, but a wave of wind farm project cancellations and rising costs have left many people with doubts about the industry’s future in the US.

Several big hitters, including Ørsted, Equinor, BP, and Avangrid, have canceled contracts or sought to renegotiate them in recent months. Pulling out meant the companies faced cancellation penalties ranging from $16 million to several hundred million dollars per project. It also resulted in Siemens Energy, the world’s largest maker of offshore wind turbines, anticipating financial losses in 2024 of around $2.2 billion.

Altogether, projects that had been canceled by the end of 2023 were expected to total more than 12 gigawatts of power, representing more than half of the capacity in the project pipeline.

So, what happened, and can the US offshore wind industry recover?

I lead the University of Massachusetts Lowell’s Center for Wind-Energy Science, Technology, and Research (WindSTAR) and Center for Energy Innovation, and follow the industry closely. The offshore wind industry’s troubles are complicated, but it’s far from dead in the US, and some policy changes may help it find firmer footing.

A Cascade of Approval Challenges

Getting offshore wind projects permitted and approved in the US takes years and is fraught with uncertainty for developers, more so than in Europe or Asia.

Before a company bids on a US project, the developer must plan the procurement of the entire wind farm, including making reservations to purchase components such as turbines and cables, construction equipment, and ships. The bid must also be cost-competitive, so companies have a tendency to bid low and not anticipate unexpected costs, which adds to financial uncertainty and risk.

The winning US bidder then purchases an expensive ocean lease, costing in the hundreds of millions of dollars. But it has no right to build a wind project yet.

Before starting to build, the developer must conduct site assessments to determine what kind of foundations are possible and identify the scale of the project. The developer must consummate an agreement to sell the power it produces, identify a point of interconnection to the power grid, and then prepare a construction and operation plan, which is subject to further environmental review. All of that takes about five years, and it’s only the beginning.

For a project to move forward, developers may need to secure dozens of permits from local, tribal, state, regional, and federal agencies. The federal Bureau of Ocean Energy Management, which has jurisdiction over leasing and management of the seabed, must consult with agencies that have regulatory responsibilities over different aspects in the ocean, such as the armed forces, Environmental Protection Agency, and National Marine Fisheries Service, as well as groups including commercial and recreational fishing, Indigenous groups, shipping, harbor managers, and property owners.

In December 2023, the majority of offshore wind power capacity was in China and Europe. The United States had just 42 megawatts, but it was about to launch two new wind farms. (Data source: WFO Global Wind Offshore Wind Report 2023.)

For Vineyard Wind I—which began sending power from five of its 62 planned wind turbines off Martha’s Vineyard in early 2024—the time from BOEM’s lease auction to getting its first electricity to the grid was about nine years.

Costs Balloon During Regulatory Delays

Until recently, these contracts didn’t include any mechanisms to adjust for rising supply costs during the long approval time, adding to the risk for developers.

From the time today’s projects were bid to the time they were approved for construction, the world dealt with the Covid-19 pandemic, inflation, global supply chain problems, increased financing costs, and the war in Ukraine. Steep increases in commodity prices, including for steel and copper as well as in construction and operating costs, made many contracts signed years earlier no longer financially viable.

Led by China and the UK, the world had 67,412 megawatts of offshore wind power capacity in operation by the end of 2023. (Source: WTO Global Offshore Wind Report.)

New and rebid contracts are now allowing for price adjustments after the environmental approvals have been given, which is making projects more attractive to developers in the US. Many of the companies that canceled projects are now rebidding.

The regulatory process is becoming more streamlined, but it still takes about six years, while other countries are building projects at a faster pace and larger scale.

Shipping Rules, Power Connections

Another significant hurdle for offshore wind development in the US involves a century-old law known as the Jones Act.

The Jones Act requires vessels carrying cargo between US points to be US-built, US-operated, and US-owned. It was written to boost the shipping industry after World War I. However, there are only three offshore wind turbine installation vessels in the world that are large enough for the turbines proposed for US projects, and none are compliant with the Jones Act.

That means wind turbine components must be transported by smaller barges from US ports and then installed by a foreign installation vessel waiting offshore, which raises the cost and likelihood of delays.

Dominion Energy is building a new ship, the Charybdis, that will comply with the Jones Act. But a typical offshore wind farm needs more than 25 different types of vessels—for crew transfers, surveying, environmental monitoring, cable-laying, heavy lifting, and many other roles.

The nation also lacks a well-trained workforce for manufacturing, construction, and operation of offshore wind farms.

For power to flow from offshore wind farms, the electricity grid also requires significant upgrades. The Department of Energy is working on regional transmission plans, but permitting will undoubtedly be slow.

Lawsuits and Disinfo

Numerous lawsuits from advocacy groups that oppose offshore wind projects have further slowed development.

Wealthy homeowners have tried to stop wind farms that might appear in their ocean view. Astroturfing groups that claim to be advocates of the environment, but are actually supported by fossil fuel industry interests, have launched disinformation campaigns.

In 2023, many Republican politicians and conservative groups immediately cast blame for whale deaths off the coast of New York and New Jersey on the offshore wind developers, but the evidence points instead to increased ship traffic collisions and entanglements with fishing gear.

Such disinformation can reduce public support and slow projects’ progress.

Just Keep Spinnin’

The Biden administration set a goal to install 30 gigawatts of offshore wind capacity by 2030, but recent estimates indicate that the actual number will be closer to half that.

Despite the challenges, developers have reason to move ahead.

The Inflation Reduction Act provides incentives, including federal tax credits for the development of clean energy projects and for developers that build port facilities in locations that previously relied on fossil fuel industries. Most coastal state governments are also facilitating projects by allowing for a price readjustment after environmental approvals have been given. They view offshore wind as an opportunity for economic growth.

These financial benefits can make building an offshore wind industry more attractive to companies that need market stability and a pipeline of projects to help lower costs—projects that can create jobs and boost economic growth and a cleaner environment.

[dykeishheart]

Intellectualism is such a fraught subject because we live in a zeitgeist of anti-intellectualism being pushed from major political bodies onto constituencies via religious and economic propaganda (defunding schools, creationist leaders defrauding sciences, anti-medical conspiracy riddled through middle class and even some legislators, etc), while at the same time certain vectors of social power and legitimacy are undeniably rooted in at least the appearance of intellect. See Musk, Zuckerberg, and Gates fanboys praising their genius, the entire social movement that I like to call 'r/atheism debate me era' being beholden to talking heads they idolize like Richard Dawkins and Sam Harris, and just the general millieu of Smart Man captivating the social media feed every few minutes with Smart quotes. Society loves an Einstein, to the point that that is a recognizable name every person on earth knows how to correlate to a specific kind of person: a smart person idolized for intelligence.

Actual education is the least accessible it's ever been in terms of the kinds that offer prestige (university) while raw knowledge and data is the most accessible it's ever been for people willing to comb through digital archives. Being too smart is to invite insult and dismissal if it's coupled with any personality flaws but it's almost universally stated that people want intelligence in a partner. The smartest people aren't given acclaim or status for being smart, but the highest wages are by and large earned by people with the most education. Legislators want less actual education happening but more educating happening, because they want more people in schools to generate revenue and fill the school-to-industry training pipeline, but they don't want those people learning the kinds of things that would teach them that pipeline exists.

There's also an incredibly annoying phenomenon where every two or three years a new pseudoscience pops up trying to explain what intelligence is and what personalities are and why we need to have more societal divisions over it, and it's been like this since The Bell Curve came out in 1994, and it's basically been eugenics every time with the license plates changed. The political benefit of being able to categorize people for division and domination dovetails perfectly with the bizarre hook that factoids have on the collective brainspace; everyone loves feeling like they know a special secret about the world, and a good many people like the feeling of that secret so much that they turn off their critical thinking for just long enough to accept any old bullshit or bigotry or conspiracy or ancient form of racism if it's packaged to look like the underside of a Snapple cap. Needless to say, you can find the absolute most esoteric, bizarre, niche, and useless info only two digital footprints away from the most baseline of common knowledge, and both will be wrong in different but equal measure, each spawned by a different abberant strain of social psychosis with a different degree of popularity determined by god drawing sticks blind out of a cup. The layers of disinformation, misinformation, lies, half remembered factoids, hallucinations, misconceptions, willful obtusenesses, deliberate obfuscations, and general fuckeries that exist within the All Encompassing Brain Soup that is the digital age could make one decide epistemology is overrated if the day of encounter was sufficiently exhausting.

And at the end of the day it largely doesn't matter. Intelligence is fake anyway. Truth is authored and knowledge unwrites it each and every minute to rewrite to the shape of power. What does it mean to be smart? God help you I don't fucking know. Will I be able to afford rent six months from now? That's a real question. Tell me when you figure that one out.

[spacetimewithstuartgary]

The universe on display: The powerful instruments that allow us to observe the cosmos

Starting today, the Earth will be passing through a meteor shower. But in astronomy, the human eye is very much a limited tool. But increasingly powerful instruments are allowing us to peer ever deeper into the cosmos and ever further back in time, shedding new light on the origins of the universe.

Today, scientists are able to observe an exoplanet orbiting its star, an individual galaxy and even the entire universe. "The universe is actually mostly empty space," says Jean-Paul Kneib, a professor at EPFL's Laboratory of Astrophysics. "There isn't much that's hidden."

The key is to know what you're looking for, build the right instrument, and look in the right direction. And then to do a little housekeeping.

"Our galaxy sits in the foreground of our field of vision, blocking our view beyond it," explains Kneib. "So if we want to map hydrogen in the early universe, for example, we first have to model this entire foreground then remove it from our images until we obtain a signal a million times smaller than the one emitted by the Milky Way."

Galileo could draw only what he saw with his telescope. But today, astronomers can see the universe in its entirety, right back to its very beginnings. This is largely because of rapid advancements in the instruments they use. And more developments are expected in the years ahead.

The James Webb Space Telescope (JWST), launched in December 2021, aims to observe events that happened 13 billion years ago when the first stars and galaxies were forming. The Square Kilometre Array (SKA) radio telescope—currently under construction and scheduled for completion by the end of the decade—will look back even further to a time when there were no stars and the cosmos contained mainly hydrogen—the element that makes up 92% of all atoms in the universe.

"An easy way to detect this gas is to operate in the radio frequency range, which is exactly what the SKA will do," says Kneib. "The aim is to detect a signal a million times smaller than the foreground signals."

Another project in the pipeline is the Laser Interferometer Space Antenna (LISA), run by the European Space Agency (ESA). Scheduled for launch in 2035, the antenna will observe gravitational waves, shedding light on the growth of black holes and possibly the waves created just after the Big Bang.

Playing digital catch-up

These new instruments wouldn't be so enlightening without advancements in other fields. "As things stand, we don't have the software to process data from the SKA," says Kneib, who's confident that we'll get there eventually thanks to progress in computer and computational science, artificial intelligence (AI) and processing power. AI is invaluable for sorting through vast quantities of data to find an interesting anomaly and for calculating the mass of galaxies, for example.

"Scientists can use the gravitational lensing effect, whereby a large object bends light from a distant source, to calculate the mass of galaxy clusters to within a range of one percent, just as if they were using a scale," explains Kneib. "And we can train AI models to spot distortions in images caused by gravitational lenses. Given that there are probably 200 billion galaxies in the universe, that's a huge help—even if we can measure the mass of only one galaxy in every thousand."

But do the images we see depict what's really out there? A famous image published in 2019 showed a donut-shaped ring of light surrounding a black hole. Would we actually see that ring if we got close to it?

"It wasn't an optical photo," says Kneib. "It was a purely digital rendering. In order to accurately observe the millimeter-wavelength signals emitted by the black hole, scientists had to combine multiple ground-based telescopes to create one roughly the size of the globe. The image was then reconstructed via interferometry [a measurement method using wave interference].

"But the image nevertheless represents a real signal, linked to the amount of matter in the dust cloud surrounding the black hole. In simple terms, the dark part is the black hole and the lighter part is the matter orbiting it."

Seeing in four dimensions

"Calculations are only part of the equation in astronomy—you need to be able to visualize things, which also helps you check that your calculations are correct," says Kneib, who is capable of reading the majestic image of the Lagoon Nebula, situated 4,000 light-years away, like a book.

"That image was produced using optical observations at different wavelengths to depict the various gases. Of course, there was a bit of artistry involved in enhancing the colors. But the image also has a great deal of significance for physicists. The colors indicate the presence of different gases: red for hydrogen, blue for oxygen and green for nitrogen. The compact, black areas contain large quantities of dust. These are typically the regions where stars form."

Visualization is especially important when observing objects in more than two dimensions. "By studying the cosmos in three dimensions, we're able to measure the distance between celestial objects," says Kneib.

In early April, scientists working on the Dark Energy Spectroscopic Instrument (DESI) project—including astrophysicists from EPFL—announced they had created the largest ever 3D map of the universe's galaxies and quasars.

But that's not all: researchers are also studying the universe in the fourth dimension—time—and, in doing so, opening up incredible possibilities for observing bright yet fleeting phenomena. "For example, we don't really understand the origin of fast radio bursts, which are incredibly bright blasts of electromagnetic radiation that last only a few seconds at most, and sometimes just a fraction of a millisecond," says Kneib.

Will we ever find life on an exoplanet? Kneib replies, "With infrared interferometry, there's a very real prospect that we could take a photo of a planet orbiting around another star. The image would likely be blurry, but we'd be able to observe and characterize features such as clouds and structural variations on the planet's surface. That's definitely a possibility, maybe 20 or 30 years from now."

When it comes to some fundamental questions, however, we're unlikely to find the answers through imaging alone. Why is the universe expanding at an accelerating rate? Is it because of dark energy? Why is 80% of matter invisible? Are we completely wrong about gravity? Future generations of astrophysicists will keep their eyes trained on the skies or glued to their screens as they try to unravel the deepest mysteries of our universe.

IMAGE: The Lagoon Nebula. Credit: NASA, ESA, STSCI

[rjzimmerman]

Excerpt from this story from DeSmog Blog:

Earlier this year a far-right group called Canada Proud began running Facebook ads to its more than 534,000 followers attacking the climate change technology favored by conservative leaders as well as the country’s largest oil and gas producers. 

“Carbon capture is billed as a green technology that stops carbon from entering the atmosphere,” the ad explains. “But is it really good for the environment? As it turns out, not really.” The technology, Canada Proud claimed, “can poison groundwater, it can put carcinogens in the soil and even has a record of causing earthquakes.” 

Major oil sands companies and their political allies in Alberta and Ottawa have for years pushed the opposite message — that carbon capture and storage, also known as CCS, is necessary to ensure the survival of oil and gas while also addressing climate change. 

So far the loudest attacks against carbon capture have come from environmental groups and progressive politicians which see it as an expensive false solution to climate change that furthers our dependency on oil and gas. 

But as more of these projects move forward, they’re also activating opposition from the right, creating new political divisions between establishment conservatives and groups attempting to catalyze grassroots anger towards expensive industrial megaprojects in rural areas. 

“It’s very interesting that groups like Canada Proud are seemingly mobilizing, or testing the waters to mobilize, against carbon capture and storage,” Bob Neubauer, an assistant professor in communications at the University of Manitoba who studies rightwing populism and climate change disinformation, told DeSmog. 

“Their base doesn’t appear to be full of people who are excited about a technocratic post-carbon scenario,” he added.

Dissatisfaction with the technology has been edging into the mainstream of rightwing discourse. “We might as well take tax money at gunpoint and burn it,” Canadian conservative influencer Jordan Peterson in February wrote on X to his 5.3 million followers in response to a CCS project in Wyoming. 

U.S. presidential candidate Robert F. Kennedy Jr., who has been frequently interviewed on conservative media platforms, last year called carbon capture a “boondoggle.” Vivek Ramaswamy, who ran a failed primary campaign this year against Donald Trump for the Republican leadership, called pipelines in Iowa that can transport captured carbon to sites where it can be buried underground “the greatest violation of property rights.” 

These tensions are growing in Alberta, the heart of Canada’s oil and gas industry, where a consortium of six top oil sands companies known as the Pathways Alliance applied this spring for regulatory approval to build a $16.5 billion carbon capture and storage project. It’s been blanketing the country in ads stating that “carbon capture is an important step towards carbon neutral resource extraction.” 

Alberta’s premier Danielle Smith, who earlier this year shared a stage with Tucker Carlson and was recently interviewed on Peterson’s podcast, has announced taxpayer support of up to $5.3 billion for the plan. “Let me tell you, we are only going to strengthen the case for carbon capture, utilization and storage in the years ahead,” she said during an industry convention last year. 

Grassroots Opposition Growing

Rural northern Alberta, where the project will be built, is definitely no hotbed of environmental activism. The region is home to an anti-renewable energy group called Wind Concerns whose leader earlier told DeSmog that climate science is “ridden with fraudulent data and outright lies.”

Yet locals there have created a new group called No to CO2 Landowners Group, which has teamed up with the Athabasca Chipewyan First Nation and environmental organizations to oppose the Pathways Alliance carbon capture plan. 

“Despite their claims, this is unproven technology with far-reaching implications into the future,” Amil Shapka, one of No to CO2’s representatives, has said. “With this being Canada’s largest CO2 pipeline and storage project, is our community ready for the potential health, safety and environmental risks to our water?”

The increasingly scrambled politics of carbon capture are now creating tensions at the national level in Canada. Because the federal Liberal government has proposed investment tax credits up to $10 billion to support the Pathways Alliance plan, Prime Minister Justin Trudeau is now associated with a mega-project opposed by some rural Canadians. 

[womaneng]

Data Science

📌Data scientists use a variety of tools and technologies to help them collect, process, analyze, and visualize data. Here are some of the most common tools that data scientists use:

👩🏻‍💻Programming languages: Data scientists typically use programming languages such as Python, R, and SQL for data analysis and machine learning.

📊Data visualization tools: Tools such as Tableau, Power BI, and matplotlib allow data scientists to create visualizations that help them better understand and communicate their findings.

🛢Big data technologies: Data scientists often work with large datasets, so they use technologies like Hadoop, Spark, and Apache Cassandra to manage and process big data.

🧮Machine learning frameworks: Machine learning frameworks like TensorFlow, PyTorch, and scikit-learn provide data scientists with tools to build and train machine learning models.

☁️Cloud platforms: Cloud platforms like Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure provide data scientists with access to powerful computing resources and tools for data processing and analysis.

📌Data management tools: Tools like Apache Kafka and Apache NiFi allow data scientists to manage data pipelines and automate data ingestion and processing.

🧹Data cleaning tools: Data scientists use tools like OpenRefine and Trifacta to clean and preprocess data before analysis.

☎️Collaboration tools: Data scientists often work in teams, so they use tools like GitHub and Jupyter Notebook to collaborate and share code and analysis.

For more follow @woman.engineer

[jcmarchi]

Elaine Liu: Charging ahead

New Post has been published on https://thedigitalinsider.com/elaine-liu-charging-ahead/

Elaine Liu: Charging ahead

MIT senior Elaine Siyu Liu doesn’t own an electric car, or any car. But she sees the impact of electric vehicles (EVs) and renewables on the grid as two pieces of an energy puzzle she wants to solve.

The U.S. Department of Energy reports that the number of public and private EV charging ports nearly doubled in the past three years, and many more are in the works. Users expect to plug in at their convenience, charge up, and drive away. But what if the grid can’t handle it?

Electricity demand, long stagnant in the United States, has spiked due to EVs, data centers that drive artificial intelligence, and industry. Grid planners forecast an increase of 2.6 percent to 4.7 percent in electricity demand over the next five years, according to data reported to federal regulators. Everyone from EV charging-station operators to utility-system operators needs help navigating a system in flux.

That’s where Liu’s work comes in.

Liu, who is studying mathematics and electrical engineering and computer science (EECS), is interested in distribution — how to get electricity from a centralized location to consumers. “I see power systems as a good venue for theoretical research as an application tool,” she says. “I’m interested in it because I’m familiar with the optimization and probability techniques used to map this level of problem.”

Liu grew up in Beijing, then after middle school moved with her parents to Canada and enrolled in a prep school in Oakville, Ontario, 30 miles outside Toronto.

Liu stumbled upon an opportunity to take part in a regional math competition and eventually started a math club, but at the time, the school’s culture surrounding math surprised her. Being exposed to what seemed to be some students’ aversion to math, she says, “I don’t think my feelings about math changed. I think my feelings about how people feel about math changed.”

Liu brought her passion for math to MIT. The summer after her sophomore year, she took on the first of the two Undergraduate Research Opportunity Program projects she completed with electric power system expert Marija Ilić, a joint adjunct professor in EECS and a senior research scientist at the MIT Laboratory for Information and Decision Systems.

Predicting the grid

Since 2022, with the help of funding from the MIT Energy Initiative (MITEI), Liu has been working with Ilić on identifying ways in which the grid is challenged.

One factor is the addition of renewables to the energy pipeline. A gap in wind or sun might cause a lag in power generation. If this lag occurs during peak demand, it could mean trouble for a grid already taxed by extreme weather and other unforeseen events.

If you think of the grid as a network of dozens of interconnected parts, once an element in the network fails — say, a tree downs a transmission line — the electricity that used to go through that line needs to be rerouted. This may overload other lines, creating what’s known as a cascade failure.

“This all happens really quickly and has very large downstream effects,” Liu says. “Millions of people will have instant blackouts.”

Even if the system can handle a single downed line, Liu notes that “the nuance is that there are now a lot of renewables, and renewables are less predictable. You can’t predict a gap in wind or sun. When such things happen, there’s suddenly not enough generation and too much demand. So the same kind of failure would happen, but on a larger and more uncontrollable scale.”

Renewables’ varying output has the added complication of causing voltage fluctuations. “We plug in our devices expecting a voltage of 110, but because of oscillations, you will never get exactly 110,” Liu says. “So even when you can deliver enough electricity, if you can’t deliver it at the specific voltage level that is required, that’s a problem.”

Liu and Ilić are building a model to predict how and when the grid might fail. Lacking access to privatized data, Liu runs her models with European industry data and test cases made available to universities. “I have a fake power grid that I run my experiments on,” she says. “You can take the same tool and run it on the real power grid.”

Liu’s model predicts cascade failures as they evolve. Supply from a wind generator, for example, might drop precipitously over the course of an hour. The model analyzes which substations and which households will be affected. “After we know we need to do something, this prediction tool can enable system operators to strategically intervene ahead of time,” Liu says.

Dictating price and power

Last year, Liu turned her attention to EVs, which provide a different kind of challenge than renewables.

In 2022, S&P Global reported that lawmakers argued that the U.S. Federal Energy Regulatory Commission’s (FERC) wholesale power rate structure was unfair for EV charging station operators.

In addition to operators paying by the kilowatt-hour, some also pay more for electricity during peak demand hours. Only a few EVs charging up during those hours could result in higher costs for the operator even if their overall energy use is low.

Anticipating how much power EVs will need is more complex than predicting energy needed for, say, heating and cooling. Unlike buildings, EVs move around, making it difficult to predict energy consumption at any given time. “If users don’t like the price at one charging station or how long the line is, they’ll go somewhere else,” Liu says. “Where to allocate EV chargers is a problem that a lot of people are dealing with right now.”

One approach would be for FERC to dictate to EV users when and where to charge and what price they’ll pay. To Liu, this isn’t an attractive option. “No one likes to be told what to do,” she says.

Liu is looking at optimizing a market-based solution that would be acceptable to top-level energy producers — wind and solar farms and nuclear plants — all the way down to the municipal aggregators that secure electricity at competitive rates and oversee distribution to the consumer.

Analyzing the location, movement, and behavior patterns of all the EVs driven daily in Boston and other major energy hubs, she notes, could help demand aggregators determine where to place EV chargers and how much to charge consumers, akin to Walmart deciding how much to mark up wholesale eggs in different markets.

Last year, Liu presented the work at MITEI’s annual research conference. This spring, Liu and Ilić are submitting a paper on the market optimization analysis to a journal of the Institute of Electrical and Electronics Engineers.

Liu has come to terms with her early introduction to attitudes toward STEM that struck her as markedly different from those in China. She says, “I think the (prep) school had a very strong ‘math is for nerds’ vibe, especially for girls. There was a ‘why are you giving yourself more work?’ kind of mentality. But over time, I just learned to disregard that.”

After graduation, Liu, the only undergraduate researcher in Ilić’s MIT Electric Energy Systems Group, plans to apply to fellowships and graduate programs in EECS, applied math, and operations research.

Based on her analysis, Liu says that the market could effectively determine the price and availability of charging stations. Offering incentives for EV owners to charge during the day instead of at night when demand is high could help avoid grid overload and prevent extra costs to operators. “People would still retain the ability to go to a different charging station if they chose to,” she says. “I’m arguing that this works.”