#computational data processing

[techtoio]

How Big Data Analytics is Changing Scientific Discoveries

Introduction

In the contemporary world of the prevailing sciences and technologies, big data analytics becomes a powerful agent in such a way that scientific discoveries are being orchestrated. At Techtovio, we explore this renewed approach to reshaping research methodologies for better data interpretation and new insights into its hastening process. Read to continue

[centralbunnyunit]

[stone-cold-groove]

60s era Sperry Rand UNIVAC nameplate.

[37-feral-raccoons]

comp sci majors who also hate generative AI reblog please I need to know some people in my field are sane 😭

[cherry-bomb-ships]

Was talking with my gf last night about PPG ship stuff and realized I've made my self insert a programmer and coder... in a series that takes place within the turn of the century... bro Y2K gonna fucking kill her 😭😭😭

[lazer-exe]

"spotify wrapped was clearly AI"

Two questions. What, exactly, do you think AI is? And did you think spotify had people HAND PICKING your top songs before this???? be for real

[frank-olivier]

Deep Learning, Deconstructed: A Physics-Informed Perspective on AI’s Inner Workings

Dr. Yasaman Bahri’s seminar offers a profound glimpse into the complexities of deep learning, merging empirical successes with theoretical foundations. Dr. Bahri’s distinct background, weaving together statistical physics, machine learning, and condensed matter physics, uniquely positions her to dissect the intricacies of deep neural networks. Her journey from a physics-centric PhD at UC Berkeley, influenced by computer science seminars, exemplifies the burgeoning synergy between physics and machine learning, underscoring the value of interdisciplinary approaches in elucidating deep learning’s mysteries.

At the heart of Dr. Bahri’s research lies the intriguing equivalence between neural networks and Gaussian processes in the infinite width limit, facilitated by the Central Limit Theorem. This theorem, by implying that the distribution of outputs from a neural network will approach a Gaussian distribution as the width of the network increases, provides a probabilistic framework for understanding neural network behavior. The derivation of Gaussian processes from various neural network architectures not only yields state-of-the-art kernels but also sheds light on the dynamics of optimization, enabling more precise predictions of model performance.

The discussion on scaling laws is multifaceted, encompassing empirical observations, theoretical underpinnings, and the intricate dance between model size, computational resources, and the volume of training data. While model quality often improves monotonically with these factors, reaching a point of diminishing returns, understanding these dynamics is crucial for efficient model design. Interestingly, the strategic selection of data emerges as a critical factor in surpassing the limitations imposed by power-law scaling, though this approach also presents challenges, including the risk of introducing biases and the need for domain-specific strategies.

As the field of deep learning continues to evolve, Dr. Bahri’s work serves as a beacon, illuminating the path forward. The imperative for interdisciplinary collaboration, combining the rigor of physics with the adaptability of machine learning, cannot be overstated. Moreover, the pursuit of personalized scaling laws, tailored to the unique characteristics of each problem domain, promises to revolutionize model efficiency. As researchers and practitioners navigate this complex landscape, they are left to ponder: What unforeseen synergies await discovery at the intersection of physics and deep learning, and how might these transform the future of artificial intelligence?

Yasaman Bahri: A First-Principle Approach to Understanding Deep Learning (DDPS Webinar, Lawrence Livermore National Laboratory, November 2024)

Sunday, November 24, 2024

[techdriveplay]

Why Quantum Computing Will Change the Tech Landscape

The technology industry has seen significant advancements over the past few decades, but nothing quite as transformative as quantum computing promises to be. Why Quantum Computing Will Change the Tech Landscape is not just a matter of speculation; it’s grounded in the science of how we compute and the immense potential of quantum mechanics to revolutionise various sectors. As traditional…

[anxiously-going]

I'm currently looking for a second job and I think I may vomit.

[unopenablebox]

apparently, by happening to know that you need to preallocate arrays in matlab, i've saved my labmate like five days of simulation time

the shock of having had useful programming-related knowledge not already known to all of mankind was so enormous that i may need to lie down

[mohammed44c]

Edge Computing, Real-Time Data Processing, and Intelligent Automation

In the dynamic landscape of the power industry, staying ahead of the curve requires a fusion of cutting-edge technologies and strategic operations. With over four years of experience in the field, our journey has been marked by innovation, efficiency, and resilience. In this article, we explore how the convergence of edge computing, real-time data processing, predictive fault diagnosis, and intelligent automation is revolutionizing the energy sector.

Edge Computing: A Powerhouse at the Edge

Edge computing is the bedrock upon which modern utility IT operations are built. By processing data closer to the source, we've reduced latency and increased responsiveness. This real-time capability has enabled us to make critical decisions swiftly, optimizing grid operations and minimizing downtime. The result? A more reliable and efficient energy distribution system.

Real-Time Data Processing: Harnessing the Flow

The ability to handle vast volumes of real-time data has unlocked new possibilities for the power industry. We've implemented advanced data analytics to monitor and control grid assets proactively. Predictive fault diagnosis and anomaly detection algorithms have become our allies in preventing potential failures, thus averting costly disruptions.

Predictive Fault Diagnosis: Proactive Maintenance

Predictive fault diagnosis is a game-changer in the energy industry. By leveraging historical data and machine learning models, we've gained the capability to predict equipment failures before they occur. This predictive maintenance approach has not only extended the lifespan of critical assets but has also significantly reduced operational costs.

Robotic Process Automation (RPA): Streamlining Operations

RPA has automated routine tasks, freeing up human resources for more complex problem-solving. In the power sector, this has led to improved efficiency in billing, customer service, and administrative functions. It's allowed us to allocate resources strategically and ensure a seamless experience for customers.

Intelligent Automation (IA): Powering the Future

Intelligent Automation (IA) goes beyond RPA, integrating AI and machine learning to make autonomous decisions. IA systems continuously learn from data, optimizing grid operations in real-time. It's a crucial component in our journey toward a smart grid, where energy generation, distribution, and consumption are finely tuned to meet demand efficiently.

In conclusion, the synergy of edge computing, real-time data processing, predictive fault diagnosis, RPA, and IA has transformed the power industry. We are no longer just energy providers; we are orchestrators of a reliable, efficient, and sustainable energy ecosystem. As we look to the future, our commitment to innovation remains unwavering, ensuring that the lights stay on and the power flows seamlessly for generations to come.

[quaranmine]

You put so much research and detail into Incandescence of a Dying Light and it's amazing. Do you have any tips for someone trying to research for a story? One of my characters is an elderly Floridian lifeguard but the best sources I can find are some NYT articles about Long Island lifeguards

Oh thank you! I appreciate that.

As for researching, I think the best way to go about it is to try to research specific parts of your problem. You might not be able to find resources for an elderly Floridian lifeguard, but you can probably find resources for lifeguards, lifeguards in a particular area, and elderly people doing swimming/rescues. I can try to give you some pointers but without knowing your story or specific needs some of these tips might not work. Just use them as a jumping off point! Specific problems can be easier to research than broad problems--unless it's too specific, and then you lose all your results. Researching is a balancing act between those two.

Are they a lifeguard at the ocean, or a pool? If it's the ocean, where in Florida? I'd look up information about the sea currents in that area. That might give you an idea of the kind of risks your lifeguard is looking at, like if they work at a beach in a place known for rip currents or something. That will add some realism and you can probably find resources on what your lifeguard character is looking for. If it's a pool, your job is probably much easier because I have to assume most of the same rules apply for elsewhere.

Is there a specific time period you are looking at? I'm no lifeguard, so anyone can correct me if I'm wrong, but I imagine the profession hasn't changed very much in the past few decades or longer. If someone is a lifeguard in the 80s my gut feeling is that their job is still pretty similar to what it is today. So, that might make your research easier if you can expand the time periods you are looking at. My story is set in 1989 so I'm always looking for info from that time period. But most of what I'm doing is looking at fire lookout resources from the 50s or early 2000s, and then matching the technology in my fic to the late 80s.

I would also look up things like lifeguard handbook, lifeguard skills, lifeguard employee handbook, lifeguard training materials, etc. For example, just by looking up "florida lifeguard" I found the Jacksonville Beach Ocean Rescue Lifeguard Academy, which gives some details about the requirements and steps to become a lifeguard there. Could be useful information. I also found a 400 page Red Cross lifeguarding manual pdf. For more personal information, perhaps add stuff like "interview" to the search? I'm sure you would be able to find people talking about their jobs.

As for your character's age, that might not require much research. Being older does not mean a character can't be fit or strong (but I don't know if you mean 60 or 90 when you say "elderly" and that matters.) You could explore sources about active elderly people too, if you wanted. This will just depend on the details of your character though.

I would also recommend using some advanced search techniques. For example, if you only want examples about Florida, write it as "florida" with the quotes around it, and you'll only receive pages that contain the word Florida. Or, if you want to exclude something that is muddying your search results, put the word with a minus. Tragically at one point during my chapter 8 research i had to add -maui to the search to try and exclude news articles associated with it.

Generally though I would just look for lots of sources of many types, and then add them together. It is unlikely you will find sources that match everything you need. Break down what you need to know into smaller pieces. I may have 20+ tabs open for information that amounts to....a few paragraphs. Vary your search queries a lot, try different key words. My research for Firewatch AU has been helped by the fact that the job involves the federal government, which is great at recordkeeping and often has a lot of publicly available information. Your mileage may vary with other subjects. Watch your sources for accuracy. Or, if you rely on sources from a different location (Long Island vs Florida) be prepared to try and identify and fill in the gaps where there are differences between the locations.

Good luck!

[stone-cold-groove]

A page from Sperry UNIVAC’s computer brochure - 1976.

[roseband]

oof i just realized since i have a newer phone now and outlook app works on it, not only can i work on teams off my wrist, but i can do EMAILS off my wrist

[jcmarchi]

Choosing the Eyes of the Autonomous Vehicle: A Battle of Sensors, Strategies, and Trade-Offs

New Post has been published on https://thedigitalinsider.com/choosing-the-eyes-of-the-autonomous-vehicle-a-battle-of-sensors-strategies-and-trade-offs/

Choosing the Eyes of the Autonomous Vehicle: A Battle of Sensors, Strategies, and Trade-Offs

By 2030, the autonomous vehicle market is expected to surpass $2.2 trillion, with millions of cars navigating roads using AI  and advanced sensor systems. Yet amid this rapid growth, a fundamental debate remains unresolved: which sensors are best suited for autonomous driving — lidars, cameras, radars, or something entirely new?

This question is far from academic. The choice of sensors affects everything from safety and performance to cost and energy efficiency. Some companies, like Waymo, bet on redundancy and variety, outfitting their vehicles with a full suite of lidars, cameras, and radars. Others, like Tesla, pursue a more minimalist and cost-effective approach, relying heavily on cameras and software innovation.

Let’s explore these diverging strategies, the technical paradoxes they face, and the business logic driving their decisions.

Why Smarter Machines Demand Smarter Energy Solutions

This is indeed an important issue. I faced a similar dilemma when I launched a drone-related startup in 2013. We were trying to create drones capable of tracking human movement. At that time, the idea was ahead, but it soon became clear that there was a technical paradox.

For a drone to track an object, it must analyze sensor data, which requires computational power — an onboard computer. However, the more powerful the computer needs to be, the higher the energy consumption. Consequently, a battery with more capacity is needed. However, a larger battery increases the drone’s weight, and more weight requires even more energy. A vicious cycle arises: increasing power demands lead to higher energy consumption, weight, and ultimately, cost.

The same problem applies to autonomous vehicles. On the one hand, you want to equip the vehicle with all possible sensors to collect as much data as possible, synchronize it, and make the most accurate decisions. On the other hand, this significantly increases the system’s cost and energy consumption. It’s important to consider not only the cost of the sensors themselves but also the energy required to process their data.

The amount of data is increasing, and the computational load is growing. Of course, over time, computing systems have become more compact and energy-efficient, and software has become more optimized. In the 1980s, processing a 10×10 pixel image could take hours; today, systems analyze 4K video in real-time and perform additional computations on the device without consuming excessive energy. However, the performance dilemma still remains, and AV companies are improving not only sensors but also computational hardware and optimization algorithms.

Processing or Perception?

The performance issues where the system must decide which data to drop are primarily due to computational limitations rather than problems with LiDAR, camera, or radar sensors. These sensors function as the vehicle’s eyes and ears, continuously capturing vast amounts of environmental data. However, if the onboard computing “brain” lacks the processing power to handle all this information in real time, it becomes overwhelming. As a result, the system must prioritize certain data streams over others, potentially ignoring some objects or scenes in specific situations to focus on higher-priority tasks.

This computational bottleneck means that even if the sensors are functioning perfectly, and often they have redundancies to ensure reliability, the vehicle may still struggle to process all the data effectively. Blaming the sensors isn’t appropriate in this context because the issue lies in the data processing capacity. Enhancing computational hardware and optimizing algorithms are essential steps to mitigate these challenges. By improving the system’s ability to handle large data volumes, autonomous vehicles can reduce the likelihood of missing critical information, leading to safer and more reliable operations.

Lidar, Сamera, and Radar systems: Pros & Cons

It’s impossible to say that one type of sensor is better than another — each serves its own purpose. Problems are solved by selecting the appropriate sensor for a specific task.

LiDAR, while offering precise 3D mapping, is expensive and struggles in adverse weather conditions like rain and fog, which can scatter its laser signals. It also requires significant computational resources to process its dense data.

Cameras, though cost-effective, are highly dependent on lighting conditions, performing poorly in low light, glare, or rapid lighting changes. They also lack inherent depth perception and struggle with obstructions like dirt, rain, or snow on the lens.

Radar is reliable in detecting objects in various weather conditions, but its low resolution makes it hard to distinguish between small or closely spaced objects. It often generates false positives, detecting irrelevant items that can trigger unnecessary responses. Additionally, radar cannot decipher context or help identify objects visually, unlike with cameras.

By leveraging sensor fusion — combining data from LiDAR, radar, and cameras — these systems gain a more holistic and accurate understanding of their environment, which in turn enhances both safety and real-time decision-making. Keymakr’s collaboration with leading ADAS developers has shown how critical this approach is to system reliability. We’ve consistently worked on diverse, high-quality datasets to support model training and refinement.

Waymo VS Tesla: A Tale of Two Autonomous Visions

In AV, few comparisons spark as much debate as Tesla and Waymo. Both are pioneering the future of mobility — but with radically different philosophies. So, why does a Waymo car look like a sensor-packed spaceship, while Tesla appears almost free of external sensors?

Let’s take a look at the Waymo vehicle. It’s a base Jaguar modified for autonomous driving. On its roof are dozens of sensors: lidars, cameras, spinning laser systems (so-called “spinners”), and radars. There are truly many of them: cameras in the mirrors, sensors on the front and rear bumpers, long-range viewing systems — all of this is synchronized.

If such a vehicle gets into an accident, the engineering team adds new sensors to gather the missing information. Their approach is to use the maximum number of available technologies.

So why doesn’t Tesla follow the same path? One of the main reasons is that Tesla has not yet released its Robotaxi to the market. Also, their approach focuses on cost minimization and innovation. Tesla believes using lidars is impractical due to their high cost: the manufacturing cost of an RGB camera is about $3, whereas a lidar can cost $400 or more. Furthermore, lidars contain mechanical parts — rotating mirrors and motors—which makes them more prone to failure and replacement.

Cameras, by contrast, are static. They have no moving parts, are much more reliable, and can function for decades until the casing degrades or the lens dims. Moreover, cameras are easier to integrate into a car’s design: they can be hidden inside the body, made nearly invisible.

Production approaches also differ significantly. Waymo uses an existing platform — a production Jaguar — onto which sensors are mounted. They don’t have a choice. Tesla, on the other hand, manufactures vehicles from scratch and can plan sensor integration into the body from the outset, concealing them from view. Formally, they will be listed in the specs, but visually, they’ll be almost unnoticeable.

Currently, Tesla uses eight cameras around the car — in the front, rear, side mirrors, and doors. Will they use additional sensors? I believe so.

Based on my experience as a Tesla driver who has also ridden in Waymo vehicles, I believe that incorporating lidar would improve Tesla’s Full Self-Driving system. It feels to me that Tesla’s FSD currently lacks some accuracy when driving. Adding lidar technology could enhance its ability to navigate challenging conditions like significant sun glare, airborne dust, or fog. This improvement would potentially make the system safer and more reliable compared to relying solely on cameras.

But from the business perspective, when a company develops its own technology, it aims for a competitive advantage — a technological edge. If it can create a solution that is dramatically more efficient and cheaper, it opens the door to market dominance.

Tesla follows this logic. Musk doesn’t want to take the path of other companies like Volkswagen or Baidu, which have also made considerable progress. Even systems like Mobileye and iSight, installed in older cars, already demonstrate decent autonomy.

But Tesla aims to be unique — and that’s business logic. If you don’t offer something radically better, the market won’t choose you.

[celiaelise]

Was thinking about how a couple of episodes of a certain TV show may have flowed better with the scenes rearranged in a different order, and then I realized there's not a lot stopping me from downloading them and making that edit myself??

I probably won't actually do that, cause video editing is a lot of work, I'm not sure my current laptop could even handle something like that, (it's technically a tablet) and the show itself is not that important to me, but it's cool to realize that that is a project that is within my creative power to make happen!