howlongisthedata

[howlongisthedata]

P-Hacking and Causal Modeling, a Forewarning

P-Hacking

John Oliver has a great episode called "Scientific Studies: Last Week Tonight with John Oliver (HBO)" that covers p-hacking. Google AI Search Labs gives a nice summary of p-hacking as follows: "

P-hacking is a statistical practice that involves manipulating data to produce a desired p-value. It's also known as data dredging or data snooping. P-hacking can lead to the publication of false positive results. 

Here are some examples of p-hacking:

Selective reporting: Only reporting results that support a hypothesis, while ignoring those that don't 

Multiple comparisons: Conducting multiple comparisons on the same data set without adjusting for them 

Excluding participants: Excluding certain participants from the study 

P-hacking can be difficult to detect because the results can be indistinguishable from genuine studies. However, some tips for avoiding p-hacking include:

Establishing hypotheses and sample sizes before collecting data

Avoiding testing for significance multiple times on the same data set

Not choosing a subset of data to analyze after observing the results 

To prevent p-hacking, researchers can pre-register all the relevant details of their intended analysis, including the script they plan to use. They can do this using a site like OSF. 

"

Causal Modeling

Causal modeling is when you have a hypothesis and you try to prove that hypothesis with data, more specifically trying to find evidence of a causal relationship. Things that fall under the heading of causal modeling include:

A/B Testing

Propensity Score Matching

Exactly Matching

Counterfactual Causal Inference

Etc.

Forewarning

When doing causal modeling, sometimes we don't get the answer we're looking for, and so we try again. And again. And again. And eventually, we can find ourselves p-hacking or data dredging.

I think it's important to remember this caveat of causal modeling as we use these techniques. It can be a slippery slope, and it's always good to evaluate your approaches and assumptions as you move forward.

The End

Happy Learning everyone! :)

[howlongisthedata]

Advancements in AGI

François Chollet, the original creator of the deep learning framework Keras, created a dataset called ARC to assess a systems level of AGI. The ARC dataset is composed of roughly 900 puzzles to be solved, so this is not a traditional ML dataset. François Chollet also makes claims that the dataset could not be solved by LLMs. Prior to 2024, the highest score was around 30% correct and human-level performance was around 84% correct.

In this article (https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50-sota-on-arc-agi-with-gpt-4o) Ryan Greenblat discusses his roughly 70% correct solution using GPT-4o on the public validation set. Essentially he uses an AI Agent to write a unique Python program for each puzzle. The AI Agent will write a program, test it, then think about what it did wrong, then revise the code and test again.

Why is this important? And why write about it? #1) I think this isn’t as well known as it should be. #2) Personally I was someone who thought the latest advancements in LLMs hardly qualified for the expansive use of the term “AI”. Now I think the hype and use of the term “AI” is nearly appropriate. The ability for an AI Agent to write code, think about what it did wrong, revise that code, and re-test feels like a system that is approaching the functionality of our prefrontal cortex. I feel that François Chollet and other AI researchers have sort of touched on the point that we’ve never really had systems that even remotely mimicked the prefrontal cortex. But now, with AI Agents, I’m not sure we can say that anymore. I'm not saying that AI Agents can solve all problems, not in the least, but I do think that claims from Yann Lecun that auto-regressive inference is dumb … might not be true. Maybe by itself, but coupled with prompt engineering and repeated LLM calls, it seems to show its strength without a doubt.

[howlongisthedata]

If you're a data scientist, this is a must watch. Andrej Karpathy continues to be one of my favorite data scientists. In this video, he explains and walks through back-prop in the most intuitive way I've ever seen to date.

[howlongisthedata]

Very interesting overview of the limitations of Gen AI and the latest AI research from Yann LeCun

[howlongisthedata]

Nice overview of RLHF from AWS!

[howlongisthedata]

3 Lectures That Changed My Data Science Career

There is a lot of excitement around AI. Recently there has been an incredible amount of buzz around the demos of models like ChatGPT and Dall-E-2. As impressive as these systems are, I think it becomes increasingly important to keep a level head, and not get carried away in a sea of excitement.

The following videos/lectures are more focused on how to think about data science projects, and how to attack a problem. I’ve found these lectures to be highly impactful in my career and enabled me to build effective and practical solutions that fit the exact needs of the companies I’ve worked for.

Read the full article on Medium here.

[howlongisthedata]

Fantastic talk on AI Ethics, great episode!

[howlongisthedata]

Top 5 Tips for Aspiring Data Scientists Regarding Self-Study

5 lessons for aspiring data scientists who are self-studying.

Introduction

I learned these the hard way. I still live by these rules as a tenured data scientist.

To read the full article on Medium, click here.

[howlongisthedata]

Amazing overview of the Transformer architecture using Google Sheets!

[howlongisthedata]

Excellent and very quick overview of “What is a webserver?” :-D

[howlongisthedata]

1 Rule to Transition from Data Scientist to Full-Stack Data Scientist

Introduction / Disclaimer

This is primarily an opinion article, so take it with a grain of salt.

That said, this is advice that’s helped me in my career.

To hear the rule, read the full Medium article here.  Thanks and happy learning! :-)

[howlongisthedata]

Andrew Ng shares what he believes to be the skills he sees as fundamental to the next generation of machine learning practitioners - i.e., moving from model-centric to data-centric.  Excellent talk!  

[howlongisthedata]

Notable Data Science Platforms of 2021

I did a similar and more detailed article for 2020 on Medium titled Notable Data Science Platforms of 2020. This article is an extension of the 2020 article, containing updates and new platforms as of 2021!

Click here to read my follow-up article on Medium. Thanks and happy reading!

[howlongisthedata]

Excellent very short demo on deploying a Flask web app using Azure Web App service! 

[howlongisthedata]

Excellent detailed 20 min demo to Azure App Service which allows you to deploy a serverless and scalable (Kubernetes based) web application!

[howlongisthedata]

From Prototype to Production for Data Scientists

You might be thinking “yeah, I know what a prototype is, what else is there to know?” 

You might be right, but do you know some of the different methodologies and terminology? These reduce friction when you’re trying to implement something new. Do you know the difference between an ‘evolutionary’ and a ‘throwaway’ prototype? Do you know what a rollout is? Do you know the pitfalls of doing a pilot? If so, maybe you can skip this article. If not, then understanding these concepts could help your company.

To read my full article on Medium, click here and happy reading!

[howlongisthedata]

Excellent tutorial on using LIME for complicated models.  Since LIME is model agnostic, you can define any scoring function and LIME still works!  I referenced this video often in my efforts to use LIME for a complex text model (mix of LSTM and XGB models).  It took a lot of debugging, but in the end I was able to get the job done with some help from this video!