#google data API

[pixiedust111]

Yeah, I've finally made it! Though, with limited functionality, 'cause I use free accounts everywhere 😬! (I've been so stupid, I accidentally added my name to the domain 🙁)

[newsdataapi]

Get Free News API to scrape news articles

NewsData.io offered a free news API that developers could use to access news articles and headlines from various sources. This API provided endpoints for fetching news articles, headlines, and other related data. Get a Free News API Key in 3 steps:

Visit NewsData.io website

Create an account on it

Get Free API Keys from dashboard

[scrapingdog]

We scraped thousands of posts from popular subreddits to uncover real opinions, pros, and cons of moving to New York. Here's what the data tells us.

[mayurashinde]

Data Scraping Made Simple: What It Really Means

Data Scraping Made Simple: What It Really Means

In the digital world, data scraping is a powerful way to collect information from websites automatically. But what exactly does that mean—and why is it important?

Let’s break it down in simple terms.

What Is Data Scraping?

Data scraping (also called web scraping) is the process of using bots or scripts to extract data from websites. Instead of copying and pasting information manually, scraping tools do the job automatically—much faster and more efficiently.

You can scrape product prices, news headlines, job listings, real estate data, weather reports, and more.

Imagine visiting a website with hundreds of items. Now imagine a tool that can read all that content and save it in a spreadsheet in seconds. That’s what data scraping does.

Why Is It So Useful?

Businesses, researchers, and marketers use data scraping to:

Track competitors' prices

Monitor customer reviews

Gather contact info for leads

Collect news for trend analysis

Keep up with changing market data

In short, data scraping helps people get useful information without wasting time.

Is Data Scraping Legal?

It depends. Public data (like product prices or news articles) is usually okay to scrape, but private or copyrighted content is not. Always check a website’s terms of service before scraping it.

Tools for Data Scraping

There are many tools that make data scraping easy:

Beautiful Soup (for Python developers)

Octoparse (no coding needed)

Scrapy (for advanced scraping tasks)

SERPHouse APIs (for SEO and search engine data)

Some are code-based, others are point-and-click tools. Choose what suits your need and skill level.

Final Thoughts

What is data scraping? It’s the smart way to extract website content for business, research, or insights. With the right tools, it saves time, increases productivity, and opens up access to valuable online data.

Just remember: scrape responsibly.

[munaeem]

Who’s Spying on You the Most? Microsoft, Google, Nvidia, or Steam? Let’s Find Out!

Hey everyone! Let’s talk about something we all know but don’t always think about: tech companies spying on us. Yeah, it’s no secret that our devices and apps are constantly collecting data, but have you ever wondered who’s doing it the most? Is it Microsoft? Google? Nvidia? Steam? Or maybe even EA? I decided to run a little experiment to find out, and the results were… eye-opening. The…

[iwebscrapingblogs]

Google Flight API - Easy Real-Time Data Extraction

In today's fast-paced travel industry, the ability to access real-time flight data is essential for businesses and travelers alike. Whether you’re a travel agent, an airline, a developer, or simply a tech enthusiast, Google Flight API provides a powerful tool to obtain real-time flight information. This blog will delve into how you can harness the Google Flight API for easy real-time data extraction, highlighting its benefits, applications, and a step-by-step guide to get you started.

What is Google Flight API?

Google Flight API is an interface that allows developers to access and manipulate flight data provided by Google Flights. This includes information about flight schedules, prices, availability, and more. The API enables seamless integration of this data into applications, websites, and other platforms, allowing for real-time updates and a more dynamic user experience.

Benefits of Using Google Flight API

Real-Time Data: The API provides up-to-the-minute information on flight schedules, delays, and cancellations. This is crucial for travel agencies and airlines to keep their customers informed and to make timely decisions.

Wide Coverage: Google Flights covers a vast number of airlines and routes worldwide. This extensive database ensures that users have access to comprehensive flight information.

Cost-Effective: Using the API can be more cost-effective than purchasing data from third-party vendors. It reduces the need for manual data collection and entry, saving time and resources.

Customizable: The API allows for customization to fit the specific needs of your application. You can filter results based on parameters such as price range, airlines, layovers, and more.

Enhanced User Experience: Integrating real-time flight data into your platform can significantly enhance the user experience. Users can search for flights, compare prices, and make bookings without leaving your site or app.

Applications of Google Flight API

Travel Booking Platforms: Websites and apps that facilitate flight bookings can use the API to display the latest flight options, prices, and availability to their users.

Travel Agencies: Agencies can automate the process of fetching flight data, providing clients with real-time information and recommendations.

Airlines: Airlines can use the API to update their own systems and customer interfaces with the latest flight schedules and status updates.

Travel Planning Tools: Apps that help users plan their trips can integrate the API to provide accurate flight information, helping users to make informed decisions.

Data Analysis: Researchers and analysts can use the API to gather flight data for various analytical purposes, such as studying travel trends, pricing strategies, and more.

Getting Started with Google Flight API

Accessing the API: Currently, Google Flight API is not publicly available as a standalone service. However, there are ways to access similar data through other means, such as third-party APIs or scraping Google Flights with web scraping tools. Alternatively, you can explore Google's QPX Express API, which provides flight search capabilities, though it has limited availability.

API Key: If you’re using an official API or a third-party service, you’ll need to obtain an API key. This key authenticates your requests and tracks your usage.

Setting Up Your Environment: Ensure you have a development environment set up with the necessary libraries and tools for making HTTP requests and handling JSON data. Popular choices include Python with libraries like requests and json, or JavaScript with axios and fetch.

Making Requests: Use the API documentation to understand the endpoints and parameters available. Construct your HTTP requests to fetch the data you need. For example, a typical request might look like this in Python:pythonCopy codeimport requests url = "https://api.example.com/flights" params = { "origin": "JFK", "destination": "LAX", "departure_date": "2024-07-20" } headers = { "Authorization": "Bearer YOUR_API_KEY" } response = requests.get(url, headers=headers, params=params) data = response.json() print(data)

Parsing the Data: Once you receive the response, parse the JSON data to extract the information you need. This could include flight numbers, departure and arrival times, prices, and more.

Integrating the Data: Integrate the parsed data into your application or website. This might involve updating the UI with the latest flight information, storing data in a database, or performing additional processing.

Conclusion

Google Flight API, or similar services, offers a robust solution for accessing real-time flight data. Whether you’re building a travel booking platform, enhancing an airline’s digital presence, or developing a travel planning tool, the ability to fetch and display accurate flight information is invaluable. By understanding the benefits, applications, and steps to get started, you can leverage this powerful tool to create more dynamic and user-friendly experiences in the travel industry.

[foodspark-scraper]

How To Extract 1000s of Restaurant Data from Google Maps?

In today's digital age, having access to accurate and up-to-date data is crucial for businesses to stay competitive. This is especially true for the restaurant industry, where trends and customer preferences are constantly changing. One of the best sources for this data is Google Maps, which contains a wealth of information on restaurants around the world. In this article, we will discuss how to extract thousands of restaurant data from Google Maps and how it can benefit your business.

Why Extract Restaurant Data from Google Maps?

Google Maps is the go-to source for many customers when searching for restaurants in their area. By extracting data from Google Maps, you can gain valuable insights into the current trends and preferences of customers in your target market. This data can help you make informed decisions about your menu, pricing, and marketing strategies. It can also give you a competitive edge by allowing you to stay ahead of the curve and adapt to changing trends.

How To Extract Restaurant Data from Google Maps?

There are several ways to extract restaurant data from Google Maps, but the most efficient and accurate method is by using a web scraping tool. These tools use automated bots to extract data from websites, including Google Maps, and compile it into a usable format. This eliminates the need for manual data entry and saves you time and effort.

To extract restaurant data from Google Maps, you can follow these steps:

Choose a reliable web scraping tool that is specifically designed for extracting data from Google Maps.

Enter the search criteria for the restaurants you want to extract data from, such as location, cuisine, or ratings.

The tool will then scrape the data from the search results, including restaurant names, addresses, contact information, ratings, and reviews.

You can then export the data into a spreadsheet or database for further analysis.

Benefits of Extracting Restaurant Data from Google Maps

Extracting restaurant data from Google Maps can provide numerous benefits for your business, including:

Identifying Trends and Preferences

By analyzing the data extracted from Google Maps, you can identify current trends and preferences in the restaurant industry. This can help you make informed decisions about your menu, pricing, and marketing strategies to attract more customers.

Improving SEO

Having accurate and up-to-date data on your restaurant's Google Maps listing can improve your search engine optimization (SEO). This means that your restaurant will appear higher in search results, making it easier for potential customers to find you.

Competitive Analysis

Extracting data from Google Maps can also help you keep an eye on your competitors. By analyzing their data, you can identify their strengths and weaknesses and use this information to improve your own business strategies.

conclusion:

extracting restaurant data from Google Maps can provide valuable insights and benefits for your business. By using a web scraping tool, you can easily extract thousands of data points and use them to make informed decisions and stay ahead of the competition. So why wait? Start extracting restaurant data from Google Maps today and take your business to the next level.

[iwebdatascrape]

Google Shopping Data Scraper

Scrape Google Shopping data, including price, merchant name, etc., using Google Shopping data scraper. Our Google Shopping scraping tool can scrape data across the USA, UK,

[trendingnow3-blog]

YouTube Most Watched Videos Finder: Python Code

YouTube Most Watched Videos Finder: Python Code

Description: This article contains a Python script that utilizes the YouTube Data API to find the most watched videos in the past 24 hours for a specific region. The script fetches data from the YouTube API, extracts relevant information such as video title, view count, like count, and creates a list of the most popular videos in the desired region. How it Works: The script is written in…

[feminist-space]

"Artists have finally had enough with Meta’s predatory AI policies, but Meta’s loss is Cara’s gain. An artist-run, anti-AI social platform, Cara has grown from 40,000 to 650,000 users within the last week, catapulting it to the top of the App Store charts.

Instagram is a necessity for many artists, who use the platform to promote their work and solicit paying clients. But Meta is using public posts to train its generative AI systems, and only European users can opt out, since they’re protected by GDPR laws. Generative AI has become so front-and-center on Meta’s apps that artists reached their breaking point.

“When you put [AI] so much in their face, and then give them the option to opt out, but then increase the friction to opt out… I think that increases their anger level — like, okay now I’ve really had enough,” Jingna Zhang, a renowned photographer and founder of Cara, told TechCrunch.

Cara, which has both a web and mobile app, is like a combination of Instagram and X, but built specifically for artists. On your profile, you can host a portfolio of work, but you can also post updates to your feed like any other microblogging site.

Zhang is perfectly positioned to helm an artist-centric social network, where they can post without the risk of becoming part of a training dataset for AI. Zhang has fought on behalf of artists, recently winning an appeal in a Luxembourg court over a painter who copied one of her photographs, which she shot for Harper’s Bazaar Vietnam.

“Using a different medium was irrelevant. My work being ‘available online’ was irrelevant. Consent was necessary,” Zhang wrote on X.

Zhang and three other artists are also suing Google for allegedly using their copyrighted work to train Imagen, an AI image generator. She’s also a plaintiff in a similar lawsuit against Stability AI, Midjourney, DeviantArt and Runway AI.

“Words can’t describe how dehumanizing it is to see my name used 20,000+ times in MidJourney,” she wrote in an Instagram post. “My life’s work and who I am—reduced to meaningless fodder for a commercial image slot machine.”

Artists are so resistant to AI because the training data behind many of these image generators includes their work without their consent. These models amass such a large swath of artwork by scraping the internet for images, without regard for whether or not those images are copyrighted. It’s a slap in the face for artists – not only are their jobs endangered by AI, but that same AI is often powered by their work.

“When it comes to art, unfortunately, we just come from a fundamentally different perspective and point of view, because on the tech side, you have this strong history of open source, and people are just thinking like, well, you put it out there, so it’s for people to use,” Zhang said. “For artists, it’s a part of our selves and our identity. I would not want my best friend to make a manipulation of my work without asking me. There’s a nuance to how we see things, but I don’t think people understand that the art we do is not a product.”

This commitment to protecting artists from copyright infringement extends to Cara, which partners with the University of Chicago’s Glaze project. By using Glaze, artists who manually apply Glaze to their work on Cara have an added layer of protection against being scraped for AI.

Other projects have also stepped up to defend artists. Spawning AI, an artist-led company, has created an API that allows artists to remove their work from popular datasets. But that opt-out only works if the companies that use those datasets honor artists’ requests. So far, HuggingFace and Stability have agreed to respect Spawning’s Do Not Train registry, but artists’ work cannot be retroactively removed from models that have already been trained.

“I think there is this clash between backgrounds and expectations on what we put on the internet,” Zhang said. “For artists, we want to share our work with the world. We put it online, and we don’t charge people to view this piece of work, but it doesn’t mean that we give up our copyright, or any ownership of our work.”"

Read the rest of the article here:

https://techcrunch.com/2024/06/06/a-social-app-for-creatives-cara-grew-from-40k-to-650k-users-in-a-week-because-artists-are-fed-up-with-metas-ai-policies/

[newsdataapi]

If you're looking for an alternative to Google News API, the NewsData.io API is the way to go. It's got a ton of news source coverage, cool features, and a super easy-to-use interface. Plus, it's got flexible pricing plans, so you can get the most out of it. Whether you're building a news aggregation tool, a financial platform, or a social media monitoring tool - NewsData.io has got you covered.

[mostlysignssomeportents]

Too big to care

I'm on tour with my new, nationally bestselling novel The Bezzle! Catch me in BOSTON with Randall "XKCD" Munroe (Apr 11), then PROVIDENCE (Apr 12), and beyond!

Remember the first time you used Google search? It was like magic. After years of progressively worsening search quality from Altavista and Yahoo, Google was literally stunning, a gateway to the very best things on the internet.

Today, Google has a 90% search market-share. They got it the hard way: they cheated. Google spends tens of billions of dollars on payola in order to ensure that they are the default search engine behind every search box you encounter on every device, every service and every website:

https://pluralistic.net/2023/10/03/not-feeling-lucky/#fundamental-laws-of-economics

Not coincidentally, Google's search is getting progressively, monotonically worse. It is a cesspool of botshit, spam, scams, and nonsense. Important resources that I never bothered to bookmark because I could find them with a quick Google search no longer show up in the first ten screens of results:

https://pluralistic.net/2024/02/21/im-feeling-unlucky/#not-up-to-the-task

Even after all that payola, Google is still absurdly profitable. They have so much money, they were able to do a $80 billion stock buyback. Just a few months later, Google fired 12,000 skilled technical workers. Essentially, Google is saying that they don't need to spend money on quality, because we're all locked into using Google search. It's cheaper to buy the default search box everywhere in the world than it is to make a product that is so good that even if we tried another search engine, we'd still prefer Google.

This is enshittification. Google is shifting value away from end users (searchers) and business customers (advertisers, publishers and merchants) to itself:

https://pluralistic.net/2024/03/05/the-map-is-not-the-territory/#apor-locksmith

And here's the thing: there are search engines out there that are so good that if you just try them, you'll get that same feeling you got the first time you tried Google.

When I was in Tucson last month on my book-tour for my new novel The Bezzle, I crashed with my pals Patrick and Teresa Nielsen Hayden. I've know them since I was a teenager (Patrick is my editor).

We were sitting in his living room on our laptops – just like old times! – and Patrick asked me if I'd tried Kagi, a new search-engine.

Teresa chimed in, extolling the advanced search features, the "lenses" that surfaced specific kinds of resources on the web.

I hadn't even heard of Kagi, but the Nielsen Haydens are among the most effective researchers I know – both in their professional editorial lives and in their many obsessive hobbies. If it was good enough for them…

I tried it. It was magic.

No, seriously. All those things Google couldn't find anymore? Top of the search pile. Queries that generated pages of spam in Google results? Fucking pristine on Kagi – the right answers, over and over again.

That was before I started playing with Kagi's lenses and other bells and whistles, which elevated the search experience from "magic" to sorcerous.

The catch is that Kagi costs money – after 100 queries, they want you to cough up $10/month ($14 for a couple or $20 for a family with up to six accounts, and some kid-specific features):

https://kagi.com/settings?p=billing_plan&plan=family

I immediately bought a family plan. I've been using it for a month. I've basically stopped using Google search altogether.

Kagi just let me get a lot more done, and I assumed that they were some kind of wildly capitalized startup that was running their own crawl and and their own data-centers. But this morning, I read Jason Koebler's 404 Media report on his own experiences using it:

https://www.404media.co/friendship-ended-with-google-now-kagi-is-my-best-friend/

Koebler's piece contained a key detail that I'd somehow missed:

When you search on Kagi, the service makes a series of “anonymized API calls to traditional search indexes like Google, Yandex, Mojeek, and Brave,” as well as a handful of other specialized search engines, Wikimedia Commons, Flickr, etc. Kagi then combines this with its own web index and news index (for news searches) to build the results pages that you see. So, essentially, you are getting some mix of Google search results combined with results from other indexes.

In other words: Kagi is a heavily customized, anonymized front-end to Google.

The implications of this are stunning. It means that Google's enshittified search-results are a choice. Those ad-strewn, sub-Altavista, spam-drowned search pages are a feature, not a bug. Google prefers those results to Kagi, because Google makes more money out of shit than they would out of delivering a good product:

https://www.theverge.com/2024/4/2/24117976/best-printer-2024-home-use-office-use-labels-school-homework

No wonder Google spends a whole-ass Twitter every year to make sure you never try a rival search engine. Bottom line: they ran the numbers and figured out their most profitable course of action is to enshittify their flagship product and bribe their "competitors" like Apple and Samsung so that you never try another search engine and have another one of those magic moments that sent all those Jeeves-askin' Yahooers to Google a quarter-century ago.

One of my favorite TV comedy bits is Lily Tomlin as Ernestine the AT&T operator; Tomlin would do these pitches for the Bell System and end every ad with "We don't care. We don't have to. We're the phone company":

https://snltranscripts.jt.org/76/76aphonecompany.phtml

Speaking of TV comedy: this week saw FTC chair Lina Khan appear on The Daily Show with Jon Stewart. It was amazing:

https://www.youtube.com/watch?v=oaDTiWaYfcM

The coverage of Khan's appearance has focused on Stewart's revelation that when he was doing a show on Apple TV, the company prohibited him from interviewing her (presumably because of her hostility to tech monopolies):

https://www.thebignewsletter.com/p/apple-got-caught-censoring-its-own

But for me, the big moment came when Khan described tech monopolists as "too big to care."

What a phrase!

Since the subprime crisis, we're all familiar with businesses being "too big to fail" and "too big to jail." But "too big to care?" Oof, that got me right in the feels.

Because that's what it feels like to use enshittified Google. That's what it feels like to discover that Kagi – the good search engine – is mostly Google with the weights adjusted to serve users, not shareholders.

Google used to care. They cared because they were worried about competitors and regulators. They cared because their workers made them care:

https://www.vox.com/future-perfect/2019/4/4/18295933/google-cancels-ai-ethics-board

Google doesn't care anymore. They don't have to. They're the search company.

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2024/04/04/teach-me-how-to-shruggie/#kagi

[mayurashinde]

Web Scraping in Java: A Powerful Approach to Data Extraction

Web Scraping in Java enables developers to collect and process web data efficiently. With libraries like JSoup and Selenium, Java simplifies handling HTML parsing, automating browsers, and extracting real-time information. Java offers a scalable and reliable scraping solution for market research, SEO, or analytics.

[jcmarchi]

10 Best Free Social Media Plugins for WordPress – Speckyboy

New Post has been published on https://thedigitalinsider.com/10-best-free-social-media-plugins-for-wordpress-speckyboy/

10 Best Free Social Media Plugins for WordPress – Speckyboy

WordPress and social media are a powerful combination. And there are plenty of ways to integrate platforms like Facebook, Instagram, X, or TikTok into your website. You can display social content on your site, allow users to share your content, or add functionality like social logins.

The social platform landscape is also changing. The Fediverse makes it easier to integrate with services like Mastodon or Bluesky. It allows you to publish in multiple places via a unified protocol. The WordPress ecosystem has embraced this model.

Social media’s importance is evident in the number of available plugins. You’ll find options for top platforms that offer a variety of functions. There’s something available for every niche.

With that in mind, let’s look at some top social media plugins for WordPress!

ActivityPub turns your WordPress website into a federated profile. It publishes your content to multiple platforms that support the Fediverse (Mastodon, among others). Followers will then see your blog posts in their feeds. The neat aspect is that your profile name is consistent across federated platforms.

Users follow you on the service of their choice and enjoy your content. It’s a great way to nurture an online community.

This plugin solves a longstanding problem for those posting to multiple social networks. It combines your Facebook, Instagram, Pinterest, and Twitter/X profiles into an attractive grid layout embedded on your site. It creates better awareness of your profiles and ensures users won’t miss anything.

AddToAny is a longtime favorite that makes it easy for visitors to share your content. It works with every social platform you can imagine – there’s also an option to share via email. In addition, you can customize the design and location of the plugin’s sharing UI. This plugin is perfect for taking full control of social sharing.

Open Graph is a protocol for sharing rich content on social platforms like Facebook and Twitter/X. This plugin allows you to customize the Open Graph tags generated by your content.

Thus, content looks how you want when embedded into social media. The plugin works with all WordPress content and integrates with WooCommerce, Yoast SEO, and others.

Posting links to your content across multiple platforms is a pain. Blog2Social makes it easier by automating the process.

The free version works with Bluesky, Facebook, LinkedIn, Threads, and more. There’s also an add-on that works with Twitter/X and a premium version with advanced features. It turns a tedious task into a piece of cake.

WP Social is a suite of tools for adding social media to your WordPress website. First, you’ll find sharing buttons that work with a selection of social platforms. You can customize the layout and services displayed.

A social counter shows how many times your content was shared on a particular service. Lastly, a social login/registration function lets users create an account using one of nine compatible services (Facebook, GitHub, Google, etc.). It may be the only social plugin you need.

You can use Jetpack Social to automate content sharing and increase engagement. Share posts and WooCommerce products automatically and use the included scheduling tools to choose the perfect timing.

The plugin integrates with multiple services and adds helpful tools to the Block Editor. If you have a Jetpack Social plan, you’ll have access to the social image generator feature.

Here’s a plugin that helps spread the word about your WooCommerce products. It adds a custom set of social sharing icons to individual product pages.

Here are options for icon placement, shape, and order. There’s also a handy “copy to clipboard” option so users can grab your product’s URL. The pro version includes a “floating” icon bar, tooltips, and more.

Video creators will love Feeds for YouTube. It’s a tool for easily embedding a YouTube channel into your WordPress website. You can customize the look of your feed and even display multiple feeds.

The plugin will check for new videos hourly (or choose a custom interval), so your site will always show the latest content. The Pro version adds compatibility with live broadcasts via the YouTube Live API.

Use AdTribes to expand the reach of your WooCommerce store. The plugin creates product feeds that integrate with services like Facebook and Google. Filtering options let you choose which products and what product data is included in your feed. Now, customers can view your products right from their social feeds.

Seamlessly Add Social Media to WordPress

Integrating social media into your website has real benefits. For example, embedding a feed adds dynamic content and builds awareness. Even better is that feeds are relatively simple to set up. From there, it’s smooth sailing.

The other big benefit is the opportunity to add functionality. Social logins make it easier for users to register for your site. And product feeds entice users to visit your online shop.

The idea is to reach people where they are. The options above (and the untold number of other social plugins) help you do so. The possibilities are almost limitless.

Social Media WordPress Plugin FAQs

What Are Social Media Plugins for WordPress?

They’re plugins that allow you to connect your WordPress site with various social media platforms. They offer sharing buttons, auto-posting to social media, and allowing you to display your social feeds on your site.

Who Should Use Social Media Plugins?

They’re perfect for website owners who want to improve their social media presence, like bloggers, businesses, and online retailers, or anyone looking to increase their site’s engagement through social media.

Can Social Media Plugins Help in Growing My Site’s Audience?

Yes, by making sharing effortless and displaying your social activity, these plugins can attract more visitors and encourage them to interact with your content.

More Essential Free WordPress Plugins

Related Topics

[rythyme]

Introducing the Thai Drama AO3 Trends Dashboard! (Beta) 🇹🇭

Over the last several weeks or so I've been building an auto-scraping setup to get AO3 stats on Thai Drama fandoms. Now I finally have it ready to share out!

Take a look if you're interested and let me know what you think :)

(More details and process info under the cut.)

Main Features

This dashboard pulls in data about the quantity of Thai Drama fics over time.

Using filters, it allows you to break that data down by drama, fandom size, air date, and a select number of MyDramaList tags.

You can also see which fandoms have had the most new fics added on a weekly basis, plus the growth as a percentage of the total.

My hope is that this will make it easier to compare Thai Drama fandoms as a collective and pick out trends that otherwise might be difficult to see in an all-AO3 dataset.

Process

Okay -- now for the crunchy stuff...

Scraping 🔎

Welcome to the most over-complicated Google Sheets spreadsheet ever made.

I used Google Sheets formulas to scrape certain info from each Thai Drama tag, and then I wrote some app scripts to refresh the data once a day. There are 5 second breaks between the refreshes for each fandom to avoid overwhelming AO3's servers.

Archiving 📁

Once all the data is scraped, it gets transferred to a different Archive spreadsheet that feeds directly into the data dashboard. The dashboard will update automatically when new data is added to the spreadsheet, so I don't have to do anything manually.

Show Metadata 📊

I decided to be extra and use a (currently unofficial) MyDramaList API to pull in data about each show, such as the year it came out and the MDL tags associated with it. Fun! I might pull in even more info in the future if the mood strikes me.

Bonus - Pan-Fandom AO3 Search

Do you ever find it a bit tedious to have like, 15 different tabs open for the shows you're currently reading fic for?

While making this dash, I also put together this insane URL that basically serves as a "feed" for any and all new Thai drama fics. You can check it out here! It could be useful if you like checking for new fics in multiple fandoms at once. :)

Other Notes

Consider this dashboard the "beta" version -- please let me know if you notice anything that looks off. Also let me know if there are any fandoms missing! Thanks for checking it out!

The inspiration for this dashboard came from @ao3-anonymous 's AO3 Fandom Trend Analysis Dashboard, which I used as a jumping off point for my own data dash. Please give them some love <3