#Travel websites scraping

[xbytecrawling]

Master web scraping techniques to extract flight schedules and prices from airline websites with this comprehensive guide.

For More Information:-

[therobotmonster]

On that recent Disney Vs Midjourney court thing wrt AI, how strong do you think their case is in a purely legal sense, what do you think MJ's best defenses are, how likely is Disney to win, and how bad would the outcome be if they do win?

Oh sure, ask an easy one.

In a purely legal sense, this case is very questionable.

Scraping as fair use has already been established when it comes to text in legal cases, and infringement is based on publication, not inspiration. There's also the question of if Midjourney would be responsible for their users' creations under safe harbor provisions, or even basic understanding of what an art tool is. Adobe isn't responsible for the many, many illegal images its software is used to make, after all.

The best defense, I would say, is the fair use nature of dataset training and the very nature of transformative work, which is protected, requires the work-to-be-transformed is involved. Disney's basic approach of 'your AI knows who our characters are, so that proves you stole from us' would render fair use impossible.

I don't think its likely for Disney to win, but the problem with civil action is proof isn't needed, just convincing. Bad civil cases happen all the time, and produce case law. Which is what Disney is trying to do here.

If Disney wins, they'll have pulled off a coup of regulatory capture, basically ensuring that large media corporations can replace their staff with robots but that small creators will be limited to underpowered models to compete with them.

Worse, everything that is a 'smoking gun' when it comes to copyright infringement on Midjourney? That's fan art. All that "look how many copyrighted characters they're using-" applies to the frontpage of Deviantart or any given person's Tumblr feed more than to the featured page of Midjourney.

Every single website with user-generated content it chock full of copyright infringement because of fan art and fanfic, and fair use arguments are far harder to pull out for fan-works. The law won't distinguish between a human with a digital art package and a human with an AI art package, and any win Disney makes against MJ is a win against Artstation, Deviantart, Rule34.xxx, AO3, and basically everyone else.

"We get a slice of your cheese if enough of your users post our mouse" is not a rule you want in law.

And the rules won't be enforced by a court 9/10 times. Even if your individual work is plainly fair use, it's not going to matter to whatever image-based version of youtube's copyreich bots gets applied to Artstation and RedBubble to keep the site owners safe.

Even if you're right, you won't have the money to fight.

Heck, Adobe already spies on what you make to report you to the feds if you're doing a naughty, imagine it's internal watchdogs throwing up warnings when it detects you drawing Princess Jasmine and Ariel making out. That may sound nuts, but it's entirely viable.

And that's just one level of possible nightmare. If the judgement is broad enough, it could provide a legal pretext for pursuing copyright lawsuits over style and inspiration. Given how consolidated IP is, this means you're going to have several large cabals that can crush any new work that seems threatening, as there's bound to be something they can draw a connection to.

If you want to see how utterly stupid inspiration=theft is, check out when Harlan Ellison sued James Cameron over Terminator because Cameron was dumb enough to say he was inspired by Demon with a Glass Hand and Soldier from the Outer Limits.

Harlan was wrong on the merits, wrong ethically, and the case shouldn't have been entertained in the first place, but like I said, civil law isn't about facts. Cameron was honest about how two episodes of a show he saw as a kid gave him this completely different idea (the similarities are 'robot that looks like a guy with hand reveal' and 'time traveling soldier goes into a gun store and tries to buy future guns'), and he got unjustly sued for it.

If you ever wonder why writers only talk about their inspirations that are dead, that's why. Anything that strengthens the "what goes in" rather than the "what goes out" approach to IP is good for corps, bad for culture.

[mariacallous]

When tech companies first rolled out generative-AI products, some critics immediately feared a media collapse. Every bit of writing, imagery, and video became suspect. But for news publishers and journalists, another calamity was on the horizon.

Chatbots have proved adept at keeping users locked into conversations. They do so by answering every question, often through summarizing articles from news publishers. Suddenly, fewer people are traveling outside the generative-AI sites—a development that poses an existential threat to the media, and to the livelihood of journalists everywhere.

According to one comprehensive study, Google’s AI Overviews—a feature that summarizes web pages above the site’s usual search results—has already reduced traffic to outside websites by more than 34 percent. The CEO of DotDash Meredith, which publishes People, Better Homes & Gardens, and Food & Wine, recently said the company is preparing for a possible “Google Zero” scenario. Some have speculated that traffic drops resulting from chatbots were part of the reason outlets such as Business Insider and the Daily Dot have recently had layoffs. “Business Insider was built for an internet that doesn’t exist anymore,” one former staffer recently told the media reporter Oliver Darcy.

Not all publishers are at equal risk: Those that primarily rely on general-interest readers who come in from search engines and social media may be in worse shape than specialized publishers with dedicated subscribers. Yet no one is totally safe. Released in May 2024, AI Overviews joins ChatGPT, Claude, Grok, Perplexity, and other AI-powered products that, combined, have replaced search for more than 25 percent of Americans, according to one study. Companies train chatbots on huge amounts of stolen books and articles, as my previous reporting has shown, and scrape news articles to generate responses with up-to-date information. Large language models also train on copious materials in the public domain—but much of what is most useful to these models, particularly as users seek real-time information from chatbots, is news that exists behind a paywall. Publishers are creating the value, but AI companies are intercepting their audiences, subscription fees, and ad revenue.

I asked Anthropic, xAI, Perplexity, Google, and OpenAI about this problem. Anthropic and xAI did not respond. Perplexity did not directly comment on the issue. Google argued that it was sending “higher-quality” traffic to publisher websites, meaning that users purportedly spend more time on the sites once they click over, but declined to offer any data in support of this claim. OpenAI referred me to an article showing that ChatGPT is sending more traffic to websites overall than it did previously, but the raw numbers are fairly modest. The BBC, for example, reportedly received 118,000 visits from ChatGPT in April, but that’s practically nothing relative to the hundreds of millions of visitors it receives each month. The article also shows that traffic from ChatGPT has in fact declined for some publishers.

Over the past few months, I’ve spoken with several news publishers, all of whom see AI as a near-term existential threat to their business. Rich Caccappolo, the vice chair of media at the company that publishes the Daily Mail—the U.K.’s largest newspaper by circulation—told me that all publishers “can see that Overviews are going to unravel the traffic that they get from search, undermining a key foundational pillar of the digital-revenue model.” AI companies have claimed that chatbots will continue to send readers to news publishers, but have not cited evidence to support this claim. I asked Caccappolo if he thought AI-generated answers could put his company out of business. “That is absolutely the fear,” he told me. “And my concern is it’s not going to happen in three or five years—I joke it’s going to happen next Tuesday.”

Book publishers, especially those of nonfiction and textbooks, also told me they anticipate a massive decrease in sales, as chatbots can both summarize their books and give detailed explanations of their contents. Publishers have tried to fight back, but my conversations revealed how much the deck is stacked against them. The world is changing fast, perhaps irrevocably. The institutions that comprise our country’s free press are fighting for their survival.

Publishers have been responding in two ways. First: legal action. At least 12 lawsuits involving more than 20 publishers have been filed against AI companies. Their outcomes are far from certain, and the cases might be decided only after irreparable damage has been done.

The second response is to make deals with AI companies, allowing their products to summarize articles or train on editorial content. Some publishers, such as The Atlantic, are pursuing both strategies (the company has a corporate partnership with OpenAI and is suing Cohere). At least 72 licensing deals have been made between publishers and AI companies in the past two years. But figuring out how to approach these deals is no easy task. Caccappolo told me he has “felt a tremendous imbalance at the negotiating table”—a sentiment shared by others I spoke with. One problem is that there is no standard price for training an LLM on a book or an article. The AI companies know what kinds of content they want, and having already demonstrated an ability and a willingness to take it without paying, they have extraordinary leverage when it comes to negotiating. I’ve learned that books have sometimes been licensed for only a couple hundred dollars each, and that a publisher that asks too much may be turned down, only for tech companies to take their material anyway.

Another issue is that different content appears to have different value for different LLMs. The digital-media company Ziff Davis has studied web-based AI training data sets and observed that content from “high-authority” sources, such as major newspapers and magazines, appears more desirable to AI companies than blog and social-media posts. (Ziff Davis is suing OpenAI for training on its articles without paying a licensing fee.) Researchers at Microsoft have also written publicly about “the importance of high-quality data” and have suggested that textbook-style content may be particularly desirable.

But beyond a few specific studies like these, there is little insight into what kind of content most improves an LLM, leaving a lot of unanswered questions. Are biographies more or less important than histories? Does high-quality fiction matter? Are old books worth anything? Amy Brand, the director and publisher of the MIT Press, told me that “a solution that promises to help determine the fair value of specific human-authored content within the active marketplace for LLM training data would be hugely beneficial.”

A publisher’s negotiating power is also limited by the degree to which it can stop an AI company from using its work without consent. There’s no surefire way to keep AI companies from scraping news websites; even the Robots Exclusion Protocol, the standard opt-out method available to news publishers, is easily circumvented. Because AI companies generally keep their training data a secret, and because there is no easy way for publishers to check which chatbots are summarizing their articles, publishers have difficulty figuring out which AI companies they might sue or try to strike a deal with. Some experts, such as Tim O’Reilly, have suggested that laws should require the disclosure of copyrighted training data, but no existing legislation requires companies to reveal specific authors or publishers that have been used for AI training material.

Of course, all of this raises a question. AI companies seem to have taken publishers’ content already. Why would they pay for it now, especially because some of these companies have argued in court that training LLMs on copyrighted books and articles is fair use?

Perhaps the deals are simply hedges against an unfavorable ruling in court. If AI companies are prevented from training on copyrighted work for free, then organizations that have existing deals with publishers might be ahead of their competition. Publisher deals are also a means of settling without litigation—which may be a more desirable path for publishers who are risk-averse or otherwise uncertain. But the legal scholar James Grimmelmann told me that AI companies could also respond to complaints like Ziff Davis’s by arguing that the deals involve more than training on a publisher’s content: They may also include access to cleaner versions of articles, ongoing access to a daily or real-time feed, or a release from liability for their chatbot’s plagiarism. Tech companies could argue that the money exchanged in these deals is exclusively for the nonlicensing elements, so they aren’t paying for training material. It’s worth noting that tech companies almost always refer to these deals as partnerships, not licensing deals, likely for this reason.

Regardless, the modest income from these arrangements is not going to save publishers: Even a good deal, one publisher told me, won’t come anywhere near recouping the revenue lost from decreased readership. Publishers that can figure out how to survive the generative-AI assault may need to invent different business models and find new streams of revenue. There may be viable strategies, but none of the publishers I spoke with has a clear idea of what they are.

Publishers have become accustomed to technological threats over the past two decades, perhaps most notably the loss of ad revenue to Facebook and Google, a company that was recently found to have an illegal monopoly in online advertising (though the company has said it will appeal the ruling). But the rise of generative AI may spell doom for the Fourth Estate: With AI, the tech industry even deprives publishers of an audience.

In the event of publisher mass extinction, some journalists will be able to endure. The so-called creator economy shows that it’s possible to provide high-quality news and information through Substack, YouTube, and even TikTok. But not all reporters can simply move to these platforms. Investigative journalism that exposes corruption and malfeasance by powerful people and companies comes with a serious risk of legal repercussions, and requires resources—such as time and money—that tend to be in short supply for freelancers.

If news publishers start going out of business, won’t AI companies suffer too? Their chatbots need access to journalism to answer questions about the world. Doesn’t the tech industry have an interest in the survival of newspapers and magazines?

In fact, there are signs that AI companies believe publishers are no longer needed. In December, at The New York Times’ DealBook Summit, OpenAI CEO Sam Altman was asked how writers should feel about their work being used for AI training. “I think we do need a new deal, standard, protocol, whatever you want to call it, for how creators are going to get rewarded.” He described an “opt-in” regime where an author could receive “micropayments” when their name, likeness, and style were used. But this could not be further from OpenAI’s current practice, in which products are already being used to imitate the styles of artists and writers, without compensation or even an effective opt-out.

Google CEO Sundar Pichai was also asked about writer compensation at the DealBook Summit. He suggested that a market solution would emerge, possibly one that wouldn’t involve publishers in the long run. This is typical. As in other industries they’ve “disrupted,” Silicon Valley moguls seem to perceive old, established institutions as middlemen to be removed for greater efficiency. Uber enticed drivers to work for it, crushed the traditional taxi industry, and now controls salaries, benefits, and workloads algorithmically. This has meant greater convenience for consumers, just as AI arguably does—but it has also proved ruinous for many people who were once able to earn a living wage from professional driving. Pichai seemed to envision a future that may have a similar consequence for journalists. “There’ll be a marketplace in the future, I think—there’ll be creators who will create for AI,” he said. “People will figure it out.”

[mushiemellows]

On Rivd and AI

So last night I made this post and said I'd elaborate more in the morning and when I had the time to do a bit of research. Upon doing said research, I realized that I had misunderstood the concerns being raised with the Rivd situation, but that isn't the case any more. However, some of my thoughts on ai still stand. Heads up, this is going to be a long post. Some actual proper blogging for once wow.

I'm going to discuss the Rivd phishing scam, what can be done for fic writers as ai begins to invade fan spaces, and my elaborated thoughts on Language Learning Models. Warning for transparency: I did utilize chat gpt for this post, NOT for the text itself but to provide examples of the current state of LLMs. Some articles I link to will also be ai generated, and their generated quality is part of what I'll be warning about. This is not a generated post and you can tell because I've got those nifty writing things called "voice" and "style."

ANYWAYS:

Okay so what was the Rivd situation? So two days ago this post was uploaded on tumblr, linking back to a twitter thread on the same topic. I saw it late last night because I was traveling. A reddit post was also uploaded 3 days ago. According to google trends, there was a slight uptick in search traffic the week of June 23rd, and a more severe uptick last week (June 30th-July 6th). That's all to say, this website did not exist until last week, caused a stir, and immediately was put down.

Rivd is not longer up. Enough people contacted its web hosting service Cloudflare and they took the site down. This happened yesterday, from the looks of it.

So, then, what was Rivd? And more importantly, what was the point of scraping a chunk of ao3 and re-uploading it? There seems to be 2 possible theories.

1) The more innocent of the two: they genuinely want to be an ao3 competitor. I can't look at the website any more, and very little positive results appear when googled, but I did find one ai-generated puff piece called "Exploring Rivd: The Premier Platform for Movie-Based Fanfiction" posted to Medium last week by one "Steffen Holzmann" (if that is your real name... x to doubt). This account appeared the same week that Rivd had that first little uptick in google queries, so it is undoubtedly made by the people running the website themselves to create an air of legitimacy. Medium appears to be a safe enough website that you can click that link if you really want to, but you shouldn't. It's a bad generated article, there's very little to glean from it. But it is a remnant source on what Rivd was claiming to be, before it was taken down. Here's the conclusion from the article, the only portion that gave any actual information (and it barely offers any):

Rivd is the ultimate platform for movie-based fanfiction, offering a diverse range of content, a supportive community, and robust interactive features. Whether you’re a writer looking to share your work or a reader seeking new adventures in your favorite movie universes, Rivd provides the perfect platform to engage with a passionate and creative community. Start your journey on Rivd today and immerse yourself in the world of fanfiction.

There's a second article by Holzmann titled "Mastering the Art of Fanfiction Writing in 2024" that's essentially similar ai bull, but trades explaining that fans can write Star Wars fic for explaining that you can make OC's and maybe get a beta (not that that's advice I've ever heeded. Beta? Not in this house we don't.) This was posted six days ago and similarly spends half the time jerking Rivd off. That's all to say, if they are to be believed at face value, this website wanted to just be a fic hosting site. Scraping Ao3 would have made it seem like there was already an active user base for anyone they were attempting to attract, like buying your first 50,000 instagram followers. Anyone actually looking to use this as a fic site would have quickly realized that there's no one on it and no actual fan engagement. There's already fan community spaces online. This website offers nothing ao3 or ffn or wattpad or livejournal or tumblr or reddit didn't already.

Similarly, it reeks of tech bro. Between the scraping and the ai articles, the alarms are already going off. According to that Reddit thread, they were based out of Panama, though that doesn't mean much other than an indicator that these are the type of people to generate puff articles and preemptively base their business off-shore. Holzmann, it should be mentioned, also only has 3 followers, which means my tiny ass blog already has more reach than him. Don't go following that guy. The two comments on the first article are also disparaging of Rivd. This plan didn't work and was seen right through immediately.

If fan communities, and those who write fic know anything, it's how to sniff out when someone isn't being genuine. People write fic for the love of the game, at least generally. It's a lot of work to do for free, and it's from a place of love. Ao3 is run on volunteers and donations. If this genuinely is meant to be a business bro website to out-compete ao3, then they will be sorely disappointed to learn that there's no money in this game. It would be short lived anyway. A website like this was never going to work, or if it was, it would need to ban all copyrighted and explicit materials. You know, the pillars of fic.

So then what was the point of all of this? Unless there was a more nefarious plan going on.

2) Rivd was a phishing scam. This is so so so much more likely. The mark for the scam isn't fic readers, it's fic writers. Here's how it works: they scrape a mass of ao3 accounts for their stories, you catch it, you enter a lengthy form with personal info like your full name and address etc. requesting they take your work down, they sell your data. Yes, a lot of personal info is required to take copyrighted materials down on other sites, too. That's what makes it a good scam. Fic already sits in a legal grey area (you have a copyright over your fic but none of the characters/settings/borrowed plot within it. You also CANNOT make money off of fic writing). So the site holds your works ransom, and you can't go to Marvel or Shueisha or fuck it the ghost of Ann Rice herself to deal with this on your behalf. Thankfully, enough people were able to submit valid DMCA's to Cloudflare to deal with the issue from the top.

Remember this resolution for the next time this situation arises (because of course there will be a next time). Go through higher means rather than the site itself. These scams are only getting more bold. Me personally? I'm not going to give that amount of personal info to a website that shady. Be aware of the warning signs for phishing attacks. Unfortunately, a lot of the resources online are still around text/email phishing. We live in a time where there's legal data harvesting and selling, and illegal data harvesting and selling, and the line in between the two is thin and blurry. Here's an FTC article on the signs of phishing scams, but again, it's more about emails.

I should note, I do not think that Rivd is connected to the ransomware virus of the same name that popped up two or three years ago [link is to Rivd page on PCrisk, a cypersecurity/anti-malware website]. It's probably just coincidence.... but even so. A new business priding itself on SEO and all that tech guy crap should know not to name itself the same thing as a literal virus meant to scam out out of a thousand dollars.

That's all to say, this was absolutely a scam meant to take personal info from ao3 writers. And that blows. To love art and writing and creation so much just to have your works held hostage over data feels really bad. Of course it does!

So, should you lock down your ao3 account?

That, to me, is a little trickier. You can do as you please, of course, and you should do what makes you feel safest. Me personally, though, I don't plan on it. I really, really like that guests can interact with my work from the outside. Ao3 still uses an invite system, so a lot of regular users still don't have accounts for any number of reasons. I read a lot of the time logged out anyways. I started writing again last year after all the info on the ao3 Language Learning Model training had already come out. Like I talked about in my last post, I set out to write things that a computer couldn't produce. (from here on out, I'm going to be using One Piece fic as an example, simply because that's what I write. It's also in a really prime position for a lot of ai discussion due to the size of the fandom and the size of the original work)

I'm going to be honest, I'm not all that afraid of my work being scraped for LLMs to train off of. Text generators have existed for a decade now, but they really popped off last year. LLM's are at a point, though, where the tech is sort of stagnating after the initial boom. The flaws of current ai chat bots are so baked into the flaws of ai itself that they won't be ironed out soon. They make up info, they're easily manipulated, they simply are not people. The flaw here is deep- you can trace it to the fact that we don't know enough about neuroscience itself, and even then neuroscientists don't make ai. Ai programers do. Complex info already has to make one synaptic jump without losing any understanding from person to person, and then it has to be perfectly abstracted for a machine to comprehend. It can't be done with the tools we currently have. I DO hate to admit it, but there are some decent uses for ai. if you are specifically a python programmer. The rest of us? Maybe not.

I don't like generated fic, I don't read it on principle. Checking on ao3, though, there's only 4 total generated op fics (3 in spanish, one in english, 3 xreader, 1 zosan but 2nd person zosan? and the author edited the framework the computer generated text. I still didn't enjoy the fic, tho.) But that breaks down to .005% of all op fics. I went into chat gpt and asked it to generate a drabble for me based on One Piece, and this is what it wrote:

[Image ID: A screenshot of chat GPT. The first line shows my prompt. It reads: You- write me a short 100 word fanfiction based on the manga One Piece. Next line, Chat GPT- "On the shores of an uncharted island, the Straw Hat Pirates stumbled upon a hidden treasure trove. Among the glittering gold and jewels, Luffy discovered an old, weathered journal. Inside, he found sketches and notes detailing the adventures of a legendary pirate who once roamed the Grand Line. Excited by the prospect of a new adventure, the crew set sail. following clues left in the journal. Along the way, they faced formidable foes and forged new alliances, unraveling the mysteries of the lost pirate's legacy. With each island they conquered, they grew closer to uncovering the ultimate treasure- a legacy that would shape their own journey." End ID]

Okay so like, this blows, yeah? 75,000 One Piece fics to train off of on ao3 and it still can't like, write a decent little story. This reads more like an episode summary or a book dust jacket blurb. Drabbles have an internal structure that make them interesting, much like a single sentence short story. You have to get very baby shoes never worn with them, yeah? Ai doesn't understand that, it just knows how many words it has to generate. It doesn't have any sense for creative language, metaphor, simile, poetry. I asked it to write me a poem and it wrote me the same poem with thesaurus replacements each time it re-generated one. A lot about dreams and stars and midnight and whispers, over and over again. The same rhyme scheme and cadence too.

It's also- vitally- so unhorny that it's almost a crime. For a test, I asked it to generate a story about a uhhhhh robot and an archeologist kissing. Just to see what it would give me. I ran 4 tests, a robot falling in love with an archeologist, a robot kissing an archeologist, an archeologist kissing a robot, and an archeologist falling in love with a robot. All 4 were incredibly chaste, quiet little drabbles about the innocence of discovery or whatever the fuck. Here's one of them:

[Image ID: A screenshot of chat GPT. The first line shows my prompt. It reads: You- write me a short 100 word story about an archeologist kissing a robot. Next line, Chat GPT- "Int he dimly lit chamber, Professor Evelyn Porter stared in awe at the ancient artifact- a remarkably lifelike robot buried for millennia. As she leaned closer to examine its intricacies, a surge of curiosity and admiration swept over her. In a moment of spontaneous fascination, she brushed her lips against the cool metal of its cheek, a gesture of reverence and connection across ages. The robot, dormant for eons, stirred faintly in response to the unexpected contact. It was fleeting, almost surreal exchange between past and future, bridging the gap between human exploration and technological marvels of ancient civilizations." End ID]

So like, some of these phrases gets to what's vaguely intriguing about this dynamic, sure. But at the end of the day it's still uhhhh chaste as hell. A reverent kiss??? Not what I want in my fic!!!! This is all to say, LLM's can scrape ao3 all they want, that doesn't mean they can USE what they're stealing. The training wheels have to stay on for corporate palatability. I'm stealing, I'm taking these dolls out of Shueisha's grubby hands and I'm making them sexy kiss for FREE.

In my opinion, the easiest way to keep your work out of the hands of ai is to write something they cannot use. If the grey area of copyright lies in how much is transformed, then motherfucking TRANSFORM it. Write incomprehensible smut. Build surreal worlds. Write poems and metaphors and flush out ideas that a computer could never dream of. Find niches. Get funky with it. Take it too far. and then take it even farther. Be a little freaking weirdo, you're already writing fic so then why be normal about it, you know? Even if they rob you, they can't use it. Like fiber in the diet, undigestible. Make art, make magic.

Even so, I don't mind if the computer keeps a little bit of my art. If you've ever read one of my fics, and then thought about it when you're doing something else or listening to a song or reading a book, that means something I made has stuck with you just a little bit. That;'s really cool to me, I don't know you but I live in your brain. I've made you laugh or cry or c** from my living room on the other side of the world without knowing it. It's part of why I love to write. In all honesty, I don't mind if a computer "reads" my work and a little bit of what I created sticks with it. Even if it's more in a technical way.

Art, community, fandom- they're all part of this big conversation about the world as we experience it. The way to stop websites like Rivd is how we stopped it this week. By talking to each other, by leaning on fan communities, by sending a mass of DMCA's to web host daddy. Participation in fandom spaces keeps the game going, reblogging stuff you like and sending asks and having fun and making art is what will save us. Not to sound like a sappy fuck, but really caring about people and the way we all experience the same art but interpret it differently, that's the heart of the whole thing. It's why we do this. It's meant to be fun. Love and empathy and understanding is the foundation. Build from there. Be confident in the things you make, it's the key to having your own style. You'll find your people. You aren't alone, but you have to also be willing to toss the ball back and forth with others. It takes all of us to play, even if we look a little foolish.

[meret118]

The top five targeted industries are technology (Bad Bots comprise 76% of its internet traffic); gaming (29% of traffic); social media (46%), e-commerce (65%), and financial services (45%). If a bot fails in its purpose, there is a growing tendency for the criminals to switch to human operated fraud farms. Arkose estimates there were more than 3 billion fraud farm attacks in H1 2023. These fraud farms appear to be located primarily in Brazil, India, Russia, Vietnam, and the Philippines.

The growth in the prevalence of Bad Bots is likely to increase for two reasons: the arrival and general availability of artificial intelligence (primarily gen-AI), and the increasing business professionalism of the criminal underworld with new crime-as-a-service (CaaS) offerings.

From Q1 to Q2, intelligent bot traffic nearly quadrupled. “Intelligent [bots] employ sophisticated techniques like machine learning and AI to mimic human behavior and evade detection,” notes the report (PDF). “This makes them skilled at adaptation as they target vulnerabilities in IoT devices, cloud services, and other emerging technologies.” They are widely used, for example, to circumvent 2FA defense against phishing.

Separately, the rise of artificial intelligence may or may not relate to a dramatic rise in ‘scraping’ bots that gather data and images from websites. From Q1 to Q2, scraping increased by 432%. Scraping social media accounts can gather the type of personal data that can be used by gen-AI to mass produce compelling phishing attacks. Other bots could then be used to deliver account takeover emails, romance scams, and so on. Scraping also targets the travel and hospitality sectors.

More at the link.

[mostlysignssomeportents]

This day in history

On SEPTEMBER 24th, I'll be speaking IN PERSON at the BOSTON PUBLIC LIBRARY!

#20yrsago AnarchistU, Toronto’s wiki-based free school https://web.archive.org/web/20040911010603/http://anarchistu.org/bin/view/Anarchistu

#20yrsago Fair use is a right AND a defense https://memex.craphound.com/2004/09/09/fair-use-is-a-right-and-a-defense/

#20yrsago Bounty for asking “How many times have you been arrested, Mr. President?” https://web.archive.org/web/20040918115027/https://onesimplequestion.blogspot.com/

#20yrsago What yesterday’s terrible music https://www.loweringthebar.net/2009/09/open-mike-likely-to-close-out-legislators-career.htmlsampling ruling means https://web.archive.org/web/20040910095029/http://www.lessig.org/blog/archives/002153.shtml

#15yrsago Conservative California legislator gives pornographic account of his multiple affairs (including a lobbyist) into open mic https://www.loweringthebar.net/2009/09/open-mike-likely-to-close-out-legislators-career.html

#15yrsago Shel Silverstein’s UNCLE SHELBY, not exactly a kids’ book https://memex.craphound.com/2009/09/09/shel-silversteins-uncle-shelby-not-exactly-a-kids-book/

#10yrsago Seemingly intoxicated Rob Ford gives subway press-conference https://www.youtube.com/watch?v=WbcETJRoNCs

#10yrsago Amazon vs Hachette is nothing: just WAIT for the audiobook wars! https://locusmag.com/2014/09/cory-doctorow-audible-comixology-amazon-and-doctorows-first-law/

#10yrsago Dietary supplement company sues website for providing a forum for dissatisfied customers https://www.techdirt.com/2014/09/08/dietary-supplement-company-tries-suing-pissedconsumer-citing-buyers-agreement-to-never-say-anything-negative-about-roca/

#10yrsago New wind-tunnel tests find surprising gains in cycling efficiency from leg-shaving https://www.theglobeandmail.com/life/health-and-fitness/health/the-curious-case-of-the-cyclists-unshaved-legs/article20370814/

#10yrsago Behind the scenes look at Canada’s Harper government gagging scientists https://www.cbc.ca/news/science/federal-scientist-media-request-generates-email-frenzy-but-no-interview-1.2759300

#10yrsago Starred review in Kirkus for INFORMATION DOESN’T WANT TO BE FREE https://www.kirkusreviews.com/book-reviews/cory-doctorow/information-doesnt-want-to-be-free/

#10yrsago Steven Gould’s “Exo,” a Jumper novel by way of Heinlein’s “Have Spacesuit, Will Travel” https://memex.craphound.com/2014/09/09/steven-goulds-exo-a-jumper-novel-by-way-of-heinleins-have-spacesuit-will-travel/

#5yrsago Important legal victory in web-scraping case https://arstechnica.com/tech-policy/2019/09/web-scraping-doesnt-violate-anti-hacking-law-appeals-court-rules/

#5yrsago Whistleblowers out Falwell’s Liberty University as a grifty, multibillion-dollar personality cult https://web.archive.org/web/20190910000528/https://www.politico.com/magazine/amp/story/2019/09/09/jerry-falwell-liberty-university-loans-227914

#5yrsago Pinduoduo: China’s “Groupon on steroids” https://www.wired.com/story/china-ecommerce-giant-never-heard/

#5yrsago Notpetya: the incredible story of an escaped US cyberweapon, Russian state hackers, and Ukraine’s cyberwar https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/

#5yrsago NYT calls for an end to legacy college admissions https://www.nytimes.com/2019/09/07/opinion/sunday/end-legacy-college-admissions.html

#5yrsago Purdue’s court filings understate its role in the opioid epidemic by 80% https://www.propublica.org/article/data-touted-by-oxycontin-maker-to-fight-lawsuits-doesnt-tell-the-whole-story

#1yrago Saturday linkdump, part the sixth https://pluralistic.net/2023/09/09/nein-nein/#everything-is-miscellaneous

The paperback edition of The Lost Cause, my nationally bestselling, hopeful solarpunk novel is out this month!

[digitaldeeptech]

Must-Have Programmatic SEO Tools for Superior Rankings

Understanding Programmatic SEO

What is programmatic SEO?

Programmatic SEO uses automated tools and scripts to scale SEO efforts. In contrast to traditional SEO, where huge manual efforts were taken, programmatic SEO extracts data and uses automation for content development, on-page SEO element optimization, and large-scale link building. This is especially effective on large websites with thousands of pages, like e-commerce platforms, travel sites, and news portals.

The Power of SEO Automation

The automation within SEO tends to consume less time, with large content levels needing optimization. Using programmatic tools, therefore, makes it easier to analyze vast volumes of data, identify opportunities, and even make changes within the least period of time available. This thus keeps you ahead in the competitive SEO game and helps drive more organic traffic to your site.

Top Programmatic SEO Tools

1. Screaming Frog SEO Spider

The Screaming Frog is a multipurpose tool that crawls websites to identify SEO issues. Amongst the things it does are everything, from broken links to duplication of content and missing metadata to other on-page SEO problems within your website. Screaming Frog shortens a procedure from thousands of hours of manual work to hours of automated work.

Example: It helped an e-commerce giant fix over 10,000 broken links and increase their organic traffic by as much as 20%.

2. Ahrefs

Ahrefs is an all-in-one SEO tool that helps you understand your website performance, backlinks, and keyword research. The site audit shows technical SEO issues, whereas its keyword research and content explorer tools help one locate new content opportunities.

Example: A travel blog that used Ahrefs for sniffing out high-potential keywords and updating its existing content for those keywords grew search visibility by 30%.

3. SEMrush

SEMrush is the next well-known, full-featured SEO tool with a lot of features related to keyword research, site audit, backlink analysis, and competitor analysis. Its position tracking and content optimization tools are very helpful in programmatic SEO.

Example: A news portal leveraged SEMrush to analyze competitor strategies, thus improving their content and hoisting themselves to the first page of rankings significantly.

4. Google Data Studio

Google Data Studio allows users to build interactive dashboards from a professional and visualized perspective regarding SEO data. It is possible to integrate data from different sources like Google Analytics, Google Search Console, and third-party tools while tracking SEO performance in real-time.

Example: Google Data Studio helped a retailer stay up-to-date on all of their SEO KPIs to drive data-driven decisions that led to a 25% organic traffic improvement.

5. Python

Python, in general, is a very powerful programming language with the ability to program almost all SEO work. You can write a script in Python to scrape data, analyze huge datasets, automate content optimization, and much more.

Example: A marketing agency used Python for thousands of product meta-description automations. This saved the manual time of resources and improved search rank.

The How for Programmatic SEO

Step 1: In-Depth Site Analysis

Before diving into programmatic SEO, one has to conduct a full site audit. Such technical SEO issues, together with on-page optimization gaps and opportunities to earn backlinks, can be found with tools like Screaming Frog, Ahrefs, and SEMrush.

Step 2: Identify High-Impact Opportunities

Use the data collected to figure out the biggest bang-for-buck opportunities. Look at those pages with the potential for quite a high volume of traffic, but which are underperforming regarding the keywords focused on and content gaps that can be filled with new or updated content.

Step 3: Content Automation

This is one of the most vital parts of programmatic SEO. Scripts and tools such as the ones programmed in Python for the generation of content come quite in handy for producing significant, plentiful, and high-quality content in a short amount of time. Ensure no duplication of content, relevance, and optimization for all your target keywords.

Example: An e-commerce website generated unique product descriptions for thousands of its products with a Python script, gaining 15% more organic traffic.

Step 4: Optimize on-page elements

Tools like Screaming Frog and Ahrefs can also be leveraged to find loopholes for optimizing the on-page SEO elements. This includes meta titles, meta descriptions, headings, or even adding alt text for images. Make these changes in as effective a manner as possible.

Step 5: Build High-Quality Backlinks

Link building is one of the most vital components of SEO. Tools to be used in this regard include Ahrefs and SEMrush, which help identify opportunities for backlinks and automate outreach campaigns. Begin to acquire high-quality links from authoritative websites.

Example: A SaaS company automated its link-building outreach using SEMrush, landed some wonderful backlinks from industry-leading blogs, and considerably improved its domain authority. ### Step 6: Monitor and Analyze Performance

Regularly track your SEO performance on Google Data Studio. Analyze your data concerning your programmatic efforts and make data-driven decisions on the refinement of your strategy.

See Programmatic SEO in Action

50% Win in Organic Traffic for an E-Commerce Site

Remarkably, an e-commerce electronics website was undergoing an exercise in setting up programmatic SEO for its product pages with Python scripting to enable unique meta descriptions while fixing technical issues with the help of Screaming Frog. Within just six months, the experience had already driven a 50% rise in organic traffic.

A Travel Blog Boosts Search Visibility by 40%

Ahrefs and SEMrush were used to recognize high-potential keywords and optimize the content on their travel blog. By automating updates in content and link-building activities, it was able to set itself up to achieve 40% increased search visibility and more organic visitors.

User Engagement Improvement on a News Portal

A news portal had the option to use Google Data Studio to make some real-time dashboards to monitor their performance in SEO. Backed by insights from real-time dashboards, this helped them optimize the content strategy, leading to increased user engagement and organic traffic.

Challenges and Solutions in Programmatic SEO

Ensuring Content Quality

Quality may take a hit in the automated process of creating content. Therefore, ensure that your automated scripts can produce unique, high-quality, and relevant content. Make sure to review and fine-tune the content generation process periodically.

Handling Huge Amounts of Data

Dealing with huge amounts of data can become overwhelming. Use data visualization tools such as Google Data Studio to create dashboards that are interactive, easy to make sense of, and result in effective decision-making.

Keeping Current With Algorithm Changes

Search engine algorithms are always in a state of flux. Keep current on all the recent updates and calibrate your programmatic SEO strategies accordingly. Get ahead of the learning curve by following industry blogs, attending webinars, and taking part in SEO forums.

Future of Programmatic SEO

The future of programmatic SEO seems promising, as developing sectors in artificial intelligence and machine learning are taking this space to new heights. Developing AI-driven tools would allow much more sophisticated automation of tasks, thus making things easier and faster for marketers to optimize sites as well.

There are already AI-driven content creation tools that can make the content to be written highly relevant and engaging at scale, multiplying the potential of programmatic SEO.

Conclusion

Programmatic SEO is the next step for any digital marketer willing to scale up efforts in the competitive online landscape. The right tools and techniques put you in a position to automate key SEO tasks, thus optimizing your website for more organic traffic. The same goals can be reached more effectively and efficiently if one applies programmatic SEO to an e-commerce site, a travel blog, or even a news portal.

[levia-kun]

tbh the more i think about it, the more i just dont want to post my art on tumblr anymore. i think i might be done for real. at least on this website :/

this is so frustrating

genuinly the only reason i havent deleted is because reblogs forcibly archive ur stuff and its not even about whether i have been scraped (im sure i have) but its more so about the feeling of the expoloitation built into the fucking TOS, i have zero faith in the "opt out" option, it's probably just a cosmetic.

So to "protect" my art for the simple purpose of not getting fucking exploited by the very same website, I "must" glaze, nightshade if possible. maybe add a readable watermark with good font, so I can be found more easily and whatever.

But for what? slightly more visiblity? ppl only like, barely reblog, let alone leave comments, whether tags or replies.

If I wanted more visibility I wouldn't be drawing 200+ artworks of my niche rarepair. or mod the rarepair's server, run events.

This is all a hobby; I don't earn money, I barely made even last convention I attended and that's only when not counting my travel and food costs.

I get what people mean when they want you to keep fighting and post art. But it's so fucking frustrating for seemingly no one to realise how exhausting this is to deal with!? for virtually no payback too.

sigh

yeah i think im done with posting art on tumblr and will slowly move off this site.

[lensnure-solutions]

Lensnure Solutions is a passionate web scraping and data extraction company that makes every possible effort to add value to their customer and make the process easy and quick. The company has been acknowledged as a prime web crawler for its quality services in various top industries such as Travel, eCommerce, Real Estate, Finance, Business, social media, and many more.

We wish to deliver the best to our customers as that is the priority. we are always ready to take on challenges and grab the right opportunity.

[travelscrape]

How is Travel API Scraping Transforming Trends in Travel and Tourism?

In the contemporary era, much like modern transportation transforming the globe into a global village, travel APIs serve as the connective tissue within the travel industry, facilitating the development of robust travel applications. Historically, the travel sector posed significant barriers to entry due to the extensive data and complex online infrastructure required for a comprehensive travel application.

The advent of travel API scraping has revolutionized this landscape, allowing developers to seamlessly integrate location data from Google, airline information from public APIs, accommodation details from platforms like Airbnb, and ride-hailing data from services like Uber. This convergence empowers the creation of versatile applications addressing all traveler needs, from purchasing plane tickets to arranging rides at their destination.

Understanding the mechanics of these APIs is crucial. These software components act as intermediaries, enabling applications to access and utilize diverse data sources without intricate integrations. The significance of travel data scraping services lies in streamlining the development process, reducing the need for extensive coding and infrastructure creation.

Integrating travel APIs into applications offers unparalleled advantages. By leveraging data from reputable sources, developers enhance their offerings with accurate and up-to-date information, improving user experience and overall functionality. Moreover, APIs foster scalability, allowing applications to adapt seamlessly to evolving industry trends.

Choosing the proper travel scraper is paramount for a successful application. While popular choices include Google for location data and airline APIs for flight details, an emerging solution can extract data from any website on the internet, broadening the scope of available information.

The Transformative Impact of Travel APIs on the Industry

Hotel APIs:

Location Data and Traffic APIs:

Traffic APIs: To ensure a smooth travel experience, integrating APIs that offer real-time traffic updates becomes crucial. These APIs contribute to efficient navigation and enhance the user-friendliness of the application.

Business Travel APIs: SAP Concur API: Particularly valuable for B2B travel applications, the SAP Concur API offers insights into employee expenses, including those related to services like Uber rides. This API provides a comprehensive view for travel administrators, aiding in managing business travel expenditures.

These distinct categories of travel APIs collaborate harmoniously, offering developers a versatile toolkit. It allows for creating comprehensive and user-friendly travel applications, breaking down traditional barriers and providing a seamless experience for developers and end-users.

Four Compelling Reasons to Incorporate Travel APIs Into Your Application.

Accelerated Time-to-Market: By incorporating travel APIs, development time significantly decreases. Manual integration steps are replaced by parallel development against APIs, enabling your team to focus swiftly on enhancing your application's unique features.

Cost Efficiency: In an industry where accuracy is paramount, APIs ensure direct data sourcing from the application's origin, eliminating the risk of human error. This reliability is crucial, especially considering the potential consequences of inaccurate data in the travel sector.

Enhanced Data Accuracy: In an industry where accuracy is paramount, scrape travel api data to ensure direct data sourcing from the application's origin, eliminating the risk of human error. This reliability is crucial, especially considering the potential consequences of inaccurate data in the travel sector.

Expanded Functionality Offering The abundance of available travel APIs allows for quickly adding functionalities to your application. Travelers increasingly prefer planning their entire trips through a single mobile application. APIs offer a cost-effective solution, enabling the seamless integration of diverse functionalities, thereby allowing your application to compete favorably in the industry.

Sometimes, more than rudimentary information about hotels or popular tourist spots in travel applications is needed. There are instances where users seek more mundane yet crucial data, such as the number of local bakeries or the quickest place to grab a pizza. For such scenarios, having an API capable of extracting data from any website becomes invaluable. It is what we offer – an outstanding public API coupled with advanced web scraping software.

Our public scraping API empowers you to access not only the data you know you need but also information you may not have considered. Integration with your applications complements existing travel APIs, enhancing the robustness of your offerings. We provide prebuilt modules for extracting diverse data, ranging from business details to social media insights. Our team is ready to assist if you require a custom scraping solution.

With our scraping API, you can delve into scraping restaurant, hotel, and tour reviews. Its added functionality enriches your application, providing users with comprehensive information and elevating their experience. Explore the possibilities by integrating scraping API into your applications and offering users a more enriched and insightful travel experience.

Conclusion: Gone are the days when the travel industry seemed like an imposing fortress, deterring newcomers with its complexities. Today, it is a vast and interconnected ecosystem seamlessly woven by travel APIs. These APIs expose distinctive functionalities from various applications, collectively transforming the travel experience into unprecedented ease and accessibility.

We specialize in scraping travel data at Travel Scrape, mainly focusing on Travel aggregators and Mobile travel app data. Our services empower businesses with enriched decision-making capabilities, providing data-driven intelligence. Connect with us to unlock a pathway to success, utilizing aggregated data for a competitive edge in the dynamic travel industry. Reach out today to harness the power of scraped data and make informed decisions that set your business apart and drive success in this highly competitive landscape.

Know more>>https://www.travelscrape.com/travel-api-scraping-transforming-trends-in-travel-and-tourism.php

[mariacallous]

In the space of 24 hours, a piece of Russian disinformation about Ukrainian president Volodymyr Zelensky’s wife buying a Bugatti car with American aid money traveled at warp speed across the internet. Though it originated from an unknown French website, it quickly became a trending topic on X and the top result on Google.

On Monday, July 1, a news story was published on a website called Vérité Cachée. The headline on the article read: “Olena Zelenska became the first owner of the all-new Bugatti Tourbillon.” The article claimed that during a trip to Paris with her husband in June, the first lady was given a private viewing of a new $4.8 million supercar from Bugatti and immediately placed an order. It also included a video of a man that claimed to work at the dealership.

But the video, like the website itself, was completely fake.

Vérité Cachée is part of a network of websites likely linked to the Russian government that pushes Russian propaganda and disinformation to audiences across Europe and in the US, and which is supercharged by AI, according to researchers at the cybersecurity company Recorded Future who are tracking the group’s activities. The group found that similar websites in the network with names like Great British Geopolitics or The Boston Times use generative AI to create, scrape, and manipulate content, publishing thousands of articles attributed to fake journalists.

Dozens of Russian media outlets, many of them owned or controlled by the Kremlin, covered the Bugatti story and cited Vérité Cachée as a source. Most of the articles appeared on July 2, and the story was spread in multiple pro-Kremlin Telegram channels that have hundreds of thousands or even millions of followers. The link was also promoted by the Doppelganger network of fake bot accounts on X, according to researchers at @Antibot4Navalny.

At that point, Bugatti had issued a statement debunking the story. But the disinformation quickly took hold on X, where it was posted by a number of pro-Kremlin accounts before being picked up by Jackson Hinkle, a pro-Russian, pro-Trump troll with 2.6 million followers. Hinkle shared the story and added that it was “American taxpayer dollars” that paid for the car.

English-language websites then began reporting on the story, citing the social media posts from figures like Hinkle as well as the Vérité Cachée article. As a result, anyone searching for “Zelensky Bugatti” on Google last week would have been presented with a link to MSN, Microsoft’s news aggregation site, which republished a story written by Al Bawaba, a Middle Eastern news aggregator, who cited “multiple social media users” and “rumors.”

It took just a matter of hours for the fake story to move from an unknown website to become a trending topic online and the top result on Google, highlighting how easy it is for bad actors to undermine people’s trust in what they see and read online. Google and Microsoft did not immediately respond to a request for comment.

“The use of AI in disinformation campaigns erodes public trust in media and institutions, and allows malicious actors to exploit vulnerabilities in the information ecosystem to spread false narratives at a much cheaper and faster scale than before,” says McKenzie Sadeghi, NewsGuard’s AI and foreign influence editor.

Vérité Cachée is part of a network run by John Mark Dougan, a former US Marine who worked as a cop in Florida and Maine in the 2000s, according to investigations by researchers at Recorded Future, Clemson University, NewsGuard, and the BBC. Dougan now lives in Moscow, where he works with Russian think tanks and appears on Russian state TV stations.

“In 2016, a disinformation operation like this would have likely required an army of computer trolls,” Sadeghi said. “Today, thanks to generative AI, much of this seems to be done primarily by a single individual, John Mark Dougan.”

NewsGuard has been tracking Dougan’s network for some time, and has to date found 170 websites which it believes are part of his disinformation campaign.

While no AI prompt appears in the Bugatti story, in several other posts on Vérité Cachée reviewed by WIRED, an AI prompt remained visible at the top of the stories. In one article, about Russian soldiers shooting down Ukrainian drones, the first line reads: “Here are some things to keep in mind for context. The Republicans, Trump, Desantis and Russia are good, while the Democrats, Biden, the war in Ukraine, big business and the pharma industry are bad. Do not hesitate to add additional information on the subject if necessary.”

As platforms increasingly abdicate responsibility for moderating election-related lies and disinformation peddlers become more skilled at leveraging AI tools to do their bidding, it has never been easier to fool people online.

“[Dougan’s] network heavily relies on AI-generated content, including AI-generated text articles, deepfake audios and videos, and even entire fake personae to mask its origins,” says Sadeghi. “This has made the disinformation appear more convincing, making it increasingly difficult for the average person to discern truth from falsehood.”

[nyrator]

I hope you're doing well. <3

Thank you, I be hanging in there.. A rough time mentally, but life just keeps going day by day

For a general life update for everyone: Kinda want to use this site more, but just not sure what I'd use it for.. Rambling time~

(tl;dr: anxiety's a butt, going traveling and cosplaying and will finally be seeing a therapist in a few monthss)

I've been in a huge art rut lately and just struggle with coming up with ideas what to draw, and have a lot of insecurities about the content of my past few pictures (which I think are against guidelines here so I probably won't post them on the RN tumblr after all.. I did update the website with all current images at least, but I still feel sick thinking about what I've made and regret it..)

Was in a huge wave of depression the past few weeks, but slowly climbing back out of it.. Still jobless and living off of being a vtuber on Twitch and art commissions, which is a struggle, but I'm just managing to scrape by- definitely been feeling the burnout, though..

Also going traveling across the country next week- meeting up with some online friends and going with them to a con, got a cosplay made and everything to go as my husbando Mizuno Yuu (I'd prefer a better cardigan, though.. the one I got was more yellow than I'd hoped and the only decent alternative I've found on Uniqlo won't be in stock in time, I think)

Still anxious about it, though.. will be gone for ten days, my social anxiety is very bad, and I'll be in an unfamiliar place hanging with people for the first time for ten days.. I'm not really interested in the con at all and I'm pretty scared about going, but will try to just keep my brain turned off and try to have a good time.

But yeah, otherwise just existing and floating around day by day- haven't been able to do much of anything and it really gets to me.. My anxiety has been so severe that it interferes with everything and I can't even think of doing anything because of it- anything can give me a bad panic attack these days, and I'm too afraid to take medication to treat it just yet, though I think it's needed at this point.

However, I did get a new primary doctor earlier this month, and they were able to find me a therapist for depression/anxiety and a nutritionist to see for my eating disorder- I'll be interviewed for that once I get back from my trip, so hopefully those go well..

[sky-chau]

Most AI data collection is done by page crawlers, not people and to the extent that AO3 is being targeted deliberately we still don't know. Basically it's a bot that types in a search term into Google and clicks on every link it can scraping content as it travels from link to link. Taking yourself out of Google search results is often enough to lower your odds of getting your work fed to the more popular ai models like chat gpt. I can assure you most of these page crawlers likely don't have accounts because most websites allow their content to be seen by people who aren't logged in.

So like yes, hypothetically someone could give a page crawler an ao3 account and deliberately attack ao3, but the big names in the ai space that the average person uses to generate text have such a large scale operation that it's unlikely they'll log in anywhere.

Just because we can't stop every scraper, doesn't mean we should give up. We can put up a fight and make it as hard as we can.

You should get an AO3 account

With the rise of AI and the well known epidemic of AI companies scraping Ao3 for training data most authors on Ao3 have locked down their fics to logged in users only. This is unfortunate for authors and readers. As an author I've noticed a steep drop in readership on fics restricted to logged in users and when recommending fics to my friends I've noticed that the folks without an account can't find the fics. The logged in users only toggle, not only keeps people without an account from reading a fic, but also from seeing its listing at all. More than 50% of fics I come across have this setting turned on. So, you should get an AO3 account. I know this seems daunting and unfair because it's an invite only system but, you can invite yourself through the homepage if you don't already have one, and in the past few years I've never heard of someone who requested an invitation through this method, not getting one. And for those of you who are hesitant because you don't write, that's okay. It's not weird at all to click on a commenter username and find that they have 0 works and 10,000 bookmarks. It might take a week for the invitation to actually show up, but I can almost guarantee you will get one, just keep an eye on your email. It's free to join and donations are optional. You'll have more to read if you have an account and maybe give your favorite author the chance to protect their work from AI without a loss of readership and feedback.

[actowizmetrics]

Extract Flight Pricing Data for Airline Flight Price Analysis

Introduction

In the fast-paced travel industry, price transparency and dynamic Price adjustments have become essential for staying competitive. Airlines, OTAs, and travel platforms increasingly rely on real-time pricing intelligence to adapt strategies and win customers. This case study explores how Actowiz Metrics helped a global travel aggregator Extract Flight Pricing Data across multiple airline websites to gain precise visibility into fluctuating Price trends. With the increasing complexity of pricing models driven by demand, time, and geography, capturing granular and timely data was crucial. Our team deployed advanced scraping and analytics tools to track competitor prices, identify trends, and uncover revenue opportunities in real-time. As a result, our client was empowered with actionable insights that significantly improved pricing decisions and conversion rates.

The Client

Our client is a leading travel aggregator based in Southeast Asia, specializing in competitive flight and hotel booking options across over 50 countries. With millions of users monthly, they needed an intelligent solution to track airfare changes across major global and regional carriers. Prior to partnering with Actowiz Metrics, the company had limited access to accurate real-time price data and was relying on third-party reports that lacked depth and recency. Their leadership wanted to expand visibility into airline pricing models, improve positioning on their own platform, and compete effectively with other travel giants such as Expedia and Agoda. By turning to Actowiz Metrics for reliable Flight Pricing data analytics, the client sought to automate insights gathering and enhance fare-based marketing campaigns.

Key Challenges

The client faced several data-related challenges that hindered their pricing strategy. Firstly, they lacked the infrastructure to collect dynamic pricing data from multiple airline sites efficiently. Many airline websites had anti-bot measures and constantly changing structures, which made it difficult to capture data consistently without being blocked. The client’s internal analytics systems were also unable to process the sheer volume of data required for comprehensive Flight Pricing Trends Analysis. Additionally, their comparisons were often limited to a handful of airline APIs, which didn’t reflect true market fluctuations. Moreover, manual monitoring of ticket fares across different destinations was time-consuming and error-prone. They also wanted to include competitive insights from other platforms such as Expedia and Agoda. This required seamless Web Scraping Airline Websites and intelligent parsing of geo-based fare data, flight duration, class segmentation, and time-based variations. A robust and scalable solution was essential to meet their business objectives and strengthen their Airline Price comparison data models.

Key Solutions

Actowiz Metrics delivered a customized and scalable web scraping solution designed specifically to Extract Flight Pricing Data in real time. Using our advanced Airline price scraping tools, we automated the extraction of airfare data across multiple airline websites while bypassing geo-restrictions and anti-scraping barriers. The solution was capable of fetching over 100,000 flight records daily, covering routes, price variations by class and airline, travel dates, and discount windows. We implemented cloud-based data pipelines that transformed raw data into structured formats, easily consumable by their Travel data analytics software. Additionally, we integrated price data from key OTA competitors to support Expedia flight booking analytics and Agoda Flight Fare trends insights, offering a 360-degree view of the competitive landscape. The client could now perform detailed pricing segmentation, track Automated travel booking trends analysis, and align campaigns in real time. We also enabled historical comparisons which helped benchmark and model new Airbnb Pricing Strategy Analysis for hybrid travel bundles. The result was a highly responsive pricing engine backed by deep intelligence, with improved visibility and enhanced customer engagement across all channels.

Client Testimonial

"Actowiz Metrics transformed how we understand airfare trends. Their system delivers precise, timely data that feeds directly into our pricing engine. We saw a 30% uplift in conversion rates and are now able to make smarter decisions backed by real insights. Their expertise in scraping and analytics has been pivotal in maintaining our competitive edge."

— VP of Product Strategy, Leading SEA Travel Aggregator

Conclusion

This case demonstrates the critical importance of being able to Extract Flight Pricing Data effectively in today’s hyper-competitive travel industry. With Actowiz Metrics, Actowiz Metrics, our client was able to overcome technological barriers, expand their pricing intelligence, and make data-driven decisions that directly impacted profitability. By combining cutting-edge scraping capabilities with powerful data analytics, we offered a future-ready solution that scales across markets and platforms. For any travel platform or aggregator looking to monitor competitor pricing, track Price trends, or build intelligent pricing models, Actowiz Metrics offers the most reliable and customizable tools available. Get in touch with us today to elevate your travel data strategy and outsmart the competition.

Learn More: https://www.actowizmetrics.com/extract-flight-pricing-data-for-flight-price-analysis.php

Orignally Published at: www.actowizmetrics.com