#Scrape Realtime Website Data | Explore Tumblr Posts and Blogs

mostlysignssomeportents · 4 years

Text

Pluralistic: 06 Mar 2020 (Stunning RPG dice, Shell funded climate denial, Church sends US predator priests to Mexico, South Korea is beating covid-19)

Today's links

The most beautiful RPG dice I've ever seen: And you can also make your own.

The king of Dutch climate denial was secretly in Shell's pay: Frits Böttcher was a packrat, and his papers detail exactly how he was paid to sow climate doubt. He was very good at it.

American Catholic officials helped priests who preyed on children escape to Mexico: At least 51 "credibly accused" priests left the US and took up positions abroad.

A grifty AI company conned the state of Utah into giving access to everything: Banjo claims it will predict and head off terrorist attacks, mass shootings, and child abductions without invading anyone's privacy.

Clearview AI says it only lets cops use its facial recognition tool but it's lying: Investors, cronies and pals got to literally use it as a party trick.

South Korea's beating covid-19 with free testing: Testing is part of the free national health system, and 140,000 tests have been administered.

The web is unusably beshitted with terrible ad-tech: "No, I don't want great articles."

For $3, a robolawyer will automatically force data brokers to delete you and sue the ones who don't: Donotpay meets the CCPA, it's like peanut butter and chocolate.

This day in history: 2005, 2015, 2019

Colophon: Recent publications, current writing projects, upcoming appearances, current reading

The most beautiful RPG dice I've ever seen (permalink)

Sasha is a spectacularly talented RPG dice-maker, whose online store features the most beautiful dice I've ever seen – and as if that wasn't enough, she also sells dice-making kits to use at home.

https://www.sunshadeauarts.com/sunshadeauarts-academy/

Last month, ahead of the C2E2 con, she posted a series of new, not-for-sale (argh) dice that embed a variety of materials inside large D20s to form nebulas, clouds, alien landscapes, menacing eyeballs, and eldritch scenes. Check them out for yourself!

https://twitter.com/sunshadeauarts/status/1232722877008490497 https://twitter.com/sunshadeauarts/status/1229445585717035010 https://twitter.com/sunshadeauarts/status/1232795390916911104 https://twitter.com/sunshadeauarts/status/1233370655216881664 https://twitter.com/sunshadeauarts/status/1233380666810806274

It's hard to say what these will cost; comparable dice on her site sell for $400. They're handmade, beautiful sculptures, after all.

https://www.sunshadeauarts.com/product/less-than-perfect-midnight-aurora-handmade-resin-inkless-titan-d20/

At that price, they're maybe too expensive for a gift for yourself, but as a graduation present, maybe? And that said, it's exactly the kind of thing I sometimes buy to celebrate selling a new novel, and that's on my roadmap for THE LOST CAUSE, my post-GND, truth-and-reconciliation novel, so I'm definitely putting a reminder in my calendar.

The king of Dutch climate denial was secretly in Shell's pay (permalink)

Club of Rome founder Frits Böttcher was the Netherlands' leading climate denier. He died in 2008. Investigative journalists combing through his papers, discovered that he was paid €500K by Shell and others to sow doubt about climate change.

https://www.ftm.nl/dutch-multinationals-funded-climate-sceptic

His network pushed out scientific frauds like the idea that excess atmospheric CO2 would be "good for plants" through books, lectures and reports.

He was good at it. His work was crucial to stalling action on climate change in the 1990s. Despite this, his 24 sponsors dumped him in 1998 after the signing of the Kyoto Protocol, worried that outright climate denial had lost credibility.

No wonder! This was the guy who'd called climate science "a witch-hunt on CO2" and declared "Our planet is not a greenhouse."

In his papers, Böttcher notes that after he published these frauds, Shell contacted him and offered him giant sums to keep it up and amplify it. The work was personally commissioned by Shell managing director Huub Van Engelshoven. Böttcher was a packrat. His papers in the Noord-Hollands Archief in Haarlem stack 15.9m tall. Inside of them is an eye-wateringly detailed account of how wealthy, planet-wrecking firms deliberately and maliciously paid for climate denial.

That means that we now can name names. We think of climate denial as a kind of emergent property with no human agent, but as the world drowns, roasts, and writhes with pandemic, we have the names and addresses of the people who engineered that situation for their own gain. We know who his political allies were: the VVD party. When the Netherlands' dikes fail and the country begins to drown, these politicians might still be running for office.

It's tempting to think of the climate crisis as something we all bear responsibility for, because we didn't sort our recycling or because we didn't use the underfunded, anemic public transit options available to us. But efforts like this – from Platform Authentieke Journalistiek and Follow the Money – show we were corralled into our complicity by a network of super-rich plutes for their own gain, who knew they were wrecking the world and dooming our children but did not care.

American Catholic officials helped priests who preyed on children escape to Mexico (permalink)

A new instalment in Propublica's outstanding coverage of the Catholic Church's complicity in sexual abuse by priests shows that dozens of American priests who raped children were relocated to Mexico, where they continued to rape children

https://www.propublica.org/article/dozens-of-catholic-priests-credibly-accused-of-abuse-found-work-abroad-some-with-the-churchs-blessing#178005

These priests found new postings thanks to glowing letters of recommendation from church officials who knew that they had been accused – or, in some cases convicted – of raping children in their parishes. Some fled to Mexico to avoid prison, resisting extradition for years.

Not just Mexico: Propublica found 51 "credibly accused" US priests who are currently working in Mexico, Ireland, Nigeria, and the Philippines. Some of them continued to draw pay from their US parishes while they settled in abroad. Parishoners' donations paid for the predators who victimized their children to escape justice.

One priest, Jose Antonio Pinal, wrote letters to Church officials blaming the boy he raped, saying, "that he is not innocent of the situation he wants to blame me for completely." Pinal is still ministering in Cuernavaca. He claims his longrunning rapes of a 15-year-old were consensual, but "he was a minor; so, legally, I am screwed."

When he moved to Mexico, Sacramento church officials wrote to him promising to support him, so long as the new diocese promised to "protect the diocese of Sacramento against any financial liability for any acts committed by you while working in that diocese."

Some of these priests are listed as "inactive" in Church websites, but are still ministering in Mexico. Rev. Jeffrey David Newell, admitted to sexually abusing another 15 year old, and called it a "mistake." He currently serves in Tijuana. Newell says it was a single slip up. Other survivors of his abuse have filed lawsuits against his old US archdiocese. Newell calls their claims "totally absurd." His name has been removed from Church lists of "credibly accused" sexual predators in the clergy.

These predators' survivors are alive and deeply traumatized. And thanks to the inaction, complicity and even encouragement of US Catholic church officials, these priests are ruining the lives of new children all over the world.

A grifty AI company conned the state of Utah into giving access to everything (permalink)

The State of Utah has secretly contracted with "Banjo," a grifty "AI" company, to analyze all the surveillance and internal data generated by all the state's agencies.

https://www.techdirt.com/articles/20200305/13422544042/ai-company-has-access-to-pretty-much-every-piece-surveillance-tech-state-utah-owns.shtml

Banjo gets all the 911 calls, CCTV camera feeds, license plate readers, and internal state databases, and its proprietary, secret algorithm will comb through all that to direct law enforcement.

https://www.vice.com/en_us/article/k7exem/banjo-ai-company-utah-surveillance-panopticon

The company claims there are no privacy concerns because it has a patented system for anonymizing data. The patents do not disclose their anonymization method, and every other attempt at this kind of anonymization has fallen prey to "re-identification" attacks.

Banjo gets to locate a facility inside the Utah DOT HQ, and will operate in all 29 counties, state university campuses and 23 cities (including Utah's 10 largest cities). The company's making $20.7m on this contract over five years.

Using FOIA requests, Motherboard retrieved records showing how Banjo got Utah officials to help it sell its services ot the state. When Motherboard questioned the officials about this, they flat-out lied and denied it. The Banjo pitch claims that they'll head off terrorist attacks, mass shootings and child abductions in realtime. The company provides zero evidence that they have ever done such a thing, or that they ever could.

But that lack of evidence didn't deter Utah AG Ric Cantrell:

"They do have case studies. I'm waiting for case studies from Banjo. I'm still waiting for information from them."

Uh, maybe you should have seen the studies before putting Banjo's servers behind your firewall?

Clearview AI says it only lets cops use its facial recognition tool but it's lying (permalink)

Clearview AI is another grifty "AI" company cutting secret deals with law enforcement to use its facial recognition tech, which relies on a database of nonconsensually scraped social media photos.

They claim only cops get to use this. It's a lie.

https://www.nytimes.com/2020/03/05/technology/clearview-investors.html

Clearview investors, clients and cronies all have logins to the system. Long before it was selling to cops, these people were literally using it as a party trick, getting people at parties to give them photos to subject to Clearview analysis, just for shits and giggles.

For example, billionaire John Catsimatidis used it to freak out his daughter, sneaking a pic of her data while she was at a restaurant and then IDing the guy and texting her with the guy's bio while she was still eating with him.

An investor named David Scalzo gave the app to his children: "They like to use it on themselves and their friends to see who they look like in the world. It's kind of fun for people."

It sure seems like Ashton Kutcher also got to run around and use it without limit or accountability. Last time I checked, he was also not a police officer.

https://www.youtube.com/watch?v=nNhYqLbsAGk&feature=youtu.be

One tech expert, Nicholas Cassimatis, uses the app as "a hobby."

Your 21st Century panopticon, folks, brought to you by compulsive liars who ask us to trust them not to get it wrong.

South Korea's beating covid-19 with free testing (permalink)

South Korea has tested 140,000 people for Covid-19. The tests are free for all as part of the nation's public health program. Testing has led to world-leading containment of the disease.

https://www.bloomberg.com/news/articles/2020-03-04/south-korea-tests-hundreds-of-thousands-to-fight-virus-outbreak

President Moon Jae-in calls it a "war" and has put the country on the kind of footing that you'd expect of any existential threat, sidelining the interests of industry in favor of national survival. They're testing 10,000 people/day. Results are available in hours. You can get tested at drive-through testing centers. The kits are 90% accurate and were developed by a domestic producer, Seegene Inc.

America is learning that offshoring high-tech manufacturing to save on labor costs and allowing its private sector to dominate its healthcare resulted in a brittle situation where it can't produce reliable tests, and the unreliable tests are only available to the wealthy.

The fate of uninsured, untested, untreated Americans is not theirs alone. They're the ones preparing wealthy Americans' food and cleaning their homes.

We have a shared microbial destiny that no amount of neoliberal doctrine can handwave away.

https://pluralistic.net/2020/03/01/shared-microbial-destiny/#covidclasswar

The web is unusably beshitted with terrible ad-tech (permalink)

The web is unusably beshitted and encrufted with popups, interstitials, rolldowns, nagware, paywalls, autoplaying video, ads that scroll with the page, and worse. I haven't looked at the web without an adblocker in years and it's still barely usable.

https://www.cjr.org/first_person/the-infinite-scroll.php

The modern web's equilibrium is "as terrible as possible, without being so terrible that you stop reading," or, worse, "as terrible as is necessary to get you pay to bypass the paywall."

In the CJR, David Roth publishes one of the most pitiless, accurate, evocative descriptions of using the modern internet of cruft.

"The page loads, and a little video ad box rises from the bottom left of the screen and begins buffering. Then a big box pops up over the small one with an offer to subscribe to the paper at a special promotional rate… As you contemplate it, the video begins to play in a muted spasm. This throws a scrim of gray over the rest of the page, making it impossible to read…While you've been triaging a second small video player has floated up into the middle left of the screen. You manage to close these various boxes, and now you can scroll. For a few seconds, anyway, until another ad creeps down from the banner ad above the headline."

But Roth isn't merely complaining here. He's also digging into the underlying reality: dwindling margins, short-term thinking, monopolization of the ad-market, and a buyer's market for ads that lets advertisers demand worse and worse of publishers. Publishers are staffed with people who are "perpetually maxed-out and stressed and scrabbling for a dwindling and finite amount of money." They're choosing chumboxes and other garbage because they want to keep the lights on.

This happened before, of course. It's an HTML5, CSS-enabled reprise of the pop-up wars, where exploding inventory and finite advertising allowed advertisers to play publishers off against each other with increasingly obnoxious, intrusive pop-ups.

These were unbelievably terrible, even by modern standards. Pop-ups would spawn at 1px X 1px, making them invisible, autoplaying audio. Others would sense your mouse heading for the close box and move themselves away from your pointer. They'd spawn 3 more pop-ups for every one you closed, or 300, until your computer ran out of RAM and crashed, taking all your work with it.

These pop-ups didn't go away because publishers won the battle. They went away because of pop-up blocking.

When Opera, and then Mozilla, turned pop-up blocking on by default, users finally had a meaningful reason to prefer one browser to the others. One browser was usable. The other one let pop-up ads crash your computer and eat your unsaved docs. As users switched en masse to blocking browsers, publishers could tell advertisers, "Look, we'll run any garbage ad you tell us to because we need your money. But if it's a pop-up it will be blocked by the majority of our users. They just won't see it."

The pop-up wars were won because technologists helped users exercise technological self-determination. But increasingly, browser vendors are ad-tech companies. Even when they're not, browsers are being designed to serve publishers (who are under advertisers' thumbs), not users.

https://www.eff.org/deeplinks/2017/09/open-letter-w3c-director-ceo-team-and-membership

We should address monopolies in ad-tech and browsers, we should create meaningful privacy protections via a federal privacy law with a private right of action. But all of that needs to be accompanied with legal cover for users who assert the right to unshittify their web sessions.

https://www.eff.org/deeplinks/2019/07/adblocking-how-about-nah

This won't just protect users, it will protect publishers. It's one thing to prohibit publishers from intrusive advertising. But it's another altogether to make that kind of advertising literally technically impossible.

For $3, a robolawyer will automatically force data brokers to delete you and sue the ones who don't (permalink)

The always-amazing Donotpay has a new robot-lawyer service: as part of your $3/month, they'll serve every data-broker with a demand to purge your records under the CCPA, and sue the ones who don't.

https://fortune.com/2020/03/05/delete-location-data-privacy-personal-information-donotpay/

Data-brokers don't just drive nuisance calls, they also expose you to risks like being doxed and swatted, or having your identity stolen, including by stalkers and bounty hunters who exploit mobile phone tracking to get your realtime location. Every single person should purge their data from every single data-broker, period. Donotpay targets the top 20 brokers and facial recognition companies, including Clearview AI.

Donotpay automates opt-outs for these companies. It also automates suing companies that don't comply or those that make illegal demands like requiring you to send a scan of your driver's license before they'll purge your records. Once you're signed up, you can opt out your whole family, and even your friends. If you don't want a $3/month sub (which gets you tons of other awesome robolawyering), you can just sign up once, pay $3, purge your records and cancel.

Fulfilling deletion requests costs companies about $10. You can use them punitively. Any time a company pisses you off, you can just file a data-deletion demand under CCPA.

When Donot pay started, it was Ios only and I couldn't use it. Somewhere along the way, they got a web interface, too. I just signed up. I'm gonna pay for the wifi on my flight this afternoon just so I can explore all its options.

This day in history (permalink)

#15yrsago Bram Cohen's Stanford talk on BitTorrent https://web.archive.org/web/20051124040524/http://stanford-online.stanford.edu/courses/ee380/050216-ee380-100.asx

#5yrsago DMCA abuser ordered to pay $25K to WordPress https://torrentfreak.com/wordpress-wins-25000-from-dmca-takedown-abuser-150305/

#5yrsago Albuquerque PD encrypts videos before releasing them in records request https://www.techdirt.com/articles/20150221/17074630102/albuquerque-police-dept-complies-with-records-request-releasing-password-protected-videos-not-password.shtml

#5yrsago Judge who invented Ferguson's debtor's prisons owes $170K in tax https://www.theguardian.com/us-news/2015/mar/06/ferguson-judge-owes-unpaid-taxes-ronald-brockmeyer

#5yrsago Hartford, CT says friends can't room together unless some of them are servants https://www.courant.com/news/connecticut/hc-hartford-scarborough-street-house-0218-20150217-story.html

#5yrsago Finnish millionaire gets EUR54K speeding ticket https://www.bbc.com/news/blogs-news-from-elsewhere-31709454

#1yrago Zuckerberg announces a comprehensive plan for a new, privacy-focused Facebook, but fails to mention data sharing and ad targeting https://www.wired.com/story/facebook-zuckerberg-privacy-pivot/

#1yrago Ruminations on decades spent writing stories that run more than 1,000,000 words https://www.antipope.org/charlie/blog-static/2019/03/lessons-learned-writing-really.html

#1yrago A thorough defense of Modern Monetary Theory https://www.forbes.com/sites/johntharvey/2019/03/05/mmt-sense-or-nonsense/#62e9ed235852

#1yrago GOP lawmaker driven mad by bill that would decriminalize children who take naked photos of themselves, delivers a frenzied rant about anal sex on legislature's floor https://www.thestranger.com/slog/2019/03/05/39511377/a-bill-decriminalizing-teen-sexting-passes-the-house-causing-republican-to-scream-about-anal-sex-on-the-floor

#1yrago Bounty hunters and stalkers are able to track you in realtime by lying to your phone company and pretending to be cops https://www.vice.com/en_us/article/panvkz/stalkers-debt-collectors-bounty-hunters-impersonate-cops-phone-location-data

#1yrago From prisons to factories to offices: the spread of workplace surveillance and monitoring tech https://datasociety.net/wp-content/uploads/2019/09/DandS_WorkplaceMonitoringandSurveillance-.pdf

#1yrago NH GOP lawmakers mocked gun violence survivors by wearing clutchable pearl necklaces to gun control hearing https://www.washingtonpost.com/politics/2019/03/06/gop-lawmakers-wore-pearls-while-gun-violence-victims-testified-activists-were-outraged/?utm_term=.addd1b7a24f8

Colophon (permalink)

Today's top sources: Emptywheel (https://www.emptywheel.net/), Slashdot (https://slashdot.org), Naked Capitalism (https://www.nakedcapitalism.com/), Super Punch (https://superpunch.net/, Bas van Beek (http://www.basvanbeek.com/).

Hugo nominators! My story "Unauthorized Bread" is eligible in the Novella category and you can read it free on Ars Technica: https://arstechnica.com/gaming/2020/01/unauthorized-bread-a-near-future-tale-of-refugees-and-sinister-iot-appliances/

Upcoming appearances:

Museums and the Web: March 31-April 4 2020, Los Angeles. https://mw20.museweb.net/

LA Times Festival of Books: 18 April 2020, Los Angeles. https://events.latimes.com/festivalofbooks/

Currently writing: I'm rewriting a short story, "The Canadian Miracle," for MIT Tech Review. It's a story set in the world of my next novel, "The Lost Cause," a post-GND novel about truth and reconciliation. I'm also working on "Baby Twitter," a piece of design fiction also set in The Lost Cause's prehistory, for a British think-tank. I'm getting geared up to start work on the novel afterwards.

Currently reading: Just started Lauren Beukes's forthcoming Afterland: it's Y the Last Man plus plus, and two chapters in, it's amazeballs. Last month, I finished Andrea Bernstein's "American Oligarchs"; it's a magnificent history of the Kushner and Trump families, showing how they cheated, stole and lied their way into power. I'm getting really into Anna Weiner's memoir about tech, "Uncanny Valley." I just loaded Matt Stoller's "Goliath" onto my underwater MP3 player and I'm listening to it as I swim laps.

Latest podcast: Disasters Don't Have to End in Dystopias: https://craphound.com/podcast/2020/03/01/disasters-dont-have-to-end-in-dystopias/

Upcoming books: "Poesy the Monster Slayer" (Jul 2020), a picture book about monsters, bedtime, gender, and kicking ass. Pre-order here: https://us.macmillan.com/books/9781626723627?utm_source=socialmedia&utm_medium=socialpost&utm_term=na-poesycorypreorder&utm_content=na-preorder-buynow&utm_campaign=9781626723627

(we're having a launch for it in Burbank on July 11 at Dark Delicacies and you can get me AND Poesy to sign it and Dark Del will ship it to the monster kids in your life in time for the release date).

"Attack Surface": The third Little Brother book, Oct 20, 2020.

"Little Brother/Homeland": A reissue omnibus edition with a very special, s00per s33kr1t intro.

22 notes · View notes

codexworldblog · 4 years

Text

Extract Website Content with Web Scraping API using PHP

The Web Scraping API allows the developer to scrape data from the website in a structured format. It returns realtime data from the websites based on the web page URL specified in the API settings. The Web Scraping API is very useful when you want to extract content from the HTML source of the web pages.

There are various Web Scraping API available to scrape the webpage data, Scrapestack is one…

View On WordPress

#API #PHP

0 notes

graykuglen · 7 years

Text

Data Driven Design

Both data analysis and design research share a common goal and that is to answer questions about why. It might be better to say ‘data analysis’ seeks to come up with a hypothesis based on insights found in data, and ‘design research’ looks to gather evidential insights by direct observation, but I’m still chasing the same question of why?. I’m learning more about data analysis to inform design and it’s really exciting. Now what if we lead with data analysis to inform design?

Using some simple investigation and some statistical analysis tools (”R” + “R Studio”, Excel) I explored the topic of home buying. The information I found helped inform some concepts, which I created in Sketch and then prototyped simple micro interactions using Principle, a prototyping tool. I’ll keep it brief and go over the thought process using a data analysis framework (below).

DEFINE

There’s a growing disparity between people who can afford more than they need in a home, and those that cannot afford much of anything at all. An unfortunate outcome are skyrocketing prices in metro areas with a limited supply of homes. Stated more plainly, people with more money drive up the prices for others. Nobody likes to pay too much though, could more helpful data about the market/sales help home buyers (across the income spread) not overpay?

I’ve lived in two of these metro areas - San Francisco and Portland. An interesting side note is that San Francisco’s high prices are actually driving up Portland’s from displaced Californian’s moving to Portland.

DISSASSEMBLE

How do home buyers make purchase decisions?

Is the buyer at a disadvantage (asymmetry of information) without the right information to make better informed decisions on price/value?

What information (about the value of a home) may make buyers more informed about pricing, if any? Could this new information slow pricing increases?

QUESTION 1 - How do home buyers make decisions to purchase

In a market that has so many buyers seeking the same type of home there’s gonna be competition. It’s the market economics of supply and demand, if demand is high and supply low, prices go up. More information for the buyers may not make much of a difference here. However, I spoke with a few very successful real estate agents about how their clients make purchase decisions, here’s their response (paraphrased):

“Buyers are very rational when they begin shopping for a home (mostly for first time buyers) making financial decisions around affordability and needs paramount. But in the end the deciding factor is emotion and desire.”

Let’s assume this expert view is valid. It is very probable that first time home buyers enter the buying process logically and rationally and see what they can afford. We may focus first on influencing the people who are driving up prices. AND, we'd want our focus to be in the early stages before emotion steps in.

QUESTION 2 - Is the buyer at a disadvantage without enough information to make more informed decisions?

Price/value information available for new buyers, while “still rational and pre-emotive”, may be limited. Consider the following:

MLS data shows all the facts about a property, But there is little pricing or market data in it.

Online websites offer the same data in a better presentation, but may be stale and not timely. Regional MLS's see to that.

Websites do offer price $sq/ft which is great. Some even offer estimates of value, but with no insight as to how they arrived at the price.

Buyers are beholden to their agents for “comps” or comparables. These are 1-1 comparisons of similar properties which sold recently.

Some online websites do offer comparables, but the properties can be questionable which may create doubt and require more analysis.

Getting immediate pricing of what is selling/sold is guarded. Other free options take effort and added analysis.

You may pay for sales data using services like Dataquick but this is not ideal

Both selling and buying agents get a commission on the property sale, the higher the price the more they make - just saying.

The market data THAT is available is aggregate and shows averages of what properties go for in the Zip code or neighborhood, seldom at the property level that is accurate.

So, maybe there could be an opportunity to show more value-based information.

QUESTION 3 - What information (about the value of a home) may make buyers more informed about pricing, if any?

I evaluated the information available online to buyers. There definitely is some good information to help but it’s scatterd. Key areas I find most interesting and worth exploration include:

Realtime sales data of comparable properties (may or may not be attainable)

Exploring and expanding on a metric that already exists e.g. price / sq ft

There are different qualities of homes, why not grade the properties with desirability or a property grade A, B, C... (interesting, but is this viable?)

Better yet, why not make this about ‘my needs’ for a property which could be looked at differently

EVALUATE

I scraped data online for 3BR, 2+BA homes in San Francisco (Redfin.com), active or sold within the last 90 days - there were 347 records. Using a data analysis tool called “R” and “RStudio” I began exploring the data.

Scraped active listings for 3BR, 2+BA

Scraped comps (past 90 days) for 3BR, 2+BA

Queried the two together to get a better understanding of the price per square foot for 3BR’s using summary data, median, mean, histograms etc..

Insight

Sales prices in San Francisco are higher than the list price by an estimated 12-13%. The insight here “what if this information could be more readily available to buyers”? Could it influence offers?

Look at the histogram (above) for Active listing (For Sale) 3BR/2+BA single family homes, and you’ll see the largest number of homes in the median, or middle.

The pricing for sold homes (above) is considerably higher by 12-13%. That could be for several reasons and something to investigate.

More questions arise

San Francisco is one of the hottest markets in the US. How do “cool-markets” compare to “hot-markets”?

Can I get a list-prices to compare with my sales prices for 3BR/2+BA? This may lead to a correlation analysis and possibly my first regression of 3BR/2+BA’s (sadly I cannot get this data)

Because it’s a hot-market, it may be wise to use control groups (in two different markets for hot and cool) to test effectiveness of any price/value design concepts. In other words, test with concept in hot market, and w/out concept in hot market, and do the same for a cool market.

Design insights to explore

Show buyers a histogram? If we show a live histogram (shown above), could it help buyers understand pricing of properties better?

Use price per Sq Ft to show a distribution of where ‘this’ home lays on a curve of 3BR/2+BA homes.

Define an acceptable pricing range based on actual sales data. If a property is over price range for 3BR/2+BA, indicate it somehow.

Don’t waste buyers time - show only properties that are in the rational realm of their minds and avoid properties outside their decision criteria. This may also avoid the “I love it and need to have it” scenario.

Is it possible to tailor listings to the buyer’s decision criteria? For example, what is an acceptable match to my simple stated needs --> 60% of what I want, 70% of what I want? Might there be a way to show this? Maybe we could re-look at saved searches....

DECIDE

My original question was “is there an imbalance in the available information to help home buyer’s make more informed decisions that don’t drive up prices for everyone?”

Two concepts explored

The first concept is to help buyers by better matching them with what they set out to buy originally. It would reflect the decision criteria they started with, e.g. 3BR/2+BA, 2000sq ft+ etc.. This could be a simple metric to show at what % does it meet my stated needs. Buying a home is very time consuming and stressful, this is intended get buyers to physically visit only the best matched properties.

The second concept explores the value of a home compared to actual sales data. It would have two parts - a histogram highlighting where this property falls on the distribution curve, and the second part would show variance away from the median by some measure (yet to be defined), If its over 10% (or whatever) higher than median, indicate its more expensive than properties of this type.

Queried Data

Bedrooms = 3

Bathrooms = 2+

Sq Ft = 2000

Location = Zip codes 1, 2, 3, 4

The above prototype shows someone browsing a list of found properties. In the right hand column of the interface are the two concepts and a histogram to give home buyers more information about value:

the top icon is showing an estimation on a match (94 = 94% match to my wants) with my stated needs for a home

The bottom icon (below it) indicates “is it over priced or not?”. That calculation is based on an acceptable variance (maybe SD) amongst properties sold over the last 90 days that meet my stated needs for a property

(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-93946844-1', 'auto'); ga('send', 'pageview');

0 notes

iwebdatascrape · 1 year

Text

Web Scraping API Services | Scrape Realtime Website Data

Connect to your database & retrieve the extracted data using web scraping API services. Easily access clean and well-structured data with web scraping APIs.

#Web Scraping API Services #Scrape Realtime Website Data #Realtime Web Scraping API

0 notes

suzanneshannon · 4 years

Text

What the web still is

Being a pessimist is an easy thing to fall back on, and I’m trying to be better about it. As we close the year out, I thought it would be a good exercise to take stock of the state of the web and count our blessings.

Versatile

We don't use the internet to do just one thing. With more than two decades of globally interconnected computers, the web allows us to use it for all manner of activity.

This includes platforms, processes, and products that existed before the web came into being, and also previously unimagined concepts and behaviors. Thanks to the web, we can all order takeout the same way we can all watch two women repair a space station in realtime.

Decentralized

There is still no one single arbiter you need to petition to sign off on the validity of your idea, or one accepted path for going about to make it happen. Any website can link out to, or be linked to, without having to pay a tax or file pre-approval paperwork.

While we have seen a consolidation of the services needed to run more sophisticated web apps, you can still put your ideas out for the entire world to see with nothing more than a static HTML page. This fact was, and still is, historically unprecedented.

Resilient

The internet has been called the most hostile environment to develop for. Someone who works on the web has to consider multiple browsers, the operating systems they are installed on, and all the popular release versions of both. They also need to consider screen size and quality, variable network conditions, different form factors and input modes, third party scripts, etc. This is to say nothing about serving an unknown amount of unknown users, each with their own thoughts, feelings, goals, abilities, motivations, proficiencies, and device modifications.

If you do it right, you can build a website or a web app so that it can survive a lot of damage before it is rendered completely inoperable. Frankly, the fact that the web works at all is nothing short of miraculous.

The failsafes, guardrails, redundancies, and other considerations built into the platform from the packet level on up allow this to happen. Honoring them honors the thought, care, and planning that went into the web's foundational principles.

Responsive

Most websites now make use of media queries to ensure their content reads and works well across a staggeringly large amount of devices. This efficient technology choice is fault-tolerant, has a low barrier of entry, and neatly side-steps the myriad problems you get with approaches such as device-sniffing and/or conditionally serving massive piles of JavaScript.

Responsive Design was, and still is revolutionary. It was the right answer, at the right place and time. It elegantly handled the compounding problem of viewport fragmentation as the web transformed from something new and novel into something that is woven into our everyday lives.

Adaptable

In addition to being responsive, the web works across a huge range of form factors, device capabilities, and specialized browsing modes. The post you are currently reading can show up on a laptop, a phone, a Kindle, a TV, a gas station pump, a video game console, a refrigerator, a car, a billboard, an oscilloscope—heck, even a space shuttle (if you’re reading this from space, please, please, please let me know).

It will work with a reading mode that helps a person focus, dark and high contrast modes that will help a person see, and any number of specialized browser extensions that help people get what they need. I have a friend who inverts her entire display to help prevent triggering migraines, and the web just rolls with it. How great is that?

Web content can be read, translated, spoken aloud, copied, clipped, piped into your terminal, forked, remixed, scraped by a robot, output as Braille, and even played as music. You can increase the size of its text, change its font and color, and block parts you don't want to deal with—all in the service of making it easier for you to consume. That is revolutionary when compared to the media that came before it.

Furthermore, thanks to things like Progressive Web Apps and Web Platform Features, the web now blends seamlessly into desktops and home screens. These features allow web content to behave like traditional apps and are treated as first-class citizens by the operating systems that support them. You don’t even necessarily need to be online for them to work!

Accessible

The current landscape of accessibility compliance is a depressing state of affairs. WebAIM’s Million report, and subsequent update, highlights this with a sobering level of detail.

Out of the top one million websites sampled, ~98% of home pages had programmatically detectable Web Content Accessibility Guideline (WCAG) errors. This represents a complete, categorical failure of our industry on every conceivable level, from developers and designers, to framework maintainers, all the way up to those who help steer the future of the platform.

And yet.

In that last stubborn two percent lives a promise of the web. Web accessibility—the ability for someone to use a website or web app regardless of their ability or circumstance—grants autonomy. It represents a rare space where a disabled individual may operate free from the immense amount of bias, misunderstanding, and outright hate that is pervasive throughout much of society. This autonomy represents not only freedom for social activities but also employment opportunities for a population that is routinely discriminated against.

There is a ton of work to do, and we do not have the luxury of defeatism. I’m actually optimistic about digital accessibility’s future. Things like Inclusive Design have shifted the conversation away from remediation into a more holistic, proactive approach to product design.

Accessibility, long viewed as an unglamorous topic, has started to appear as a mainstream, top-level theme in conference and workshop circuits, as well as popular industry blogs. Sophisticated automated accessibility checkers can help prevent you from shipping inaccessible code. Design systems are helping to normalize the practice at scale. And most importantly, accessibility practitioners are speaking openly about ableism.

Inexpensive

While the average size of a website continues to rise, the fact remains that you can achieve an incredible amount of functionality with a small amount of code. That’s an important thing to keep in mind.

It has never been more affordable to use the web. In the United States, you can buy an internet-ready smartphone for ~$40. Emerging markets are adopting feature phones such as the JioPhone (~$15 USD) at an incredible rate. This means that access to the world’s information is available to more people—people who traditionally may have never been able to have such a privilege.

Think about it: owning a desktop computer represented having enough steady income to be able to support permanent housing, as well as consistent power and phone service. This created an implicit barrier to entry during the web’s infancy.

The weakening of this barrier opens up unimaginable amounts of opportunity, and is an excellent reminder that the web really is for everyone. With that in mind, it remains vital to keep our payload sizes down. What might be a reflexive CMD + R for you might be an entire week’s worth of data for someone else.

Diverse

There are more browsers available than I have fingers and toes to count on. This is a good thing. Like any other category of software, each browser is an app that does the same general thing in the same general way, but with specific design decisions made to prioritize different needs and goals.

My favorite browser, Firefox, puts a lot of its attention towards maintaining the privacy and security of its users. Brave is similar in that regard. Both Edge and Safari are bundled with their respective operating systems, and have interfaces geared towards helping the widest range of users browse web content. Browsers like Opera and Vivaldi are geared towards tinkerers, people who like a highly customized browsing experience. Samsung Internet is an alternative browser for Android devices that can integrate with their proprietary hardware. KaiOS and UC browsers provide access to millions of feature phones, helping them to have smartphone-esque functionality. Chrome helps you receive more personalized ads efficiently debug JavaScript.

Browser engine diversity is important as well, although the ecosystem has been getting disturbingly small as of late. The healthy competition multiple engines generates translates directly to the experience becoming better for the most important people in the room: Those who rely on the web to live their everyday lives.

Speaking of people, let’s discuss the web’s quality of diversity and how it applies to them: Our industry, like many others, has historically been plagued by ills such as misogyny, racism, homophobia, transphobia, and classism. However, the fact remains that the ability to solve problems in the digital space represents a rare form of leverage that allows minoritized groups to have upward economic mobility.

If you can't be motivated by human decency, it’s no secret that more diverse teams perform better. We’ve made good strides in the past few years towards better representation, but there’s still a lot of work to be done.

Listen to, and signal boost the triumphs, frustrations, and fears of the underrepresented in our industry. Internalize their observations and challenge your preconceived notions and biases. Advocate for their right to be in this space. Educate yourself on our industry’s history. Support things like codes of conduct, which do the hard work of modeling and codifying expectations for behavior. All of this helps to push against a toxic status quo and makes the industry better for everyone.

Standardized

The web is built by consensus, enabling a radical kind of functionality. This interoperability—the ability for different computer systems to be able to exchange information—is built from a set of standards we have all collectively agreed on.

Chances are good that a web document written two decades ago will still work with the latest version of any major browser. Any web document written by someone else—even someone on the opposite side of the globe—will also work. It will also continue to work on browsers and devices that have yet to be invented. I challenge you to name another file format that supports this level of functionality that has an equivalent lifespan.

This futureproofing by way of standardization also allows for a solid foundation of support for whatever comes next. Remember the principle of versatile: It is important to remember that these standards are also not prescriptive. We’re free to take these building blocks use arrange them in a near-infinite number of ways.

Open

Furthermore, this consensus is transparent. While the process may seem slow sometimes, it is worth highlighting the fact that the process is highly transparent. Anyone who is invested may follow, and contribute to web standards, warts and all.

It’s this openness that helps to prevent things like hidden agendas, privatization, lock-in, and disproportionate influence from consolidating power. Open-source software and protocols and, most importantly, large-scale cooperation also sustain the web platform’s long-term growth and health. Think of web technologies that didn’t make it: Flash, Silverlight, ActiveX, etc. All closed, for-profit, brittle, and private.

It also helps to disincentive more abstract threats, things like adversarial interoperability and failure to disclose vulnerabilities. These kinds of hazards are a good thing to remember any time you find yourself frustrated with the platform.

Make no mistake: I feel a lot of what makes the web great is actively being dismantled, either inadvertently or deliberately. But as I mentioned earlier, cynicism is easy. My wish for next year? That all the qualities mentioned here are still present. My New Year’s resolution? To help ensure it.

The post What the web still is appeared first on CSS-Tricks.

What the web still is published first on https://deskbysnafu.tumblr.com/

0 notes