#how to solve 403 forbidden error

[ecargmura]

Kowloon Generic Romance Episode 10 Review - 403 Forbidden

Now that the episode count is at the double digits, I do feel as if the mysteries are on the verge of being solved. Kujirai-B’s glasses seem to be a clue to the mysteries surrounding the original and Kowloon.  I also feel like Xiaohei’s existence might also be a major clue. I do wonder how everything will wrap up in just three more episodes…

The mysteries of Kowloon keep increasing as Kujirai’s glasses suddenly show images of her original’s memories. That’s odd. Her glasses never did that before, so why are they doing it now? Some of her memories show her images of an area that’s off-limits due to how dangerous it could be. Even in the original Second Kowloon, the area was known for being shady and dangerous to the point that Kudo banned Kujirai-B from going there again and it retained with the generic. When she went there with Yaomay, the talisman-riddled walls are of intrigue. When Kujirai peels one off, the back reads ‘403 Forbidden’, hinting that phantom world is a simulation. When she decides to go around to peel off all the talismans she can find, a few are blank and some have that same error code. However, one has 202 Accepted on the back and then the last one she finds before the episode ends is “Don’t find it anymore.” I feel like Generic Terra is trying to block her from snooping around too much—that’s why people who eat the food in this world lose their memories like how Yaomay did.

Kowloon is such a mystery. It’s not even erasing the generic Xiaohei despite the original roaming around in its walls. Is it because the Generic and the Original are physically different? Xiaohei is an unusual case where his current and his past self are extremely different. Is it because he used to live in Kowloon in the form of his younger self that the system counts the original as a different person? This means that Generic Terra isn’t completely perfect if it’s unable to detect that original Xiaohei and generic Xiaohei are the same… I think that could be its major weakness.

There also seems to be some sort of romance blooming with Xiaohei and Yaomay, but who knows if it will get anywhere given that Xiaohei is conflicted on whether he should do as Yulong says or if he should continue his mission as a spy. Killing Generic Kujirai can also mean the end of a friendship with Yaomay, after all. Speaking of Yaomay, I do like that she’s essentially the key to solving everything in a way as she’s the only person in the story with no real connection to Kowloon other than stumbling upon it one day. Eating the chocolate that Yulong gave to her during her trip to Hong Kong does seem like eating food outside of Kowloon regains people’s memories—either that or Yulong is just some crazy scientist that knows how to counter Kowloon mindfuckery in the form of candy.

I can’t wait to see what the epic conclusion to this anime will be like. This is one of the shows I enjoy watching every week because it just gets me thinking. It also makes me anticipate every Saturday. Let me know your thoughts on this episode!

[crawlxpert01]

Overcoming Bot Detection While Scraping Menu Data from UberEats, DoorDash, and Just Eat

Introduction

In industries where menu data collection is concerned, web scraping would serve very well for us: UberEats, DoorDash, and Just Eat are the some examples. However, websites use very elaborate bot detection methods to stop the automated collection of information. In overcoming these factors, advanced scraping techniques would apply with huge relevance: rotating IPs, headless browsing, CAPTCHA solving, and AI methodology.

This guide will discuss how to bypass bot detection during menu data scraping and all challenges with the best practices for seamless and ethical data extraction.

Understanding Bot Detection on Food Delivery Platforms

1. Common Bot Detection Techniques

Food delivery platforms use various methods to block automated scrapers:

IP Blocking – Detects repeated requests from the same IP and blocks access.

User-Agent Tracking – Identifies and blocks non-human browsing patterns.

CAPTCHA Challenges – Requires solving puzzles to verify human presence.

JavaScript Challenges – Uses scripts to detect bots attempting to load pages without interaction.

Behavioral Analysis – Tracks mouse movements, scrolling, and keystrokes to differentiate bots from humans.

2. Rate Limiting and Request Patterns

Platforms monitor the frequency of requests coming from a specific IP or user session. If a scraper makes too many requests within a short time frame, it triggers rate limiting, causing the scraper to receive 403 Forbidden or 429 Too Many Requests errors.

3. Device Fingerprinting

Many websites use sophisticated techniques to detect unique attributes of a browser and device. This includes screen resolution, installed plugins, and system fonts. If a scraper runs on a known bot signature, it gets flagged.

Techniques to Overcome Bot Detection

1. IP Rotation and Proxy Management

Using a pool of rotating IPs helps avoid detection and blocking.

Use residential proxies instead of data center IPs.

Rotate IPs with each request to simulate different users.

Leverage proxy providers like Bright Data, ScraperAPI, and Smartproxy.

Implement session-based IP switching to maintain persistence.

2. Mimic Human Browsing Behavior

To appear more human-like, scrapers should:

Introduce random time delays between requests.

Use headless browsers like Puppeteer or Playwright to simulate real interactions.

Scroll pages and click elements programmatically to mimic real user behavior.

Randomize mouse movements and keyboard inputs.

Avoid loading pages at robotic speeds; introduce a natural browsing flow.

3. Bypassing CAPTCHA Challenges

Implement automated CAPTCHA-solving services like 2Captcha, Anti-Captcha, or DeathByCaptcha.

Use machine learning models to recognize and solve simple CAPTCHAs.

Avoid triggering CAPTCHAs by limiting request frequency and mimicking human navigation.

Employ AI-based CAPTCHA solvers that use pattern recognition to bypass common challenges.

4. Handling JavaScript-Rendered Content

Use Selenium, Puppeteer, or Playwright to interact with JavaScript-heavy pages.

Extract data directly from network requests instead of parsing the rendered HTML.

Load pages dynamically to prevent detection through static scrapers.

Emulate browser interactions by executing JavaScript code as real users would.

Cache previously scraped data to minimize redundant requests.

5. API-Based Extraction (Where Possible)

Some food delivery platforms offer APIs to access menu data. If available:

Check the official API documentation for pricing and access conditions.

Use API keys responsibly and avoid exceeding rate limits.

Combine API-based and web scraping approaches for optimal efficiency.

6. Using AI for Advanced Scraping

Machine learning models can help scrapers adapt to evolving anti-bot measures by:

Detecting and avoiding honeypots designed to catch bots.

Using natural language processing (NLP) to extract and categorize menu data efficiently.

Predicting changes in website structure to maintain scraper functionality.

Best Practices for Ethical Web Scraping

While overcoming bot detection is necessary, ethical web scraping ensures compliance with legal and industry standards:

Respect Robots.txt – Follow site policies on data access.

Avoid Excessive Requests – Scrape efficiently to prevent server overload.

Use Data Responsibly – Extracted data should be used for legitimate business insights only.

Maintain Transparency – If possible, obtain permission before scraping sensitive data.

Ensure Data Accuracy – Validate extracted data to avoid misleading information.

Challenges and Solutions for Long-Term Scraping Success

1. Managing Dynamic Website Changes

Food delivery platforms frequently update their website structure. Strategies to mitigate this include:

Monitoring website changes with automated UI tests.

Using XPath selectors instead of fixed HTML elements.

Implementing fallback scraping techniques in case of site modifications.

2. Avoiding Account Bans and Detection

If scraping requires logging into an account, prevent bans by:

Using multiple accounts to distribute request loads.

Avoiding excessive logins from the same device or IP.

Randomizing browser fingerprints using tools like Multilogin.

3. Cost Considerations for Large-Scale Scraping

Maintaining an advanced scraping infrastructure can be expensive. Cost optimization strategies include:

Using serverless functions to run scrapers on demand.

Choosing affordable proxy providers that balance performance and cost.

Optimizing scraper efficiency to reduce unnecessary requests.

Future Trends in Web Scraping for Food Delivery Data

As web scraping evolves, new advancements are shaping how businesses collect menu data:

AI-Powered Scrapers – Machine learning models will adapt more efficiently to website changes.

Increased Use of APIs – Companies will increasingly rely on API access instead of web scraping.

Stronger Anti-Scraping Technologies – Platforms will develop more advanced security measures.

Ethical Scraping Frameworks – Legal guidelines and compliance measures will become more standardized.

Conclusion

Uber Eats, DoorDash, and Just Eat represent great challenges for menu data scraping, mainly due to their advanced bot detection systems. Nevertheless, if IP rotation, headless browsing, solutions to CAPTCHA, and JavaScript execution methodologies, augmented with AI tools, are applied, businesses can easily scrape valuable data without incurring the wrath of anti-scraping measures.

If you are an automated and reliable web scraper, CrawlXpert is the solution for you, which specializes in tools and services to extract menu data with efficiency while staying legally and ethically compliant. The right techniques, along with updates on recent trends in web scrapping, will keep the food delivery data collection effort successful long into the foreseeable future.

Know More : https://www.crawlxpert.com/blog/scraping-menu-data-from-ubereats-doordash-and-just-eat

[wordpresshelper]

How to Speeding Up WordPress Site?

A Guide to Speeding up WordPress: 

 When a person lands on your site for the first time, you only have a few seconds to capture their attention to convince them to hang around.

 Get ready to lose sleep at night: according to a report by the Microsoft Bing search team, a 2-second longer delay in page responsiveness reduced user satisfaction by 3.8%, increased lost revenue per user by 4.3%, and reduced clicks by 4.3%.

 If your site takes too long to load, most people are gone, lost before you even had a chance.

 Not only that, but Google now includes site speed in its ranking algorithm. That means that your site’s speed effects SEO, so if your site is slow, you’re now losing visitors from impatience and reduced rankings in search engines. Yikes.

 1. Choose a better web hosting provider

 The major factor that influences the speed of a website is the hosting of your WordPress website. It might seem like a good idea to host your new website on a shared hosting provider that offers “unlimited” bandwidth, space, emails, domains and more. However, the point that we usually miss out on regarding this offer is that shared hosting environments fail to deliver good loading times on peak traffic hours, and most fail to provide 99 percent uptime in any given month.

 Shared hosting tends to deliver a poorer performance because you are sharing the same server space with countless other websites, and there is no telling how much resources others are using. Plus, you don’t know exactly how well the servers are optimized.

 Thankfully, the web-hosting industry has advanced with technology, and the prices of cloud hosting providers have decreased with the passage of time. In the present times, you can buy dedicated cloud servers from SiteGround, DigitalOcean, Amazon Web Services, and even Google Compute Engine at a nominal price. However, setting those servers up can be a daunting task as you are required to set servers up from scratch. There are web hosting providers like Cloudways (where I work) who make the task of setting up optimized cloud servers as easy as click and launch. You can read more about the process here: how to set up WordPress through Cloudways.

 2. Use a lightweight WordPress theme/framework

 WordPress themes (WordPress Theme Development) with a lot of dynamic elements, sliders, widgets, social icons, and many  elements and higher page sizes, then they will definitely cause your web server to take a thumping.

 The best option here is to use lightweight themes. One solution is to go for one of the default WordPress themes. Another is to try out something like Neve, built by the same guys behind CodeinWP.

 Alternatively, for a feature-rich website, you can also opt for a theme that uses a good framework like Bootstrap or Foundation.

 3. Reduce image sizes

 Images are the major contributors to the size increment of a given webpage. The trick is to reduce the size of the images without compromising on the quality.

 If you manually optimize the images using Chrome PageSpeed Insights extension or Photoshop or any other tools, the process will take a long time. Fortunately, there are plugins available for just about everything you can think of, including image optimization. The ones worth mentioning are:

 Optimal

WP Smush

EWWW Image Optimizer

 Using any of the above-mentioned plugins on your WordPress site will drastically reduce image sizes, thus improving the speed of your website.

 4. Minify JS and CSS files

 If you run your website through the Google PageSpeed Insights tool, you will probably be notified about minimizing the size of your CSS and JS files. What this means is that by reducing the number of CSS and JS calls and the size of those files, you can improve the site loading speed.

 Also, if you know your way around WordPress themes, you can study the guides provided by Google and do some manual fixing. If not, then there are plugins that will help you achieve this goal; the most popular being the Autoptimize that can help in optimizing CSS, JS and even HTML of your WordPress website.

 5. Use advanced caching mechanisms with a caching plugin

WordPress caching plugins (e.g. W3 Total Cache) has been there for a long time, making the complex tasks of adding caching rules to your website elements easier. Combining such plug-ins with advanced caching mechanisms like Varnish could help you better the loading speed of your website and ultimately speed up WordPress considerably.

 6. Use a CDN

 The people who visit your website belong to various locations in the world, and needless to say, the site-loading speed will differ if the visitors are located far away from where your site is hosted. There are many CDN (Content Delivery Networks) that help in keeping the site-loading speed to a minimum for visitors from various countries. A CDN keeps a copy of your website in various data centers located in different places. The primary function of a CDN is to serve the webpage to a visitor from the nearest possible location. Cloudflare and MaxCDN are among the most popular CDN services.

 7. Cleanup WordPress database

 Deleting unwanted data from your database will keep its size to a minimum and also helps in reducing the size of your backups. It is also necessary to delete spam comments, fake users, old drafts of your content and maybe even unwanted plugins as well as themes. All of this will reduce the size of your databases and web files, and thus speed up WordPress – your WordPress. (Fix error establishing database connection)

 8. Deactivate or uninstall plugins

 Keeping unwanted plugins on your WordPress websites will add a tremendous amount of junk to your web files. Moreover, it will also increase the size of your backup and put an overwhelming amount of load on your server resources while backup files are being generated. It is better to get rid of the plugins that you don’t use and also look for alternate methods to use third-party services for automating or scheduling tasks (like sharing of your latest posts to social media).

 IFTTT or Zapier are two web services that help in automating such tasks and reduce the burden on your website and server resources.

 9. Keep external scripts to a minimum

 The usage of external scripts on your web pages adds a big chunk of data to your total loading time. Thus, it is best to use a low number of scripts, including only the essentials such as tracking tools (like Google Analytics) or commenting systems (like Disqus).

 10. Disable pingbacks and trackbacks

Pingbacks and trackbacks are two core WordPress components that alert you whenever your blog or page receives a link. It might sound useful, but you also have things such as Google Webmaster Tools and other services to check the links of your website.

 Keeping pingbacks and trackbacks on can also put an undesirable amount of strain on your server resources. This is so because whenever anyone tries to link up to your site, it generates requests from WordPress back and forth. This functionality is also widely abused when targeting a website with DDoS attacks.

 You can turn it all off in WP-Admin → Settings → Discussion. Just deselect “Allow link notifications from other blogs (pingbacks and trackbacks).” This will help you speed up WordPress some more.

 Conclusion

 The biggest advantage of lowering your website’s loading time is that it will help tremendously in improving the experience of your visitors. The case remains the same whether they are using mobile devices or PCs. Furthermore, it will also improve your rankings in the SERPs. After all, reduced bandwidth usage of your hosting and faster site-loading speed on the client-side will only benefit you both in the short as well as in the long run.

 Call Us at 888-606-1808 (US Toll Free) to get free assistance for your WordPress Website. 

[robinrakibul]

I will fix wordpress errors, issues, problems, bugs quickly

Are you facing errors or issues on your WordPress site? Don't you have time to fix it or don't know how to fix it? Want to fix WordPress errors, issues, problems, bugs quickly? Then let me solve the problems now!

https://www.fiverr.com/s2/b81ccc271b

I can help you to fix:

White Screen of Death

HTTP Error 500

403 forbidden error

Error Establishing Database Connection

Errors in Moving WordPress website to another server

Internal server 404 / 505 errors

Fix website issues and WooCommerce Errors

Forgot the admin panel password?

Website crash

Website Migration or Backup

WordPress Maintenance

Customization of theme

WordPress Memory Error

Php error

Fix plugin issues

Fix theme issues

And more issues or problems related to WordPress can be done here!

The reasons why you can choose me,Advanced & practical experience with WordPress websites,Quick solutions,Friendly communication,100% client satisfaction

Thank you for your time. You can go ahead and order directly. If you have any queries, feel free to message me on Fiverr.

https://www.fiverr.com/s2/b81ccc271b

[globalmediacampaign]

#WDILTW – Creating examples can be hard

This week I was evaluating AWS QLDB. Specifically the verifiable history of changes to determine how to simplify present processes that perform auditing via CDC. This is not the first time I have looked at QLDB so there was nothing that new to learn. What I found was that creating a workable solution with an existing application is hard. Even harder is creating an example to publish in this blog (and the purpose of this post). First some background. Using MySQL as the source of information, how can you leverage QLDB? It’s easy to stream data from MySQL Aurora, and it’s easy to stream data from QLDB, but it not that easy to place real-time data into QLDB. AWS DMS is a good way to move data from a source to a target, previously my work has included MySQL to MySQL, MySQL to Redshift, and MySQL to Kinesis, however there is no QLDB target. Turning the problem upside down, and using QLDB as the source of information, and streaming to MySQL for compatibility seemed a way forward. After setting up the QLDB Ledger and an example table, it was time to populate with existing data. The documented reference example looked very JSON compatible. Side bar, it is actually Amazon Ion a superset of JSON. INSERT INTO Person Now, MySQL offers with the X Protocol. This is something that lefred has evangelized for many years, I have seen presented many times, but finally I had a chance to use. The MySQL Shell JSON output looked ideal. { "ID": 1523, "Name": "Wien", "CountryCode": "AUT", "District": "Wien", "Info": { "Population": 1608144 } } { "ID": 1524, "Name": "Graz", "CountryCode": "AUT", "District": "Steiermark", "Info": { "Population": 240967 } } And now, onto some of the things I learned this week. Using AWS RDS Aurora MySQL is the first stumbling block, X Protocol is not supported. As this was a example, simple, mysqldump some reference data and load it into a MySQL 8 instance, and extract into JSON, so as to potentially emulate a pipeline. Here is my experiences of trying to refactor into a demo to write up. Launch a MySQL Docker container as per my standard notes. Harmless, right? MYSQL_ROOT_PASSWORD="$(date | md5sum | cut -c1-20)#" echo $MYSQL_ROOT_PASSWORD docker run --name=qldb-mysql -p3306:3306 -v mysql-volume:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD -d mysql/mysql-server:latest docker logs qldb-mysql docker exec -it qldb-mysql /bin/bash As it's a quick demo, I shortcut credentials to make using the mysql client easier. NOTE: as I always generate a new password each container, it's included here. # echo "[mysql] user=root password='ab6ea7b0436cbc0c0d49#' > .my.cnf # mysql ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) What the? Did I make a mistake, I test manually and check # mysql -u root -p # cat .my.cnf Nothing wrong there. Next check # pwd / bash-4.2# grep root /etc/passwd root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin And there is the first Dockerism. I don't live in Docker, so these 101 learnings would be known. First I really thing using "root" by default is a horrible idea. And when you shell in, you are not dropped into the home directory? Solved, we move on. # mv /.my.cnf /root/.my.cnf Mock and example as quickly as I can think. # mysql mysql> create schema if not exists demo; Query OK, 1 row affected (0.00 sec) mysql> use demo; Database changed mysql> create table sample(id int unsigned not null auto_increment, name varchar(30) not null, location varchar(30) not null, domain varchar(50) null, primary key(id)); Query OK, 0 rows affected (0.03 sec) mysql> show create table sample; mysql> insert into sample values (null,'Demo Row','USA',null), (null,'Row 2','AUS','news.com.au'), (null,'Kiwi','NZ', null); Query OK, 3 rows affected (0.00 sec) Records: 3 Duplicates: 0 Warnings: 0 mysql> select * from sample; +----+----------+----------+-------------+ | id | name | location | domain | +----+----------+----------+-------------+ | 1 | Demo Row | USA | NULL | | 2 | Row 2 | AUS | news.com.au | | 3 | Kiwi | NZ | NULL | +----+----------+----------+-------------+ 3 rows in set (0.00 sec) Cool, now to look at it in Javascript using MySQL Shell. Hurdle 2. # mysqlsh MySQL Shell 8.0.22 Copyright (c) 2016, 2020, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. MySQL JS > var session=mysqlx.getSession('root:ab6ea7b0436cbc0c0d49#@localhost') mysqlx.getSession: Argument #1: Invalid URI: Illegal character [#] found at position 25 (ArgumentError) What the, it doesn't like the password format. I'm not a Javascript person, and well this is an example for blogging, which is not what was actually setup, so do it the right way, create a user. # mysql mysql> create user demo@localhost identified by 'qldb'; Query OK, 0 rows affected (0.01 sec) mysql> grant ALL ON sample.* to demo@localhost; Query OK, 0 rows affected, 1 warning (0.01 sec) mysql> SHOW GRANTS FOR demo@localhost; +----------------------------------------------------------+ | Grants for demo@localhost | +----------------------------------------------------------+ | GRANT USAGE ON *.* TO `demo`@`localhost` | | GRANT ALL PRIVILEGES ON `sample`.* TO `demo`@`localhost` | +----------------------------------------------------------+ 2 rows in set (0.00 sec) Back into the MySQL Shell, and hurdle 3. MySQL JS > var session=mysqlx.getSession('demo:qldb@localhost') mysqlx.getSession: Access denied for user 'demo'@'127.0.0.1' (using password: YES) (MySQL Error 1045) Did I create the creds wrong, verify. No my password is correct. # mysql -udemo -pqldb -e "SELECT NOW()" mysql: [Warning] Using a password on the command line interface can be insecure. +---------------------+ | NOW() | +---------------------+ | 2021-03-06 23:15:26 | +---------------------+ I don't have time to debug this, User take 2. mysql> drop user demo@localhost; Query OK, 0 rows affected (0.00 sec) mysql> create user demo@'%' identified by 'qldb'; Query OK, 0 rows affected (0.01 sec) mysql> grant all on demo.* to demo@'%' -> ; Query OK, 0 rows affected (0.00 sec) mysql> show grants; +-- | Grants for root@localhost | +--- | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `root`@`localhost` WITH GRANT OPTION | | GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,XA_RECOVER_ADMIN ON *.* TO `root`@`localhost` WITH GRANT OPTION | | GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION | +--- 3 rows in set (0.00 sec) mysql> show grants for demo@'%'; +--------------------------------------------------+ | Grants for demo@% | +--------------------------------------------------+ | GRANT USAGE ON *.* TO `demo`@`%` | | GRANT ALL PRIVILEGES ON `demo`.* TO `demo`@`%` | +--------------------------------------------------+ 2 rows in set (0.00 sec) Right, initially I showed grants of not new user, but note to self, I should checkout the MySQL 8 Improved grants. I wonder how RDS MySQL 8 handles these, and how Aurora MySQL 8 will (when it ever drops, that's another story). Third try is a charm, so nice to also see queries with 0.0000 execution granularity. MySQL JS > var session=mysqlx.getSession('demo:qldb@localhost') MySQL JS > var sql='SELECT * FROM demo.sample' MySQL JS > session.sql(sql) +----+----------+----------+-------------+ | id | name | location | domain | +----+----------+----------+-------------+ | 1 | Demo Row | USA | NULL | | 2 | Row 2 | AUS | news.com.au | | 3 | Kiwi | NZ | NULL | +----+----------+----------+-------------+ 3 rows in set (0.0006 sec) Get that now in JSON output. NOTE: There are 3 different JSON formats, this matched what I needed. bash-4.2# mysqlsh MySQL Shell 8.0.22 Copyright (c) 2016, 2020, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help' or '?' for help; 'quit' to exit. MySQL JS > var session=mysqlx.getSession('demo:qldb@localhost') MySQL JS > var sql='SELECT * FROM demo.sample' MySQL JS > shell.options.set('resultFormat','json/array') MySQL JS > session.sql(sql) [ {"id":1,"name":"Demo Row","location":"USA","domain":null}, {"id":2,"name":"Row 2","location":"AUS","domain":"news.com.au"}, {"id":3,"name":"Kiwi","location":"NZ","domain":null} ] 3 rows in set (0.0006 sec) Ok, that works in interactive interface, I need it scripted. # vi bash: vi: command not found # yum install vi Loaded plugins: ovl http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden Trying other mirror. ... And another downer of Docker containers, other tools or easy ways to install them, again I want to focus on the actual example, and not all this preamble, so # echo "var session=mysqlx.getSession('demo:qldb@localhost') var sql='SELECT * FROM demo.sample' shell.options.set('resultFormat','json/array') session.sql(sql)" > dump.js # mysqlsh What the? Hurdle 4. Did I typo this as well, I check the file, and cut/paste it and get what I expect. # cat dump.js var session=mysqlx.getSession('demo:qldb@localhost') var sql='SELECT * FROM demo.sample' shell.options.set('resultFormat','json/array') session.sql(sql) # mysqlsh MySQL Shell 8.0.22 Copyright (c) 2016, 2020, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help' or '?' for help; 'quit' to exit. MySQL JS > var session=mysqlx.getSession('demo:qldb@localhost') MySQL JS > var sql='SELECT * FROM demo.sample' MySQL JS > shell.options.set('resultFormat','json/array') MySQL JS > session.sql(sql) [ {"id":1,"name":"Demo Row","location":"USA","domain":null}, {"id":2,"name":"Row 2","location":"AUS","domain":"news.com.au"}, {"id":3,"name":"Kiwi","location":"NZ","domain":null} ] 3 rows in set (0.0022 sec) This is getting crazy. # echo '[ > {"id":1,"name":"Demo Row","location":"USA","domain":null}, > {"id":2,"name":"Row 2","location":"AUS","domain":"news.com.au"}, > {"id":3,"name":"Kiwi","location":"NZ","domain":null} > ]' > sample.json bash-4.2# jq . sample.json bash: jq: command not found Oh the docker!!!!. Switching back to my EC2 instance now. $ echo '[ > {"id":1,"name":"Demo Row","location":"USA","domain":null}, > {"id":2,"name":"Row 2","location":"AUS","domain":"news.com.au"}, > {"id":3,"name":"Kiwi","location":"NZ","domain":null} > ]' > sample.json $ jq . sample.json [ { "id": 1, "name": "Demo Row", "location": "USA", "domain": null }, { "id": 2, "name": "Row 2", "location": "AUS", "domain": "news.com.au" }, { "id": 3, "name": "Kiwi", "location": "NZ", "domain": null } ] I am now way of the time I would like to spend on this weekly post, and it's getting way to long, and I'm nowhere near showing what I actually want. Still we trek on. Boy, this stock EC2 image uses version 1, we need I'm sure V2, and well command does not work!!!! $ aws qldb list-ledgers ERROR: $ aws --version $ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" $ unzip awscliv2.zip $ sudo ./aws/install $ export PATH=/usr/local/bin:$PATH $ aws --version Can I finally get a ledger now. $ aws qldb create-ledger --name demo --tags JIRA=DEMO-5826,Owner=RonaldBradford --permissions-mode ALLOW_ALL --no-deletion-protection { "Name": "demo", "Arn": "arn:aws:qldb:us-east-1:999:ledger/demo", "State": "CREATING", "CreationDateTime": "2021-03-06T22:46:41.760000+00:00", "DeletionProtection": false } $ aws qldb list-ledgers { "Ledgers": [ { "Name": "xx", "State": "ACTIVE", "CreationDateTime": "2021-03-05T20:12:44.611000+00:00" }, { "Name": "demo", "State": "ACTIVE", "CreationDateTime": "2021-03-06T22:46:41.760000+00:00" } ] } $ aws qldb describe-ledger --name demo { "Name": "demo", "Arn": "arn:aws:qldb:us-east-1:999:ledger/demo", "State": "ACTIVE", "CreationDateTime": "2021-03-06T22:46:41.760000+00:00", "DeletionProtection": false } Oh the Python 2, and the lack of user packaging, more crud of getting an example. $ pip install pyqldb==3.1.0 ERROR $ echo "alias python=python3 alias pip=pip3" >> ~/.bash_profile source ~/.bash_profile $ pip --version pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6) $ python --version Python 3.6.8 $ pip install pyqldb==3.1.0 ERROR $ sudo pip install pyqldb==3.1.0 Yeah!, after all that, my example code works and data is inserted. $ cat demo.py from pyqldb.config.retry_config import RetryConfig from pyqldb.driver.qldb_driver import QldbDriver # Configure retry limit to 3 retry_config = RetryConfig(retry_limit=3) # Initialize the driver print("Initializing the driver") qldb_driver = QldbDriver("demo", retry_config=retry_config) def create_table(transaction_executor, table): print("Creating table {}".format(table)) transaction_executor.execute_statement("Create TABLE {}".format(table)) def create_index(transaction_executor, table, column): print("Creating index {}.{}".format(table, column)) transaction_executor.execute_statement("CREATE INDEX ON {}({})".format(table,column)) def insert_record(transaction_executor, table, values): print("Inserting into {}".format(table)) transaction_executor.execute_statement("INSERT INTO {} ?".format(table), values) table="sample" column="id" qldb_driver.execute_lambda(lambda executor: create_table(executor, table)) qldb_driver.execute_lambda(lambda executor: create_index(executor, table, column)) record1 = { 'id': "1", 'name': "Demo Row", 'location': "USA", 'domain': "" } qldb_driver.execute_lambda(lambda x: insert_record(x, table, record1)) $ python demo.py Initializing the driver Creating table sample Creating index sample.id Inserting into sample One vets in the AWS Console, but you cannot show that in text in this blog, so goes to find a simple client and there is qldbshell What the? I installed it and it complains about pyqldb.driver.pooled_qldb_driver. I literally used that in the last example. $ pip3 install qldbshell Collecting qldbshell Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/0f/f7/fe984d797e0882c5e141a4888709ae958eb8c48007a23e94000507439f83/qldbshell-1.2.0.tar.gz (68kB) 100% |████████████████████████████████| 71kB 55.6MB/s Requirement already satisfied: boto3>=1.9.237 in /usr/local/lib/python3.6/site-packages (from qldbshell) Collecting amazon.ion=0.5.0 (from qldbshell) Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/4e/b7/21b7a7577cc6864d1c93fd710701e4764af6cf0f7be36fae4f9673ae11fc/amazon.ion-0.5.0.tar.gz (178kB) 100% |████████████████████████████████| 184kB 78.7MB/s Requirement already satisfied: prompt_toolkit=3.0.5 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: ionhash~=1.1.0 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: s3transfer=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: jmespath=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: botocore=1.20.21 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from amazon.ion=0.5.0->qldbshell) Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/site-packages (from prompt_toolkit=3.0.5->qldbshell) Requirement already satisfied: python-dateutil=2.1 in /usr/local/lib/python3.6/site-packages (from botocore=1.20.21->boto3>=1.9.237->qldbshell) Requirement already satisfied: urllib3=1.25.4 in /usr/local/lib/python3.6/site-packages (from botocore=1.20.21->boto3>=1.9.237->qldbshell) Installing collected packages: amazon.ion, qldbshell Found existing installation: amazon.ion 0.7.0 Uninstalling amazon.ion-0.7.0: Exception: Traceback (most recent call last): File "/usr/lib64/python3.6/shutil.py", line 550, in move os.rename(src, real_dst) PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/amazon.ion-0.7.0-py3.6-nspkg.pth' -> '/tmp/pip-p8j4d45d-uninstall/usr/local/lib/python3.6/site-packages/amazon.ion-0.7.0-py3.6-nspkg.pth' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 365, in run strip_file_prefix=options.strip_file_prefix, File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 783, in install requirement.uninstall(auto_confirm=True) File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 754, in uninstall paths_to_remove.remove(auto_confirm) File "/usr/lib/python3.6/site-packages/pip/req/req_uninstall.py", line 115, in remove renames(path, new_path) File "/usr/lib/python3.6/site-packages/pip/utils/__init__.py", line 267, in renames shutil.move(old, new) File "/usr/lib64/python3.6/shutil.py", line 565, in move os.unlink(src) PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/amazon.ion-0.7.0-py3.6-nspkg.pth' [centos@ip-10-204-101-224] ~ $ sudo pip3 install qldbshell WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. Collecting qldbshell Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/0f/f7/fe984d797e0882c5e141a4888709ae958eb8c48007a23e94000507439f83/qldbshell-1.2.0.tar.gz (68kB) 100% |████████████████████████████████| 71kB 49.8MB/s Requirement already satisfied: boto3>=1.9.237 in /usr/local/lib/python3.6/site-packages (from qldbshell) Collecting amazon.ion=0.5.0 (from qldbshell) Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/4e/b7/21b7a7577cc6864d1c93fd710701e4764af6cf0f7be36fae4f9673ae11fc/amazon.ion-0.5.0.tar.gz (178kB) 100% |████████████████████████████████| 184kB 27.7MB/s Requirement already satisfied: prompt_toolkit=3.0.5 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: ionhash~=1.1.0 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: botocore=1.20.21 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: jmespath=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: s3transfer=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from amazon.ion=0.5.0->qldbshell) Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/site-packages (from prompt_toolkit=3.0.5->qldbshell) Requirement already satisfied: python-dateutil=2.1 in /usr/local/lib/python3.6/site-packages (from botocore=1.20.21->boto3>=1.9.237->qldbshell) Requirement already satisfied: urllib3=1.25.4 in /usr/local/lib/python3.6/site-packages (from botocore=1.20.21->boto3>=1.9.237->qldbshell) Installing collected packages: amazon.ion, qldbshell Found existing installation: amazon.ion 0.7.0 Uninstalling amazon.ion-0.7.0: Successfully uninstalled amazon.ion-0.7.0 Running setup.py install for amazon.ion ... done Running setup.py install for qldbshell ... done Successfully installed amazon.ion-0.5.0 qldbshell-1.2.0 $ sudo pip3 install qldbshell $ qldbshell Traceback (most recent call last): File "/usr/local/bin/qldbshell", line 11, in load_entry_point('qldbshell==1.2.0', 'console_scripts', 'qldbshell')() File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 476, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2700, in load_entry_point return ep.load() File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2318, in load return self.resolve() File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/usr/local/lib/python3.6/site-packages/qldbshell/__main__.py", line 25, in from pyqldb.driver.pooled_qldb_driver import PooledQldbDriver ModuleNotFoundError: No module named 'pyqldb.driver.pooled_qldb_driver' $ pip list qldbshell DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. amazon.ion (0.5.0) boto3 (1.17.21) botocore (1.20.21) ionhash (1.1.0) jmespath (0.10.0) pip (9.0.3) prompt-toolkit (3.0.16) pyqldb (3.1.0) python-dateutil (2.8.1) qldbshell (1.2.0) s3transfer (0.3.4) setuptools (39.2.0) six (1.15.0) urllib3 (1.26.3) So, uninstalled and re-installed and voila, my data. $ qldbshell usage: qldbshell [-h] [-v] [-s QLDB_SESSION_ENDPOINT] [-r REGION] [-p PROFILE] -l LEDGER qldbshell: error: the following arguments are required: -l/--ledger $ qldbshell -l demo Welcome to the Amazon QLDB Shell version 1.2.0 Use 'start' to initiate and interact with a transaction. 'commit' and 'abort' to commit or abort a transaction. Use 'start; statement 1; statement 2; commit; start; statement 3; commit' to create transactions non-interactively. Use 'help' for the help section. All other commands will be interpreted as PartiQL statements until the 'exit' or 'quit' command is issued. qldbshell > qldbshell > SELECT * FROM sample; INFO: { id: "1", name: "Demo Row", location: "USA", domain: "" } INFO: (0.1718s) qldbshell > q WARNING: Error while executing query: An error occurred (BadRequestException) when calling the SendCommand operation: Lexer Error: at line 1, column 1: invalid character at, '' [U+5c]; INFO: (0.1134s) qldbshell > exit Exiting QLDB Shell Right q is a mysqlism of the client, need to rewire myself. Now, I have a ledger, I created an example table, mocked a row of data and verified. Now I can just load my sample data in JSON I created earlier right? Wrong!!! $ cat load.py import json from pyqldb.config.retry_config import RetryConfig from pyqldb.driver.qldb_driver import QldbDriver # Configure retry limit to 3 retry_config = RetryConfig(retry_limit=3) # Initialize the driver print("Initializing the driver") qldb_driver = QldbDriver("demo", retry_config=retry_config) def insert_record(transaction_executor, table, values): print("Inserting into {}".format(table)) transaction_executor.execute_statement("INSERT INTO {} ?".format(table), values) table="sample" with open('sample.json') as f: data=json.load(f) qldb_driver.execute_lambda(lambda x: insert_record(x, table, data)) $ python load.py Traceback (most recent call last): File "load.py", line 2, in from pyqldb.config.retry_config import RetryConfig ModuleNotFoundError: No module named 'pyqldb' [centos@ip-10-204-101-224] ~ Oh sweet, I'd installed that, and used it, and re-installed it. $ pip list | grep pyqldb DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. [centos@ip-10-204-101-224] ~ $ sudo pip3 install pyqldb WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. Collecting pyqldb Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/5c/b4/9790b1fad87d7df5b863cbf353689db145bd009d31d854d282b31e1c1781/pyqldb-3.1.0.tar.gz Collecting amazon.ion=0.7.0 (from pyqldb) Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/7d/ac/fd1edee54cefa425c444b51ad00a20e5bc74263a3afbfd4c8743040f8f26/amazon.ion-0.7.0.tar.gz (211kB) 100% |████████████████████████████████| 215kB 24.8MB/s Requirement already satisfied: boto3=1.16.56 in /usr/local/lib/python3.6/site-packages (from pyqldb) Requirement already satisfied: botocore=1.19.56 in /usr/local/lib/python3.6/site-packages (from pyqldb) Requirement already satisfied: ionhash=1.1.0 in /usr/local/lib/python3.6/site-packages (from pyqldb) Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from amazon.ion=0.7.0->pyqldb) Requirement already satisfied: s3transfer=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3=1.16.56->pyqldb) Requirement already satisfied: jmespath=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3=1.16.56->pyqldb) Requirement already satisfied: python-dateutil=2.1 in /usr/local/lib/python3.6/site-packages (from botocore=1.19.56->pyqldb) Requirement already satisfied: urllib3=1.25.4 in /usr/local/lib/python3.6/site-packages (from botocore=1.19.56->pyqldb) Installing collected packages: amazon.ion, pyqldb Found existing installation: amazon.ion 0.5.0 Uninstalling amazon.ion-0.5.0: Successfully uninstalled amazon.ion-0.5.0 Running setup.py install for amazon.ion ... done Running setup.py install for pyqldb ... done Successfully installed amazon.ion-0.7.0 pyqldb-3.1.0 Load one more time. $ cat load.py import json from pyqldb.config.retry_config import RetryConfig from pyqldb.driver.qldb_driver import QldbDriver # Configure retry limit to 3 retry_config = RetryConfig(retry_limit=3) # Initialize the driver print("Initializing the driver") qldb_driver = QldbDriver("demo", retry_config=retry_config) def insert_record(transaction_executor, table, values): print("Inserting into {}".format(table)) transaction_executor.execute_statement("INSERT INTO {} ?".format(table), values) table="sample" with open('sample.json') as f: data=json.load(f) qldb_driver.execute_lambda(lambda x: insert_record(x, table, data)) $ python load.py Initializing the driver Inserting into sample And done, I've got my JSON extracted MySQL 8 data in QLDB. I go to vett it in the client, and boy, didn't expect yet another package screw up. Clearly, these 2 AWS python packages are incompatible. That's a venv need, but I'm now at double my desired time to show this. $ qldbshell -l demo Traceback (most recent call last): File "/usr/local/bin/qldbshell", line 11, in load_entry_point('qldbshell==1.2.0', 'console_scripts', 'qldbshell')() File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 476, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2700, in load_entry_point return ep.load() File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2318, in load return self.resolve() File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/usr/local/lib/python3.6/site-packages/qldbshell/__main__.py", line 25, in from pyqldb.driver.pooled_qldb_driver import PooledQldbDriver ModuleNotFoundError: No module named 'pyqldb.driver.pooled_qldb_driver' [centos@ip-10-204-101-224] ~ $ pip list | grep qldbshell DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. qldbshell (1.2.0) $ sudo pip uninstall qldbshell pyqldb $ sudo pip install qldbshell WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. Collecting qldbshell Downloading https://artifactory.lifion.oneadp.com/artifactory/api/pypi/pypi/packages/packages/0f/f7/fe984d797e0882c5e141a4888709ae958eb8c48007a23e94000507439f83/qldbshell-1.2.0.tar.gz (68kB) 100% |████████████████████████████████| 71kB 43.4MB/s Requirement already satisfied: boto3>=1.9.237 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: amazon.ion=0.5.0 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: prompt_toolkit=3.0.5 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: ionhash~=1.1.0 in /usr/local/lib/python3.6/site-packages (from qldbshell) Requirement already satisfied: s3transfer=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: botocore=1.20.21 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: jmespath=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3>=1.9.237->qldbshell) Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from amazon.ion=0.5.0->qldbshell) Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/site-packages (from prompt_toolkit=3.0.5->qldbshell) Requirement already satisfied: python-dateutil=2.1 in /usr/local/lib/python3.6/site-packages (from botocore=1.20.21->boto3>=1.9.237->qldbshell) Requirement already satisfied: urllib3=1.25.4 in /usr/local/lib/python3.6/site-packages (from botocore=1.20.21->boto3>=1.9.237->qldbshell) Installing collected packages: qldbshell Running setup.py install for qldbshell ... done Successfully installed qldbshell-1.2.0 Can I see my data now $ qldbshell -l demo Welcome to the Amazon QLDB Shell version 1.2.0 Use 'start' to initiate and interact with a transaction. 'commit' and 'abort' to commit or abort a transaction. Use 'start; statement 1; statement 2; commit; start; statement 3; commit' to create transactions non-interactively. Use 'help' for the help section. All other commands will be interpreted as PartiQL statements until the 'exit' or 'quit' command is issued. qldbshell > select * from sample; INFO: { id: 1, name: "Demo Row", location: "USA", domain: null }, { id: 1, name: "Demo Row", location: "USA", domain: null }, { id: "1", name: "Demo Row", location: "USA", domain: "" }, { id: 3, name: "Kiwi", location: "NZ", domain: null }, { id: 2, name: "Row 2", location: "AUS", domain: "news.com.au" }, { id: 3, name: "Kiwi", location: "NZ", domain: null }, { id: 2, name: "Row 2", location: "AUS", domain: "news.com.au" } INFO: (0.0815s) And yes, data, I see it's duplicated, so I must have in between the 10 steps run twice. This does highlight a known limitation of QLDB, no unique constraints. But wait, that data is not really correct, I don't want null. Goes back to the JSON to see the MySQL shell gives that. $ jq . sample.json [ { "id": 1, "name": "Demo Row", "location": "USA", "domain": null }, ... At some point I also got this load error, but by now I've given up documenting how to do something, in order to demonstrate something. NameError: name 'null' is not defined One has to wrap the only nullable column with IFNULL(subdomain,'') as subdomain and redo all those steps again. This is not going to be practical having to wrap all columns in a wider table with IFNULL. However, having exhausted all this time for what was supposed to be a quiet weekend few hours, my post is way to long, and I've learned "Creating examples can be hard". http://ronaldbradford.com/blog/wdiltw-creating-examples-can-be-hard-2021-03-06/

[wperror500]

When you find 403 Forbidden WordPress Error, then do you what does it mean? It is similarly an HTTP status code that can occur when a user tries to access any web page or resource but doesn’t get the permission to view the page. so, get in touch with the satisfied experts. 

[lockingishard]

Something awesome update #03 - Learning powershell & Major work finished

Spent the past week exploring how idiot I am:

Summary: Now that I’ve finished the scripts for password stealing (Windows) using Mimikatz and file exfiltration (OSX), just a few more tests needed to be done to adjust the delay time to the system response speed.

The script for password stealing basically downloads Mimikatz from GitHub repository, runs the Mimikatz.exe corresponding to the 32/64-bit system, puts in some simple commands then outputs the activity log to designated folder, which is sent to the attacker’s burner-email. All evidences, including files downloaded and created, windows run history and command line history, are cleared.

ToDo: Video on the threats of rubber ducky; ways to prevent the above attacks and other attacks by ducky in general.

Major progress:

Had no experience with powershell at all, so did a bit research before I started to figure out the correct way to implement my idea. Some of the methods, such as invoke-WebRequest, did not work for me and I wasted a few hours figuring out that it’s probably due to different versions. So I searched up some methods that actually worked for my powershell 2.0 (E.g. System.Net.WebClient).

Spent a solid day on trying to figure out why I couldn’t download file using System.Net.WebClient. Error message read: “Exception calling"DownloadFile" with "2" argument(s):"An exception occurred during a webClient request.” Somehow in the first few hours I searched up “web client access denied” instead of “web client exception” which showed no useful information and got me stuck in a deadlock. Eventually I was reminded of my own stupidness by solution here - the second parameter should have the filename appended to the path as well.

Then, half a day of research was dedicated to “the remote server returned an error: (403) Forbidden”. The eureka moment was brought by this answer, which suggested that the problem might be “Scripts should use an informative User-Agent string with contact information, or they may be IP-blocked without notice.” So I went on to search up how to set up user-agent header in powershell, and this dude here saved my life.

To unzip the downloaded mimikatz: this solution here doesn’t really work for me, so after an hour of trial-and-error I simply got rid of the loop and it worked…computer is amazing.

Attempted to send email via smtp but error message popped up “server not specified” when trying SMTPInfo.send. This dude suggests a possibility of firewall rules blocking access to server, but requires testing with telnet. Research was done to figure out how to install telnet. Tested “telnet 101.198.198.198 587” resulted in “cannot open connection to server. Port 587: Connection failed”. Another half of day of research was dedicated to hook that connection up :( Some suggested to add inbound and outbound rule to allow port 587 to connect to ISP server, but sadly that did not work for me.

Changed the port from 587 to 25 solved the problem. Speechless.

Gmail automatically blocks email from suspicious senders, so in order to receive the attachment we have to enable less secure apps in Google Account of our burner email. Aw, Google you’re nice and ethical.

Researched on how to clear up Windows run history. This solution pointed me the right direction. Further research such as here, here and here allowed me to arrive at the right place.

[bestlinuxhosting-blog]

How to Fix the 403 Forbidden Error Check for URL errors and make sure you're specifying an actual web page file name and extension, not just a directory. Most websites are configured to disallow directory browsing, so a 403 Forbidden message when trying to display a folder instead of a specific page, is normal and expected. Check or reset/rename your .htaccess file- Locate File Manager from your cPanel In the public_html directory, look for .htaccess file. If you do not come across any .htaccess file, you can click on Settings and enabled Show Hidden Files (dotfiles) option. Reset file and folder permissions- This can be fixed by allowing permission of 755 to the all the directories which is most secure permission. In case of public_html permission would be 0750 which would be much secure. For more detailed steps with screenshots visit our Blog-http://blog.redserverhost.com/how-to-solve-403-forbidden-error/ We hope that the above post was easy to follow. For more web hosting tutorials, tricks and tips Follow this Page,Hit LIKE👍 & Don't Forget to SHARE Also Follow us on Twitter Page- https://twitter.com/redserverhost 

Facebook Page-https://facebook.com/redserverhost

[marksmithusa3]

How to Resolve "403 forbidden “error" in Bitstamp Account

Crypto currencies are digital assets which are premeditated to work as a source of exchange. Bitcoin is amongst the first decentralized and most famous crypto currency. Bitstamp is a foremost bitcoin exchange having its registered head office in Luxemberg. It was established in 2011. It supports its users to have trading between US dollars and bitcoin. Bitstamp is the pioneer of bitcoin exchange in the crypto market to encourage MultiSig technology for its hot-wallet that is a completely secured cold-storage in which roughly 98% customer bitcoin are kept off-line. In addition to this few more choices of security features are also available for Bitstamp’s customers for example two-factor authentication and confirmation emails to have improved layer of account security. Bitstamp Exchange provisions its users to have immediate orders facility at the best bid auction price or ask for price as well in totaling to place limit the placed orders, stop loss, and stop orders too. Bitstamp exchange has institutional as well as individual accounts options both from all around the world and users have choice to buy and sell bitcoin 24/7 through Bitstamp API. Bitstamp is manageable in form of mobile phone apps as well apart from desktop version. Users who are using android and ios device can trade bitcoin on the wing very merely.

But like any other exchange Bitstamp has much culpability too. Users have been in progress of reporting for “Error 403” with Bitstamp. Many customers are not capable to appraise the wallet because of Bitstamp error 403. They have been receiving info that: Could not update wallet: 403 errors. Even with their 12 word phase they could not ingress that.  Users’ couldn’t send their bitcoin from Bitstamp wallet and update was also unsuccessful. If you are facing such disorder of Bitstamp error 403, and need to fix it contact our Bitstamp Support Number 1-844-331-2333. Bitstamp error 403 is an imperative thoughtful concern need to be resolved very soon. Bitstamp wallet has been having problems with their servers. Cutting long story to short, millions of transactions have just gone misdirected from users across the world. If we go onto the Bitstamp assistance and support forum, there's thousands of new issues of people losing funds or them being in limbo in Bitstamp system just because of Bitstamp error 403. We have established our expertise in solving this 403 error issue and helping concerned members for the same. Users can communicate with us at our Bitstamp Phone Number 1-844-331-2333.

[ageloire]

HTTP Error 403 Forbidden Messages: What They Are & How to Fix Them

Imagine this -- you've recently created a new website to host your content, and you're excited to see it go live. You can't wait to dive into SEO and begin ranking for keywords, and attract an audience to your brand.

But then a friend emails you and says, "Hey, is there a reason I see this when I click on your website?"

Image courtesy of Wikipedia

Undoubtedly, a 403 Forbidden Message is cause for immediate concern -- how many potential viewers are you losing, as they come across your website just to find this message?

Fortunately, there are a few quick-and-easy solutions to a 403 error. Here, we'll explain the top three, so you can get your site up and running.

403 error

A 403 Forbidden error is an HTTP status code sent to users by an HTTP server when a user is trying to access a URL. It means the page you were trying to reach is forbidden for one of two reasons -- either there is a removal or restriction of access permissions from the client-side, or there's an accidental misconfiguration of the web server.

How to Fix 403 Errors

1. Permission or ownership errors.

Your site might be showing a 403 Forbidden error because of permission issues. If you've configured your web server, you'll want to add the server to the www-data group, and set ownership of /var/www to the www-data user and www-data group.

You'll then want to make sure your directories are set to 755, your files are set to 644, and your dynamic content is set to 700.

2. Make sure you have an index page.

Your website's home page must be called index.html or index.php -- if it's not, you should rename the homepage to include one of those URL names.

Alternatively, you can upload an index page to your httpdocs directory, and then set up a redirect on the index page to your real homepage.

3. You haven't uploaded your website content to the correct directory on your server.

Lastly, you might see a 403 forbidden message if you haven't correctly uploaded your content to the directory on your server.

There are several different FTP clients you might've chosen to host your domain -- let's say you chose FileZilla, which is free and available for Windows, Mac, and Linux.

To publish your content online, you'll need to put your files into the Public-htdocs directory (if you're using FileZilla -- these instructions will vary if you use a different FTP client). Once you've dragged-and-dropped your files into the directory, you should search your website's URL to double-check they now appear online.

If you're in your FTP server and don't see the httpdocs directory, you can create a file within the directory with this title, which could also solve the issue.

from Marketing https://blog.hubspot.com/marketing/http-403-forbidden

[mojomediapro]

Common WordPress Errors

WordPress offers you plenty of benefits when you use it, however, it produces errors that limit its wonderful experience.

Are you worried about the errors displaying on your site?

If so, you should know that the error on your website has probably been reported and resolved by others.

WordPress produces a lot of errors and the list below contains the common WordPress errors.

So, before you proceed, to knowing the common WordPress errors and their fixes, backup your site.

  Internal Server Error

This type of error displays on your site to indicate something wrong.

Sometimes it appears as ‘500 internal Server Error’, so, whichever the case, it means the same thing.

When this error appears, it means that there is something wrong and the server finds it difficult to point its origin.

Due to that, you are responsible for figuring out how to go about it.

Therefore, you can try the following solutions, one at a time.

Check if your .htaccess file is corrupt

Try to Increase the PHP Memory Limit

Deactivate all the plugins installed on your site.

Freshly upload your site’s core files

Last but not least, contact your hosting provider

  The Site is experiencing technical difficulties.

This type of error comes as a result of any error in our list.

WordPress developers put in place a fatal error protection feature as from WordPress version 5.2.

So, WordPress now sends an email notification to your admin dashboard using your admin email address.

The email comes with a link which aids in accessing the backend so that you can attempt and fix the error.

However, the process of fixing becomes harder when you don’t have access to your admin email or not able to receive WordPress emails.

  Syntax Error

This kind of error occurs as a result of added code snippets into WordPress, an accidentally missed something needed or even an incorrect syntax.

Lucky enough, the error indicates the code or problem and the location as well.

Due to that, you can correct the syntax, don’t be shocked because, mostly, a missing bracket can trigger this error.

Sometimes an unexpected character may cause this error, therefore, edit the code and use the right characters.

  Error establishing a database connection

When you encounter such an error, you should know that your website is unable to link with the database.

However, the process of solving such an error seems tricky for beginners.

What causes this error, right?

Normally, this error occurs when you enter or modify your database login credentials.

Sometimes, an unresponsive database may trigger this error.

In addition to that, a corrupt database can trigger the error.

So, use the right credentials to access your database or repair your WordPress database.

You can also check with if the database server is down or update WordPress site URL.

Lastly, reboot the webserver or even contact your hosting company.

  404 page not found error

When this error occurs, users still browse on the other sections such as the admin area.

However, they cannot browse through a single post.

Mostly, the permalink settings in WordPress causes this kind of error.

Therefore, to solve this kind of problem you need to reconfigure the permalink settings.

Additionally, you can manually update the rewrite rules, for instance, updating your .htaccess file.

  White Screen of death

This error may affect one or certain sections of the site.

This type of error displays no error message hence making it very hard to know its origin.

Sometimes, incorrect server configurations may trigger this error.

However, in most cases, it happens due to a script exhausting the PHP memory limit.

Therefore, solving this error needs you to extend or increase the PHP memory limit and check for server configurations.

  WordPress Memory Exhausted Error

WordPress core actually uses PHP as the server-side programming language.

Just like your computer needs memory to function properly, the same applies to webservers.

Therefore, this site also needs memory just like any other site.

So, when you encounter this error, it means that your WordPress code needs more memory than the allocated default value.

How do you fix this type of error, right?

Simply, open, edit and define the memory limit in your ‘wp-config.php’ file located under your WordPress site’s root folder.

  ‘’Are you sure you want to do this?’’ error

This kind of error usually occurs in your admin area, however, that should not worry you.

Mostly, this error arises due to failures in the proper use of Nonce.

In other words, the error occurs if your plugins or themes fail to properly use Nonce (Special security keys).

Therefore, you need to investigate your themes or plugins to fix the error.

  Briefly Unavailable for Scheduled Maintenance in WordPress

This error shows on your screen as a result of unfinished or interrupted WordPress Update.

When this error occurs, it entirely locks your site hence making the site unavailable for both admins and visitors.

Since this error comes as a result of slow server response or low memory issues, upgrading to higher hosting plans can be a great fix.

However, you can delete the ‘’. maintenance’’ file in your site’s root directory using FTP.

Sometimes, you may not find the ‘’. maintenance’’ file, therefore, always check for hidden files.

  WordPress RSS Feed Errors

RSS Feeds enable your site’s visitors to subscribe to your blog.

Mostly, this kind of error occurs due to poor formatting and the error message may vary depending on your Browser.

That said, even a blank space in your closing PHP tag can trigger the error.

In other words, you can solve this error by writing correct PHP code or check your plugins or themes ‘’functions.php’’ file.

Also, you can disable RSS Feed related plugins and if that doesn’t work as well, temporarily switch to the default theme.

  403 Forbidden error

This type of error normally occurs due to lack of permissions granted by your server.

When your server restricts your access, you don’t get access to specific pages.

However, you may encounter such an error due to other factors.

For instance, poor coded security plugins, server configurations and incorrect file permissions can trigger this error.

How do you go about it, right?

Actually, you need to disable your plugins, repair corrupted ‘’.htaccess’’ file or even contact your hosting company for correct file permissions.

‘’The page isn’t redirecting properly’’ error

This type of error normally occurs as a result of misconfigured redirections problems.

Since WordPress features SEO friendly URL structures for redirect functions, other WordPress plugins follow the trend.

Therefore, any misconfigured redirect tool causes this type of error.

In other words, your site may redirect users to a URL which redirects them back to the original URL.

So, what does that mean?

Actually, when that happens, your visitors’ browser gets trapped between two webpages hence resulting into a redirect loop.

To resolve this issue, simply change your WordPress Address and website address.

For instance, add or remove the www prefix on your addresses.

In other words, if your addresses contain the www prefix, remove it, if not, add the prefix, then save changes.

Also, you can define the changes in ‘’wp-config.php’’ file or even try deactivating all plugins and install fresh copies as you counter check for the error.

  502 Bad Gateway error

This kind of error usually appears when a user sends the server a request but it takes lots of time to get the feedback.

Also, the error occurs due to poor server misconfigurations and poorly coded themes or plugins.

However, glitches resulting from high traffic trigger this error.

Therefore, to fix this problem, you need to reload your browser, and if it doesn’t work, clear your browser’s cache.

Try to disable the Firewalls or CDN, update your site’s themes and plugins, or contact your hosting provider.

Conclusion

WordPress produces many errors; however, the list above contains some of the most common errors.

Most of the errors described above occur due to similar causes, however, they slightly vary.

Therefore, you need to actually know what type of error appears on your end for proper fixes.

  from MojoMedia.Pro https://www.mojomedia.pro/common-wordpress-errors/

[errorcodeexpert-blog]

How To Fix Common HTTP Error Codes 500

Introduction

When accessing a web server or application, every HTTP request that is received by a server is responded to with an HTTP status code. HTTP status codes are three-digit codes, and are grouped into five different classes. The class of a status code can be quickly identified by its first digit:

1xx: Informational

2xx: Success

3xx: Redirection

4xx: Client Error

5xx: Server Error

This guide focuses on identifying and troubleshooting the most commonly encountered HTTP error code 500, i.e. 4xx and 5xx status codes, from a system administrator's perspective. There are many situations that could cause a web server to respond to a request with a particular error code--we will cover common potential causes and solutions.

Client And Server Error Overview

Client errors, or HTTP status codes from 400 to 499, are the result of HTTP requests sent by a user client (i.e. a web browser or other HTTP client). Even though these types of errors are client-related, it is often useful to know which error code a user is encountering to determine if the potential issue can be fixed by server configuration.

Server errors, or HTTP status codes from 500 to 599, are returned by a web server when it is aware that an error has occurred or is otherwise not able to process the request.

General Troubleshooting Tips

When using a web browser to test a web server, refresh the browser after making server changes

Check server logs for more details about how the server is handling the requests. For example, web servers such as Apache or Nginx produce two files called access.log and error.log that can be scanned for relevant information

Keep in mind that HTTP status code definitions are part of a standard that is implemented by the application that is serving requests. This means that the actual status code that is returned depends on how the server software handles a particular error--this guide should generally point you in the right direction

Now that you have a high-level understanding of HTTP status codes, we will look at the commonly encountered errors.

400 Bad Request

The 400 status code, or Bad Request error, means the HTTP request that was sent to the server has invalid syntax.

Here are a few examples of when a 400 Bad Request error might occur:

The user's cookie that is associated with the site is corrupt. Clearing the browser's cache and cookies could solve this issue

Malformed request due to a faulty browser

Malformed request due to human error when manually forming HTTP requests (e.g. using curlincorrectly)

401 Unauthorized

The 401 status code, or an Unauthorized error, means that the user trying to access the resource has not been authenticated or has not been authenticated correctly. This means that the user must provide credentials to be able to view the protected resource.

An example scenario where a 401 Unauthorized error would be returned is if a user tries to access a resource that is protected by HTTP authentication, as in this Nginx tutorial. In this case, the user will receive a 401 response code until they provide a valid username and password (one that exists in the .htpasswd file) to the web server.

403 Forbidden

The 403 status code, or a Forbidden error, means that the user made a valid request but the server is refusing to serve the request, due to a lack of permission to access the requested resource. If you are encountering a 403 error unexpectedly, there are a few typical causes that are explained here.

File Permissions

403 errors commonly occur when the user that is running the web server process does not have sufficient permissions to read the file that is being accessed.

To give an example of troubleshooting a 403 error, assume the following situation:

The user is trying to access the web server's index file, from http://example.com/index.html

The web server worker process is owned by the www-data user

On the server, the index file is located at /usr/share/nginx/html/index.html

If the user is getting a 403 Forbidden error, ensure that the www-data user has sufficient permissions to read the file. Typically, this means that the other permissions of the file should be set to read. There are several ways to ensure this, but the following command will work in this case:

sudo chmod o=r /usr/share/nginx/html/index.html

.Htaccess

Another potential cause of 403 errors, often intentinally, is the use of an .htaccess file. The .htaccessfile can be used to deny access of certain resources to specific IP addresses or ranges, for example.

If the user is unexpectedly getting a 403 Forbidden error, ensure that it is not being caused by your .htaccess settings.

Index File Does Not Exist

If the user is trying to access a directory that does not have a default index file, and directory listings are not enabled, the web server will return a 403 Forbidden error. For example, if the user is trying to access http://example.com/emptydir/, and there is no index file in the emptydir directory on the server, a 403 status will be returned.

If you want directory listings to be enabled, you may do so in your web server configuration.

404 Not Found

The 404 status code, or a Not Found error, means that the user is able to communicate with the server but it is unable to locate the requested file or resource.

404 errors can occur in a large variety of situations. If the user is unexpectedly receiving a 404 Not Found error, here are some questions to ask while troubleshooting:

Does the link that directed the user to your server resource have a typographical error in it?

Did the user type in the wrong URL?

Does the file exist in the correct location on the server? Was the resource was moved or deleted on the server?

Does the server configuration have the correct document root location?

Does the user that owns the web server worker process have privileges to traverse to the directory that the requested file is in? (Hint: directories require read and execute permissions to be accessed)

Is the resource being accessed a symbolic link? If so, ensure the web server is configured to follow symbolic links

500 Internal Server Error

The 500 status code, or Internal Server Error, means that server cannot process the request for an unknown reason. Sometimes this code will appear when more specific 5xx errors are more appropriate.

This most common cause for this error is server misconfiguration (e.g. a malformed .htaccess file) or missing packages (e.g. trying to execute a PHP file without PHP installed properly).

502 Bad Gateway

The 502 status code, or Bad Gateway error, means that the server is a gateway or proxy server, and it is not receiving a valid response from the backend servers that should actually fulfill the request.

If the server in question is a reverse proxy server, such as a load balancer, here are a few things to check:

The backend servers (where the HTTP requests are being forwarded to) are healthy

The reverse proxy is configured properly, with the proper backends specified

The network connection between the backend servers and reverse proxy server is healthy. If the servers can communicate on other ports, make sure that the firewall is allowing the traffic between them

If your web application is configured to listen on a socket, ensure that the socket exists in the correct location and that it has the proper permissions

503 Service Unavailable

The 503 status code, or Service Unavailable error, means that the server is overloaded or under maintenance. This error implies that the service should become available at some point.

If the server is not under maintenance, this can indicate that the server does not have enough CPU or memory resources to handle all of the incoming requests, or that the web server needs to be configured to allow more users, threads, or processes.

504 Gateway Timeout

The 504 status code, or Gateway Timeout error, means that the server is a gateway or proxy server, and it is not receiving a response from the backend servers within the allowed time period.

This typically occurs in the following situations:

The network connection between the servers is poor

The backend server that is fulfilling the request is too slow, due to poor performance

The gateway or proxy server's timeout duration is too short

Conclusion

Now that you are familiar with the most common HTTP error codes, and common solutions to those codes, you should have a good basis for troubleshooting issues with your web servers or applications.

If you encounter any error codes that were not mentioned in this guide, or if you know of other likely solutions to the ones that were described, feel free to discuss them in the comments!

Call On Our Technical Support Phone Number for fix HTTP Error Codes 500 by Errorcode Expert USA +1 (800) 848-5295 Right Now to Get Instant Help

[lindyhunt]

HTTP Error 403 Forbidden Messages: What They Are & How to Fix Them

Imagine this -- you've recently created a new website to host your content, and you're excited to see it go live. You can't wait to dive into SEO and begin ranking for keywords, and attract an audience to your brand.

But then a friend emails you and says, "Hey, is there a reason I see this when I click on your website?"

Image courtesy of Wikipedia

Undoubtedly, a 403 Forbidden Message is cause for immediate concern -- how many potential viewers are you losing, as they come across your website just to find this message?

Fortunately, there are a few quick-and-easy solutions to a 403 error. Here, we'll explain the top three, so you can get your site up and running.

403 error

A 403 Forbidden error is an HTTP status code sent to users by an HTTP server when a user is trying to access a URL. It means the page you were trying to reach is forbidden for one of two reasons -- either there is a removal or restriction of access permissions from the client-side, or there's an accidental misconfiguration of the web server.

How to Fix 403 Errors

1. Permission or ownership errors.

Your site might be showing a 403 Forbidden error because of permission issues. If you've configured your web server, you'll want to add the server to the www-data group, and set ownership of /var/www to the www-data user and www-data group.

You'll then want to make sure your directories are set to 755, your files are set to 644, and your dynamic content is set to 700.

2. Make sure you have an index page.

Your website's home page must be called index.html or index.php -- if it's not, you should rename the homepage to include one of those URL names.

Alternatively, you can upload an index page to your httpdocs directory, and then set up a redirect on the index page to your real homepage.

3. You haven't uploaded your website content to the correct directory on your server.

Lastly, you might see a 403 forbidden message if you haven't correctly uploaded your content to the directory on your server.

There are several different FTP clients you might've chosen to host your domain -- let's say you chose FileZilla, which is free and available for Windows, Mac, and Linux.

To publish your content online, you'll need to put your files into the Public-htdocs directory (if you're using FileZilla -- these instructions will vary if you use a different FTP client). Once you've dragged-and-dropped your files into the directory, you should search your website's URL to double-check they now appear online.

If you're in your FTP server and don't see the httpdocs directory, you can create a file within the directory with this title, which could also solve the issue.