#UserAgentFiltering

[attributionlabs-blog]

Choosing the Right Proxy for the Job

Today we are announcing updates to our proxy lists that remedy these common proxy pain points:

"Google blocks this proxy"

"This proxy does not work in my browser"

Proxies are fickle beasts, let me count the ways: latency and availability can fluctuate from one request to another, your requests may be logged, you may be exposed to your destination, and so on.

We do extensive testing on proxies at TheProxIsRight and expose to you everything we learn about them. Per the above list, we provide last latency, up-time over last month, type (socks4,socks5, https, http), anonymity, location, etc. But that is all old news.

New News!

We just added two new tests, "Browser User-Agent Filtering" and "Google Access" and are exposing those you via all the ways you consume our proxies lists: download, email, API, and web.

New Tests Explained

"Filtering by User-agent"

Some proxies will evaluate the user-agent of the request and may block or redirect the request if certain criteria is met (often a "Go Away" message of some variety is seen). We determined filtering by web browser user-agents was common enough, and now test proxies for such filtering. (Examples below)

"Blocking by Destination Host"

As Google is a common host that users of proxies hit, it would be valuable to know in advance if a desired proxy is blocked. A common reason for Google to temporarily or permanently block a proxy is that said proxy could abusively query Google, going over whatever rate-limit Google deems normally acceptable.

Summary Testing Insight

If we find a proxy to be active via our normal REST (for HTTP/HTTPS) or SOCKS (4+5) tests, we now test two more times, once with a "browser user-agent" to our known truth destination host, and then one more time with Google as the destination, issuing a query w/ a predictable result. We then expose these as "supports_browser" and "supports_google".

It is not an exact science, as proxies can change their behavior, or just be overloaded or offline when you seek to use them, but these additional properties will help you in determining which proxies best suit your specific needs.

Proxy On Wayne!