#Crc32 checksum calculator | Explore Tumblr Posts and Blogs

loadprofit420 · 2 years

Text

Download File Transfer For Mac 3.2

TeraCopy for Windows / Mac

Copy your files faster and more securely

Download Latest Version for Mac (17.55 MB) Advertisement. DoubleTwist is a simple and intuitive app that allows you to copy and share files on your mobile, gaming, photo and audio devices with your Mac. DoubleTwist offers a universal interface to manage photos, music and videos for most popular phones, gadgets and devices and we continuously.

Click the Download button on this page to start the download. Do one of the following: To start the installation immediately, click Open or Run this program from its current location. To copy the download to your computer for installation at a later time, click Save or Save this program to disk.

Download FileZilla Client All platforms. Download FileZilla Server Windows only. Pick the client if you want to transfer files. Get the server if you want to make files available for others. News 2021-10-29 - FileZilla Server 1.1.0 released Fixed vulnerabilities.

Free Download and Install AirMore: File Transfer For PC. We offer to install and using the AirMore: File Transfer app on a Windows 10/8/7 PC. Also, for MacOS.

DownloadBuy Suggest an ideaDevelopment status

Be in control

In case of a copy error TeraCopy will try to recover. If unable to recover in the worst case scenario it will simply skip the problematic file, not terminating the entire transfer. It can also wait for network or device reconnection.

Unnecessary files can be skipped to reduce the transfer time. During the process, TeraCopy shows detected errors, and allows you to fix the problem by recopying only the problematic files.

File verification

TeraCopy can verify files after they have been copied to ensure that they are identical. This is done by comparing hashes of source and target files.

Supported algorithms: CRC32, MD5, SHA-1, SHA-256, SHA-512, Panama, Tiger, RipeMD, Whirlpool and xxHash.

Confirm drag'n'drop

TeraCopy can optionally show a confirmation dialog on every drag and drop operation. It can prevent you from moving folders accidentally from one location to another.

Faster workflow

TeraCopy can keep history of recently used directories, as well as show folders which are open in Explorer and other file managers.

Other Features

Shell integration

TeraCopy can completely replace the Explorer copy and move functions, enabling you to work with files as usual.

Export reports Pro

Save file lists with all related information as HTML and CSV files.

Edit file lists Pro

Unnecessary files and folders can be removed to reduce the transfer time.

Preserve date timestamps

TeraCopy keeps the original time and date of files when copying.

Trusted by forensic experts

TeraCopy is included in DEFT (Digital Evidence & Forensics Toolkit).

Download File Transfer For Mac 3.2 Full

Copy locked files

Copy files using an elevated Windows Service and Volume Shadow Copy if required.

Videos

New folder with selected itemsQuickly verify downloaded filesDelete files securely

Download File Transfer For Mac 3.2 Windows 7

Testimonials & Reviews

Your app did in 30 hours what has been taking me weeks and weeks to do piecemeal at a time. The amount of time I wasted in aborted processes and babysitting file copies easily cost me thousands of dollars in lost productivity and delays in work delivery.

Richard Hoefer

TeraCopy is a nifty piece of freeware that improves the Windows copy/move function and adds useful extras like checksum calculation and permanent delete.

CNET

Feature Matrix

File Transfer Software Free

FeatureTeraCopyTeraCopy ProVerify files after copycheckcheckPreserve date timestampscheckcheckCopy locked filescheckcheckIntegrate with Windows ExplorercheckcheckRun a shell script on transfer completioncheckcheckGenerate and verify checksum filescheckcheckDelete files securelycheckcheck Use in a commercial environmentcheckEdit file listscheckManage favorite folderscheckExport reports in HTML and CSV formatscheck

0 notes

munrad · 3 years

Text

HashMyFiles Working 100% Key & {Version} {Torrent} Cracked

HashMyFiles 2.42 Torrent Cracked Calculate MD5 SHA1 hash View hash Hash SHA1 MD5 Checksum. In addition, you can save selected items to an external file (TXT, HTML, XML, CSV) or move them to the Recycle Bin, view the aforementioned file properties in a window, use a search function and copy the MD5, SHA1 or CRC32 data. HashMyFiles 2.42 2019 Download is a lightweight application which enables you…

View On WordPress

0 notes

trishtech · 3 years

Text

CyoHash : File Explorer Extension for Calculating File Hashes

CyoHash is a shell extension for Windows File Explorer using which we can easily calculate file hashes. It supports many hashing algorithms such as MD5, SHA-1, SHA-256, SHA-384, SHA-512 and also CRC32 checksums. This small application makes it very convenient to calculate file hashes under any Windows PC. After the installation of CyoHash on your Windows PC, it will add a new item in the…

View On WordPress

#Hash

0 notes

midmaxx-blog · 5 years

Photo

#Hashing is an algorithm that calculates a fixed-size bit string value from a file. A file basically contains blocks of data. Hashing transforms this data into a far shorter fixed-length value or key which represents the original string. The hash value can be considered the distilled summary of everything within that file. A good hashing #algorithm would exhibit a property called the avalanche effect, where the resulting hash output would change significantly or entirely even when a single bit or byte of data within a file is changed. A hash function that does not do this is considered to have poor randomization, which would be easy to break by hackers. A hash is usually a hexadecimal string of several characters. Hashing is also a unidirectional process so you can never work backwards to get back the original data. #Types Of Hashing: MD5 - Used as a checksum to verify data integrity. SHA 2 - A cryptographic hash function. CRC32 - cyclic redundancy check is an error-detecting code often used for detection of accidental changes to data. Credits : @yashpatil_ Join @sec_army for Daily Hacktivity & Knowledge Dose. #cybersecurityawareness #hash #cybersecuritytraining #ethicalhacking #hacking #hackingtools #learnhacking #kalilinux #kalilinuxtools #cyberhunter #cyberattack #cybersecurity #cyberarmy #instacybersecurity #wearesecarmy https://www.instagram.com/p/B3w-Vhjg1QJ/?igshid=1ioxk12xcn3ky

#hashing #algorithm #types #cybersecurityawareness #hash #cybersecuritytraining #ethicalhacking #hacking #hackingtools #learnhacking #kalilinux #kalilinuxtools #cyberhunter #cyberattack #cybersecurity #cyberarmy #instacybersecurity #wearesecarmy

0 notes

softlookup · 5 years

Text

Ryll Checksum Checker 2.0

Ryll Checksum Checker : Calculate the CRC32 checksum of any file fast and easy using this intuitive app that shows other file properties too, such as type and size source https://www.softlookup.com/display.asp?id=303027

0 notes

terabitweb · 5 years

Text

Original Post from FireEye Author: Nart Villeneuve

We observed several high-volume FormBook malware distribution campaigns primarily taking aim at Aerospace, Defense Contractor, and Manufacturing sectors within the U.S. and South Korea during the past few months. The attackers involved in these email campaigns leveraged a variety of distribution mechanisms to deliver the information stealing FormBook malware, including:

PDFs with download links

DOC and XLS files with malicious macros

Archive files (ZIP, RAR, ACE, and ISOs) containing EXE payloads

The PDF and DOC/XLS campaigns primarily impacted the United States and the Archive campaigns largely impacted the Unites States and South Korea.

FormBook Overview

FormBook is a data stealer and form grabber that has been advertised in various hacking forums since early 2016. Figure 1 and Figure 2 show the online advertisement for the malware.

Figure 1: FormBook advertisement

Figure 2: FormBook underground pricing

The malware injects itself into various processes and installs function hooks to log keystrokes, steal clipboard contents, and extract data from HTTP sessions. The malware can also execute commands from a command and control (C2) server. The commands include instructing the malware to download and execute files, start processes, shutdown and reboot the system, and steal cookies and local passwords.

One of the malware’s most interesting features is that it reads Windows’ ntdll.dll module from disk into memory, and calls its exported functions directly, rendering user-mode hooking and API monitoring mechanisms ineffective. The malware author calls this technique “Lagos Island method” (allegedly originating from a userland rootkit with this name).

It also features a persistence method that randomly changes the path, filename, file extension, and the registry key used for persistence.

The malware author does not sell the builder, but only sells the panel, and then generates the executable files as a service.

Capabilities

FormBook is a data stealer, but not a full-fledged banker (banking malware). It does not currently have any extensions or plug-ins. Its capabilities include:

Key logging 

Clipboard monitoring 

Grabbing HTTP/HTTPS/SPDY/HTTP2 forms and network requests

Grabbing passwords from browsers and email clients

Screenshots

FormBook can receive the following remote commands from the C2 server:

Update bot on host system

Download and execute file

Remove bot from host system

Launch a command via ShellExecute

Clear browser cookies

Reboot system

Shutdown system

Collect passwords and create a screenshot

Download and unpack ZIP archive

Infrastructure

The C2 domains typically leverage less widespread, newer generic top-level domains (gTLDs) such as .site, .website, .tech, .online, and .info.

The C2 domains used for this recently observed FormBook activity have been registered using the WhoisGuard privacy protection service. The server infrastructure is hosted on BlazingFast.io, a Ukrainian hosting provider. Each server typically has multiple FormBook panel installation locations, which could be indicative of an affiliate model.

Behavior Details

File Characteristics

Our analysis in this blog post is based on the following representative sample:

Filename

MD5 Hash

Size (bytes)

Compile Time

Unavailable

CE84640C3228925CC4815116DDE968CB

747,652

2012-06-09 13:19:49Z

Table 1: FormBook sample details

Packer

The malware is a self-extracting RAR file that starts an AutoIt loader. The AutoIt loader compiles and runs an AutoIt script. The script decrypts the FormBook payload file, loads it into memory, and then executes it.

Installation

The FormBook malware copies itself to a new location. The malware first chooses one of the following strings to use as a prefix for its installed filename:

ms, win, gdi, mfc, vga, igfx, user, help, config, update, regsvc, chkdsk, systray, audiodg, certmgr, autochk, taskhost, colorcpl, services, IconCache, ThumbCache, Cookies

It then generates two to five random characters and appends those to the chosen string above

followed by one of the following file extensions:

.exe, .com, .scr, .pif, .cmd, .bat

If the malware is running with elevated privileges, it copies itself to one of the following directories:

%ProgramFiles%

%CommonProgramFiles%

If running with normal privileges, it copies itself to one of the following directories:

%USERPROFILE%

%APPDATA%

%TEMP%

Persistence

The malware uses the same aforementioned string list with a random string to create a prefix, appends one to five random characters, and uses this value as the registry value name.

The malware configures persistence to one of the following two locations depending on its privileges:

(HKCU|HKLM)SOFTWAREMicrosoftWindowsCurrentVersionRun

(HKCU|HKLM)SOFTWAREMicrosoftWindowsCurrentVersionPoliciesExplorerRun

Startup

The malware creates two 16-byte mutexes. The first mutex is the client identifier (e.g., 8-3503835SZBFHHZ). The second mutex value is derived from the C2 information and the username (e.g., LL9PSC56RW7Bx3A5).

The malware then iterates over a process listing and calculates a checksum value of process names (rather than checking the name itself) to figure out which process to inject. The malware may inject itself into browser processes and explorer.exe. Depending on the target process, the malware installs different function hooks (see the Function Hooks section for further detail).

Anti-Analysis

The malware uses several techniques to complicate malware analysis:

Timing checks using the RDTSC instruction

Calls NtQueryInformationProcess with InfoClass=7 (ProcessDebugPort)

Sample path and filename checks (sample filename must be shorter than 32 characters)

Hash-based module blacklist

Hash-based process blacklist

Hash-based username blacklist

Before communicating, it checks whether the C2 server is present in the hosts file

The results of these tests are then placed into a 16-byte array, and a SHA1 hash is calculated on the array, which will be later used as the decryption key for subsequent strings (e.g. DLL names to load). Failed checks may go unnoticed until the sample tries to load the supporting DLLs (kernel32.dll and advapi32.dll).

The correct 16-byte array holding the result of the checks is:

00 00 01 01 00 00 01 00 01 00 01 00 00 00 00 00

Having a SHA1 value of:

5b85aaa14f74e7e8adb93b040b0914a10b8b19b2

After completing all anti-analysis checks, the sample manually maps ntdll.dll from disk into memory and uses its exported functions directly in the code. All API functions will have a small stub function in the code that looks up the address of the API in the mapped ntdll.dll using the CRC32 checksum of the API name, and sets up the parameters on the stack.

This will be followed by a direct register call to the mapped ntdll.dll module. This makes regular debugger breakpoints on APIs inoperable, as execution will never go through the system mapped ntdll.dll.

Process Injection

The sample loops through all the running processes to find explorer.exe by the CRC32 checksum of its process name. It then injects into explorer.exe using the following API calls (avoiding more commonly identifiable techniques such as WriteProcessMemory and CreateRemoteThread):

NtMapViewOfSection

NtSetContextThread

NtQueueUserAPC 

The injected code in the hijacked instance of explorer.exe randomly selects and launches (as a suspended process) a built-in Windows executable from the following list:

svchost.exe, msiexec.exe, wuauclt.exe, lsass.exe, wlanext.exe, msg.exe, lsm.exe, dwm.exe, help.exe, chkdsk.exe, cmmon32.exe, nbtstat.exe, spoolsv.exe, rdpclip.exe, control.exe, taskhost.exe, rundll32.exe, systray.exe, audiodg.exe, wininit.exe, services.exe, autochk.exe, autoconv.exe, autofmt.exe, cmstp.exe, colorcpl.exe, cscript.exe, explorer.exe, WWAHost.exe, ipconfig.exe, msdt.exe, mstsc.exe, NAPSTAT.EXE, netsh.exe, NETSTAT.EXE, raserver.exe, wscript.exe, wuapp.exe, cmd.exe

The original process reads the randomly selected executable from the memory of explorer.exe and migrates into this new process via NtMapViewOfSection, NtSetContextThread, and NtQueueUserAPC.  

The new process then deletes the original sample and sets up persistence (see the Persistence section for more detail). It then goes into a loop that constantly enumerates running processes and looks for targets based on the CRC32 checksum of the process name.

Targeted process names include, but are not limited to:  

iexplore.exe, firefox.exe, chrome.exe, MicrosoftEdgeCP.exe, explorer.exe, opera.exe, safari.exe, torch.exe, maxthon.exe, seamonkey.exe, avant.exe, deepnet.exe, k-meleon.exe, citrio.exe, coolnovo.exe, coowon.exe, cyberfox.exe, dooble.exe, vivaldi.exe, iridium.exe, epic.exe, midori.exe, mustang.exe, orbitum.exe, palemoon.exe, qupzilla.exe, sleipnir.exe, superbird.exe, outlook.exe, thunderbird.exe, totalcmd.exe

After injecting into any of the target processes, it sets up user-mode API hooks based on the process.

The malware installs different function hooks depending on the process. The primary purpose of these function hooks is to log keystrokes, steal clipboard data, and extract authentication information from browser HTTP sessions. The malware stores data in local password log files. The directory name is derived from the C2 information and the username (the same as the second mutex created above: LL9PSC56RW7Bx3A5).

However, only eight bytes from this value are used as the directory name (e.g., LL9PSC56). Next, the first three characters from the derived directory name are used as a prefix for the log file followed by the string log. Following this prefix are names corresponding to the type of log file. For example, for Internet Explorer passwords, the following log file would be created:

%APPDATA%LL9PSC56LL9logri.ini.

The following are the password log filenames without the prefix:

(no name): Keylog data

rg.ini: Chrome passwords

rf.ini: Firefox passwords

rt.ini: Thunderbird passwords

ri.ini: Internet Explorer passwords

rc.ini: Outlook passwords

rv.ini: Windows Vault passwords

ro.ini: Opera passwords

One additional file that does not use the .INI file extension is a screenshot file:

im.jpeg

Function Hooks

Keylog/clipboard monitoring:

GetMessageA

GetMessageW

PeekMessageA

PeekMessageW

SendMessageA

SendMessageW

Browser hooks:

PR_Write

HttpSendRequestA

HttpSendRequestW

InternetQueryOptionW

EncryptMessage

WSASend

The browser hooks look for certain strings in the content of HTTP requests and, if a match is found, information about the request is extracted. The targeted strings are:

pass

token

signin

account

persistent

Network Communications

The malware communicates with the following C2 server using HTTP requests:

www[.]clicks-track[.]info/list/hx28/

Beacon

As seen in Figure 3, FormBook sends a beacon request (controlled by a timer/counter) using HTTP GET with an “id” parameter in the URL.

Figure 3: FormBook beacon

The decoded “id” parameter is as follows:

FBNG:134C0ABB 2.9:Windows 7 Professional x86:VXNlcg==

Where:

“FBNG” – magic bytes 

“134C0ABB” – the CRC32 checksum of the user’s SID

“2.9” – the bot version 

“Windows 7 Professional” – operating system version

“x86” – operating system architecture 

“VXNlcg==” – the Base64 encoded username (i.e., “User” in this case)

Communication Encryption

The malware sends HTTP requests using hard-coded HTTP header values. The HTTP headers shown in Figure 4 are hardcoded.

Figure 4: Hard-coded HTTP header values

Messages to the C2 server are sent RC4 encrypted and Base64 encoded. The malware uses a slightly altered Base64 alphabet, and also uses the character “.” instead of “=” as the pad character:

Standard Alphabet:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

Modified Alphabet:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_

The RC4 key is created using an implementation of the SHA1 hashing algorithm with the C2 URL. The standard SHA1 algorithm reverses the DWORD endianness at the end of the algorithm. This implementation does not, which results in a reverse endian DWORDs. For example, the SHA1 hash for the aforementioned URL is “9b198a3cfa6ff461cc40b754c90740a81559b9ae,” but when reordering the DWORDs, it produces the correct RC4 key: 3c8a199b61f46ffa54b740cca84007c9aeb95915. The first DWORD “9b198a3c” becomes “3c8a199b.”

Figure 5 shows an example HTTP POST request.

Figure 5: Example HTTP POST request

In this example, the decoded result is:

ClipboardrnrnBlank Page – Windows Internet ExplorerrnrncEXN{3wutV,

Accepted Commands

When a command is sent by the C2 server, the HTTP response body has the format shown in Figure 6.

Figure 6: FormBook C2 server response with command

The data begins with the magic bytes “FBNG,” and a one-byte command code from hex bytes 31 to 39 (i.e., from “1” to “9”) in clear text. This is then followed by the RC4-encoded command data (where the RC4 key is the same as the one used for the request). In the decrypted data, another occurrence of the magic FBNG bytes indicates the end of the command data.

The malware accepts the commands shown in Table 2.

Command

Parameters (after decryption)

Purpose

‘1’ (0x31)

pe_file_data>FBNG

Download and execute file from %TEMP% directory

‘2’ (0x32)

pe_file_data>FBNG

Update bot on host machine

‘3’ (0x33)

FBNG

Remove bot from host machine

‘4’ (0x34)

command_string>FBNG

Launch a command via ShellExecute

‘5’ (0x35)

FBNG

Clear browser cookies

‘6’ (0x36)

FBNG

Reboot operating system

‘7’ (0x37)

FBNG

Shutdown operating system

‘8’ (0x38)

FBNG

Collect email/browser passwords and create a screenshot

‘9’ (0x39)

zip_file_data>FBNG

Download and unpack ZIP archive into %TEMP% directory

Table 2: FormBook accepted commands

Distribution Campaigns

FireEye researchers observed FormBook distributed via email campaigns using a variety of different attachments:

PDFs with links to the “tny.im” URL-shortening service, which then redirected to a staging server that contained FormBook executable payloads

DOC and XLS attachments that contained malicious macros that, when enabled, initiated the download of FormBook payloads

ZIP, RAR, ACE, and ISO attachments that contained FormBook executable files

The PDF Campaigns

The PDF campaigns leveraged FedEx and DHL shipping/package delivery themes (Figure 7 and Figure 8), as well as a document-sharing theme. The PDFs distributed did not contain malicious code, just a link to download the FormBook payload.

The staging servers (shown in Table 3) appeared to be compromised websites.

Figure 7: Example PDF campaign email lure with attachment

Figure 8: Example PDF campaign attachment

Sample Subject Lines

Shorted URLs

Staging Servers

Recipient’s_Name> – You have a parcel awaiting pick up

Recipient’s_Name> – I shared a file with you

tny[.]im/9TK

tny[.]im/9Uw

tny[.]im/9G1

tny[.]im/9Q6

tny[.]im/9H1

tny[.]im/9R7

tny[.]im/9Tc

tny[.]im/9RM

tny[.]im/9G0

tny[.]im/9Oq

tny[.]im/9Oh

maxsutton[.]co[.]uk

solderie[.]dream3w[.]com

lifekeeper[.]com[.]au

brinematriscript[.]com

jaimagroup[.]com

Table 3: Observed email subjects and download URLs for PDF campaign

Based on data from the tny.im-shortened links, there were a total of 716 hits across 36 countries. As seen in Figure 9, most of the malicious activity from the PDF campaign impacted the United States.

Figure 9: Geolocation statistics from tny.im URL shortener

The DOC/XLS Campaigns

The email campaigns distributing DOC and XLS files relied on the use of malicious macros to download the executable payload. When the macros are enabled, the download URL retrieves an executable file with a PDF extension. Table 4 shows observed email subjects and download URLs used in these campaigns.

Sample Subject Lines

Staging Server

URL Paths

61_Invoice_6654

ACS PO 1528

NEW ORDER – PO-074

NEW ORDER – PO#074

REQUEST FOR QUOTATION/CONTRACT OVERHAUL MV OCEAN MANTA//SUPPLY P-3PROPELLER

URGENT PURCHASE ORDER 1800027695

sdvernoms[.]ml

/oc/runpie.pdf

/sem/essen.pdf

/drops/microcore.pdf

/damp/10939453.pdf

/sem/essentials.exe

/oc/runpie.pdf

/sem/ampama.pdf

/js/21509671Packed.pdf

/sem/essen.pdf

Table 4: Observed email subjects and download URLs for the DOC/XLS campaign

FireEye detection technologies observed this malicious activity between Aug. 11 and Aug. 22, 2017 (Figure 10). Much of the activity was observed in the United States (Figure 11), and the most targeted industry vertical was Aerospace/Defense Contractors (Figure 12).

Figure 10: DOC/XLS campaign malicious activity by date

Figure 11: Top 10 countries affected by the DOC/XLS campaign

Figure 12: Top 10 industry verticals affected by the DOC/XLS campaign

The Archive Campaign

The Archive campaign delivered a variety of archive formats, including ZIP, RAR, ACE, and ISO, and accounted for the highest distribution volume. It leveraged a myriad of subject lines that were characteristically business related and often regarding payment or purchase orders:

Sample Subject Lines

HSBC MT103 PAYMENT CONFIRMATION Our Ref: HBCCTKF8003445VTC

MT103 PAYMENT CONFIRMATION Our Ref: BCCMKE806868TSC Counterparty:.

Fwd: INQUIRY RFQ-18 H0018

Fw: Remittance Confirmation

NEW ORDER FROM COBRA INDUSTRIAL MACHINES IN SHARJAH

PO. NO.: 10701 – Send Quotaion Pls

Re: bgcqatar project

Re: August korea ORDER

Purchase Order #234579

purchase order for August017

FireEye detection technologies observed this campaign activity between July 18 and Aug. 17, 2017 (Figure 13). Much of the activity was observed in South Korea and the United States (Figure 14), with the Manufacturing industry vertical being the most impacted (Figure 15).

Figure 13: Archive campaign malicious activity by date

Figure 14: Top 10 countries affected by the Archive campaign

Figure 15: Top 10 industry verticals affected by the Archive campaign

Conclusion

While FormBook is not unique in either its functionality or distribution mechanisms, its relative ease of use, affordable pricing structure, and open availability make FormBook an attractive option for cyber criminals of varying skill levels. In the last few weeks, FormBook was seen downloading other malware families such as NanoCore. The credentials and other data harvested by successful FormBook infections could be used for additional cyber crime activities including, but not limited to: identity theft, continued phishing operations, bank fraud and extortion.

#gallery-0-5 { margin: auto; } #gallery-0-5 .gallery-item { float: left; margin-top: 10px; text-align: center; width: 33%; } #gallery-0-5 img { border: 2px solid #cfcfcf; } #gallery-0-5 .gallery-caption { margin-left: 0; } /* see gallery_shortcode() in wp-includes/media.php */

Go to Source Author: Nart Villeneuve Significant FormBook Distribution Campaigns Impacting the U.S. and South Korea Original Post from FireEye Author: Nart Villeneuve We observed several high-volume FormBook malware distribution campaigns primarily taking aim at Aerospace, Defense Contractor, and…

#FireEye

0 notes

alexonrails-blog · 7 years

Text

On feature flipping, deli coffee, and hashing algorithms

Life’s full of goods and bads .

Surfing. Good. Warm beer. Bad

In-N-Out animal style fries. Really good.

Rush hour L train traffic. Really bad.

Modern digital computers might be binary, but the world isn’t. There’s a whole list of things that fall somewhere in the middle. They’re not really good, and not totally bad. They’re good enough.

- Deli coffee - its no Blue Bottle, but it gets the job done.

- Dollar slice pizza - It will never approach Prince Street’s level, but at 4am who cares?

pseudorandomness - ...

Each of the above examples is a worthy candidate for today’s post, but this is a programming blog and you bet we’ll be talking about programming. Let’s look at pseudorandomness and use it to prove why sometimes in software good enough can be, well…good enough.

What would we do without Google? (read: StackOverflow). A quick search and 200ms later Merriam-Webster dictionary is breaking this fancy word down for us:

pseudo - “being apparently rather than actually as stated” synonym: sham

random - “without definite aim, direction, rule, or method”

Pseudorandomness is when something seems like it was generated without any rhyme or reason, when it actually did follow a series of rules. Not truly random, but random enough.

Even if this is the first time you’ve seen this word, it surely isn’t the first time you’ve experienced it. Think about the last time you played music on shuffle. Wasn’t it rad how your computer randomly flipped through songs, avoiding you the pain of having to listen to that playlist in the same order as yesterday’s morning commute. The truth is the song picks weren’t actually random, but that didn’t matter to you, did it? They were chosen pseudorandomly - not truly random, but random enough.

Our computers are great at lots of things. Performing complex calculations, predicting the outcome of events, even mining cryptocurrencies, but if there’s one thing they’re horrible at, its randomness. The problem is your computer is actually really dumb. It not only needs to be told what to do, but also exactly how to do it. This includes generating randomness.

side note: You haven’t experienced how dumb these machines really are until you’ve had a compiler scream at you for forgetting a semicolon.

Luckily computer scientists have been working on this problem of randomness for a long time and they’ve gotten pretty good at it as demonstrated by the shuffle example. The other day, while reviewing a pull request on a popular open source project I collaborate on, I came across a few lines of code that show a great use case for pseudorandomness. I figure if I find them cool then you might too. Let’s walk through this code with the intention that by the end of this post we’ll be equipped with concrete examples as to how us software developers can use pseudorandomness to build software that is good enough.

Flipper is an open source Ruby gem that makes feature flipping easy, while having minimal impact on your application no matter your data store, throughput, or experience. Let’s say our engineering team at VTS just finished a brand new Net Effective Rent (NER) calculator. Before we roll it out to all users we’d like to test it with a subset. This is what Flipper lets us do. We can enable this calculator for:

specific user

group of users

percentage of users

percentage of time

all actors

Flipper calls each of these ways a feature can be enabled a gate. As for all actors; An actor is anything that responds to flipper_id. For today’s conversation we’ll keep it simple and say an actor is a user, but its good to keep in mind this could be anything.

Flipper definitely lives up to its goal of being easy to use. You create features and enable them for one of these five options. As users access a page with said feature, your code asks Flipper “is this feature enabled for this user?”. Flipper responds true or false.

eg: enabling a feature for specific users

While I’d love to spend time talking about all of these gates in depth, the README does a great job. Instead, let’s focus on two of the more interesting gates.

percentage of time

percentage of actors

Percentage of Time

The percentage of time gate allows you to enable a feature for a given percentage of time. A feature that is enabled for 20% of the time means that 20% of the time a user navigates to a page with that feature it will be on. The other 80% it will be off. Put another way if we call

flipper.enabled?(user)

one hundred times it should return true 20 out of the 100 invocations. While there aren’t tons of use cases for this, there are a few. A great use case for the percentage of time gate is load testing. We’d like to see how our application responds to different numbers of users interacting with it. We start with a small percentage, say 5%. This means that 5% of users visiting the page will see the feature. Over the next few weeks we can gradually increase this percentage.

Show me the code!

What happens when the above line is evaluated? At the end of the day, or really the end of the call-stack, Flipper invokes a function, open? that returns true if a given user can see a feature, and false otherwise. Every one of the fives gates implements open?.

Can we write an implementation of open? that returns true n% of the time it is invoked.

This is software so of course we can and we’re going to use pseudorandomness to accomplish it. It might not be very obvious at first, but the solution is actually quite simple:

If you’re not familiar with Ruby’s Kernel#rand, no problem, its a core function that when “called without an argument...returns a pseudo-random floating point number between 0.0 and 1.0, including 0.0 and excluding 1.0″. Some possible outcomes of our function given a percentage of 0.5 are:

0.2 < 0.5

0.1 < 0.5

0.6 < 0.5

Assuming rand generates a random enough number we’re claiming that ~50% of the time the generated number is going to be less than 0.5, which is true.

Just in case you don’t believe me let’s test it out by invoking our function 100 times and counting the number of times it returns true.

Great that looks about right, but to avoid disappointing statistics professors worldwide (you never know who reads these things), let’s use a more appropriate sample size. This time we’ll call open?(0.5) a total of 10,000 times, recording the number of times per 100 calls it returns true.

note: every element in the array is a sample of 100 calls to open?(0.50) and shows the number of times true was returned.

It looks like the majority of the results are ~50, and with a little more code we get a nice visualization using Ruby’s awesome Gruff library.

Let’s also take the average while we’re at it:

On average our open? implementation returns true ~50.44 percent of the time. It isn’t exactly 50%, but I’d say its damn good enough. You now know how Flipper’s percentage of time gate implements open?

Hopefully, by now its clear how Flipper is able to turn a feature on for a percentage of time by taking advantage of pseudorandomness to produce a solution that isn’t perfect but undoubtedly works good enough. That leaves us with our next and most interesting question.

Percentage of Actors

How can Flipper turn a feature on for a percentage of users, while consistently returning the same answer when asked whether a given user is enabled without computing and persisting the list of users in some datastore? Taking our current formula, adding another form of pseudorandomness, and topping it off with a bit of creative thinking is all we’ll need to produce a really clever solution.

Given we want to check if a feature is turned on for a user, let’s start with our inputs:

User: { id: 1 }

Feature: { name: “ner_calculator” }

Percentage of time this feature should be on: 30 (note: we’ll use whole number representations of percentages to keep things simple)

If we could get a deterministic pseudo-random number based on the given user and feature then we should be able to use the same approach as above. Something along the lines of:

We know we can’t use rand as it only accepts one integer argument, not a user or feature. This is where Flipper gets very clever. I’ll show a simplified example (as to avoid needing to understand any of the implementation details such as how Flipper stores values in your datastore) of how Flipper accomplishes the percent of actors gate, and then explain step-by-step what’s going on.

Genius. The interesting idea here is figuring out how to generate a random number given a user and a feature. The key point is that this random number needs to always be the same number given the same inputs. Otherwise this gate would act more like a percentage of time gate and sometimes say yes this feature is enabled for this user and other times say no. Flipper accomplishes this via the crc32 checksum, a commonly used algorithm to detect accidental changes to data sent over a network, as a hashing function. Hashing algorithms take some input and (ideally) produce a unique output. Crc32 serves as a great hashing algorithm for our case because its efficient and only produces integers unlike some other hashing algorithms such as SHA-1 and MD5 that produce alphanumerics. The important thing to note here is that crc32 isn’t a perfect solution for our use case, but its good enough. The outputs aren’t random or else we’d end up with non-deterministic values. They’re just seemingly random!

eg: crc32 checksum of two similar strings. Notice there is seemingly no similarity between the two produced values despite the similarity of inputs. Given the same inputs the output will always be the same.

One more hashing function to make sure we all understand the high-level idea because it is important for this conversation. Say we have a hash function that given a string input returns the sum of its bytes.

Hashing a few strings:

At first this might seem like a decent hash function, but its actually horrible. Without even getting into the issues with passing in strings with varying encodings (Definitely check out my post on character encodings if you’re not up to speed on how computers handle characters. hint: “Alex” might not always sum to 394), its a bad hash function because its capable of producing many collisions. A collision is when two or more inputs hash to the same output. By simply rearranging the characters we’re not changing the total number of bytes so:

hash_str(”xlAe”)

hash_str(”eAlx”)

hash_str(”lxeA”)

all hash to 394! - assuming we’re working with ASCII or UTF-8 encoded strings.

Back to Flipper.

open? takes the user’s id and concatenates it with the feature name.

Substituting id with its value for a user with id: 22 and feature name: :ner_calculator. We get:

Futher reducing the lefthand side of the equation we calculate the crc32 checksum of “22ner_calculator” to get:

Since crc32 checksums are large integers we mod the result by 100 to get a value between 0 and 100:

Finally we stated we’re checking this user and feature with a percentage of 30:

At last Flipper tells us, “Yes, Alex, this user is enabled for this feature based on this percentage”:

Hopefully you’ve learned something and have a new-found appreciation for randomness! Be sure to check out the actual source code as you’re armed with enough knowledge to know exactly how these features are implemented.

0 notes