progress-log

[progress-log]

git-commit The contributors to these repositories know that a well-crafted Git commit message is the best way to communicate context about a change to fellow developers (and indeed to their future selves). A diff will tell you what changed, but only the commit message can properly tell you why. Peter Hutterer makes this point well: Re-establishing the context of a piece of code is wasteful. We can’t avoid it completely, so our efforts should go to reducing it [as much] as possible. Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator.

A well-cared for log is a beautiful and useful thing. A project’s long-term success rests (among other things) on its maintainability, and a maintainer has few tools more powerful than his project’s log. It’s worth taking the time to learn how to care for one properly. What may be a hassle at first soon becomes habit, and eventually a source of pride and productivity for all involved. In this post, I am addressing just the most basic element of keeping a healthy commit history: how to write an individual commit message. There are other important practices like commit squashing that I am not addressing here.

In order to create a useful revision history, teams should first agree on a commit message convention that defines at least the following three things:

Style. Markup syntax, wrap margins, grammar, capitalization, punctuation. Spell these things out, remove the guesswork, and make it all as simple as possible. The end result will be a remarkably consistent log that’s not only a pleasure to read but that actually does get read on a regular basis.

Content. What kind of information should the body of the commit message (if any) contain? What should it not contain?

Metadata. How should issue tracking IDs, pull request numbers, etc. be referenced? The seven rules of a great Git commit message

Separate subject from body with a blank line

Limit the subject line to 50 characters

Capitalize the subject line

Do not end the subject line with a period

Use the imperative mood in the subject line

Wrap the body at 72 characters

Use the body to explain what and why vs. how

Firstly, not every commit requires both a subject and a body. Sometimes a single line is fine, especially when the change is so simple that no further context is necessary. For example:

Fix typo in introduction to user guide

Nothing more need be said; if the reader wonders what the typo was, she can simply take a look at the change itself, i.e. use git show or git diff or git log -p. However, when a commit merits a bit of explanation and context, you need to write a body. For example:

Derezz the master control program MCP turned out to be evil and had become intent on world domination. This commit throws Tron's disc into MCP (causing its deresolution) and turns it back into a chess game.

Commit messages with bodies are not so easy to write with the -m option. You’re better off writing the message in a proper text editor.

—

Example:

Summarize changes in around 50 characters or less More detailed explanatory text, if necessary. Wrap it to about 72 characters or so. In some contexts, the first line is treated as the subject of the commit and the rest of the text as the body. The blank line separating the summary from the body is critical (unless you omit the body entirely); various tools like `log`, `shortlog` and `rebase` can get confused if you run the two together. Explain the problem that this commit is solving. Focus on why you are making this change as opposed to how (the code explains that). Are there side effects or other unintuitive consequences of this change? Here's the place to explain them. Further paragraphs come after blank lines. - Bullet points are okay, too - Typically a hyphen or asterisk is used for the bullet, preceded by a single space, with blank lines in between, but conventions vary here If you use an issue tracker, put references to them at the bottom, like this: Resolves: #123 See also: #456, #789

5. Use the Imperative Mood The imperative can sound a little rude; that’s why we don’t often use it. But it’s perfect for Git commit subject lines. One reason for this is that Git itself uses the imperative whenever it creates a commit on your behalf. Writing this way can be a little awkward at first. We’re more used to speaking in the indicative mood, which is all about reporting facts. That’s why commit messages often end up reading like this:

Fixed bug with Y

Changing behavior of X

And sometimes commit messages get written as a description of their contents:

More fixes for broken stuff

Sweet new API methods

To remove any confusion, here’s a simple rule to get it right every time.

A properly formed Git commit subject line should always be able to complete the following sentence:

If applied, this commit will your subject line here

6. Wrap the body at 72 characters

Git never wraps text automatically. When you write the body of a commit message, you must mind its right margin, and wrap text manually.

The recommendation is to do this at 72 characters, so that Git has plenty of room to indent text while still keeping everything under 80 characters overall.

A good text editor can help here. It’s easy to configure Vim, for example, to wrap text at 72 characters when you’re writing a Git commit. Traditionally, however, IDEs have been terrible at providing smart support for text wrapping in commit messages (although in recent versions, IntelliJ IDEA has finally gotten better about this).

Take a look at the full diff and just think how much time the author is saving fellow and future committers by taking the time to provide this context here and now. If he didn’t, it would probably be lost forever.

[progress-log]

what-is-git - extra Git is an example of a DVCS (hence Distributed Version Control System). Rather than have one place for the full version history of the software as was once popular, in Git, every developer's working copy of the code is also a repository that can contain the full history of all changes. In addition to being distributed, Git was designed with performance, security and flexibility in mind. Performance The algorithms implemented inside Git take advantage of deep knowledge about common attributes of real source code file trees, how they are usually modified over time and what the access patterns are.

Unlike some version control software, Git is not fooled by the names of the files when determining what the storage and version history of the file tree should be, instead, Git focuses on the file content itself. After all, source code files are frequently renamed, split, and rearranged. The object format of Git's repository files uses a combination of delta encoding (storing content differences), compression and explicitly stores directory contents and version metadata objects. Security The content as well as the true relationships between files and directories, versions, tags and commits, all of these objects in the Git repository are secured with a cryptographically secure hashing algorithm called SHA1. This protects the code and the change history against both accidental and malicious change and ensures that the history is fully traceable. Criticism One common criticism of Git is that it can be difficult to learn. Some of the terminology in Git will be novel to newcomers and for users of other systems, the Git terminology may be different.

[progress-log]

Getting-Started-A-Short-History-of-Git - Chapter 1.2 A Short History of Git The Linux kernel is an open source software project of fairly large scope. During the early years of maintenance (1991–2002), changes to the software were passed around as patches and archived files. In 2002, the Linux kernel project began using a proprietary DVCS called BitKeeper.

In 2005, the relationship between the community that developed the Linux kernel and the company that developed BitKeeper broke down, and the tool’s free-of-charge status was revoked. This prompted the Linux development community (and in particular Linus Torvalds, the creator of Linux) to develop their own tool based on some of the lessons they learned while using BitKeeper. Some of the goals of the new system were as follows: speed, simple design, strong support for non-linear development, fully distributed, able to handle large projects efficiently. Since 2005, Git has evolved to be easy to use, and is amazingly fast, very efficient with large projects, and has an incredible branching system for non-linear development.

Getting-Started-What-is-Git - Chapter 1.3 Snapshots, Not Differences The major difference between Git and any other VCS is the way Git thinks about its data. Conceptually, most other systems store information as a list of file-based changes. They think of the information they store as a set of files and the changes made to each file over time (this is commonly described as delta-based version control). Git doesn’t think of or store its data this way. Instead, Git thinks of its data more like a series of snapshots of a miniature filesystem. Every time you commit, or save the state of your project, Git basically takes a picture of what all your files look like at that moment and stores a reference to that snapshot. If files have not changed, Git doesn’t store the file again, just a link to the previous identical file it has already stored. Git thinks about its data more like a stream of snapshots.

Nearly Every Operation Is Local

Most operations in Git need only local files and resources to operate — generally no information is needed from another computer on your network. If you’re used to a CVCS where most operations have that network latency overhead, this aspect of Git will make you think that the gods of speed have blessed Git. Because the entire history of the project is right there on your local disk, most operations seem almost instantaneous.

Ex. Git doesn’t need to go to the server to get the history — it reads it from your local database, so you see it instantly. If you want to see the changes between the current version and the file a month ago, Git can look up the file a month ago and do a local difference calculation, instead of having to either ask a remote server to do it or pull an older version of the file from the remote server to do it locally.

This also means that there is very little you can’t do if you’re offline or off VPN. If you get on an airplane and want to do a little work, you can commit happily (to your local copy, remember?) until you get to a network connection to upload. ~Checksums + Hash Everything in Git is checksummed before it is stored and is then referred to by that checksum. So it’s impossible to change the contents of any file or directory without Git knowing about it—this functionality is integral to its philosophy. You can’t lose information in transit or get file corruption without Git being able to detect it.

The mechanism that Git uses for checksumming is called a SHA-1 hash. This is a 40-character string composed of hexadecimal characters (0–9 and a–f) and calculated based on the contents of a file or directory structure in Git. A SHA-1 hash example: this:24b9da6552252987aa493b52f8696cd6d3b00373

You will see these hash values everywhere in Git as it stores everything in its database not by file name but by the hash value of its contents. — The Three States

Pay attention now — here is the main thing to remember about Git if you want the rest of your learning process to go smoothly. Git has three main states that your files can reside in: modified, staged, and committed:

Modified means that you have changed the file but have not committed it to your database yet.

Staged means that you have marked a modified file in its current version to go into your next commit snapshot.

Committed means that the data is safely stored in your local database.

This leads us to the three main sections of a Git project:

The working tree is a single checkout of one version of the project. These files are pulled out of the compressed database in the Git directory and placed on disk for you to use or modify.

The staging area is a file, generally in your Git directory, that stores information about what will go into your next commit. Its technical name in Git parlance is the “index”, but the phrase “staging area” works just as well.

The Git directory is where Git stores the metadata and object database for your project. This is the most important part of Git, and it is what is copied when you clone a repository from another computer.

The basic Git workflow goes something like this:

You modify files in your working tree.

You selectively stage just those changes you want to be part of your next commit, which adds only those changes to the staging area.

You do a commit, which takes the files as they are in the staging area and stores that snapshot permanently to your Git directory.

Getting-Started-The-Command-Line - Chapter 1.4 The Command Line The command line is the only place you can run all Git commands — most GUIs implement only a partial subset of Git functionality for simplicity. If you can run the command-line version, you can probably also figure out how to run the GUI version, while the opposite is not necessarily true. Also, while your GUI is a matter of personal taste, all users will have the command-line tools installed and available.

[progress-log]

Getting-Started-About-Version-Control - Chapter 1.1 Version Control Systems (VCS) If you are a graphic or web designer and want to keep every version of an image or layout (which you would most certainly want to), a Version Control System (VCS) is a very wise thing to use. If you screw things up or lose files, you can easily recover, and you get all this for very little overhead. Local VCS Many people’s version-control method of choice is to copy files into another directory (perhaps a time-stamped directory, if they’re clever). This is a very common approach and incredibly error prone—you can accidentally write to the wrong file or copy over files you don’t mean to. To deal with this, programmers developed VCS, a simple database that kept all the changes to files under revision control. A popular VCS tool was a system called RCS, still distributed with many computers today. RCS keeps patch sets (the differences between files) in a special format on disk; it can then re-create what any file looked like at any point in time by adding up all the patches. Centralized VCS To collaborate with developers on other systems, Centralized Version Control Systems (CVCSs) were developed. These systems (such as CVS, Subversion, and Perforce) have a single server that contains all versioned files, and clients that check out files from that central place. For many years, this has been the standard for version control.

Con: the single point of failure that the centralized server represents. If that server goes down for an hour, no one can work on it + whenever you have the entire history of the project in a single place, you risk losing everything. Distributed VCS In a DVCS (such as Git, Mercurial or Darcs), clients don’t just check out the latest snapshot of the files; rather, they fully mirror the repository + its full history. Thus, if any server dies, any of the client repositories can be copied back up to the server to restore it. Every clone is really a full backup of all the data.

[progress-log]

what-is-git-and-github - extra *helpful But Sally is only one person. How do engineering teams make sure their work doesn’t overlap?

By making branches. Think of your Git commit history as a tree. The trunk is what we refer to as the master branch. In order for teams to avoid stepping on each other’s toes, they need to work in isolation from everyone else (on a feature branch), whilst ultimately still contributing back to the main codebase (the master branch). The general workflow starts with each author branching off master when they’re doing a piece of work (maybe writing a chapter, or editing one). When the rest of the co-authors have approved the changes, the branch gets merged into the master trunk. When a branch gets merged that means its content overrides the master. So any changes to existing content will supersede what was there before, and any new content will be added in too. In essence, when a branch gets merged into master, its commits get added to the top of the master history.

Your remote repository on Github is a mirror of what you have locally. So on your computer you have a local Git repository for that project (i.e.: a folder configured to allow Git commits). Inside this local Git repo (again, a specific Git-enabled folder on your computer), you have all the files that are part of this project. This ultimately syncs up to a location online. However, as we know, there are extra steps involved in the Git/Github workflow. You have to consciously decide to take a “snapshot” of your work as it stands at that moment in time (a commit) and deliberately choose to push that commit up to Github. Only then is the work “synced” to the online location (the Github repo).

So why not automate it? Why not make it like Dropbox and have the file update on Github as you update it locally?

Mainly, because of bugs. And you don’t always want to keep everything you write.

Sometimes you want to experiment, and should that fail, you want an easy way to go back to the last correctly working state. So the minute something is working the way you want it to, commit it before you try to edit it or experiment with a different approach. There’s no harm in committing small chunks of work very frequently—many engineers pride themselves on that doing exactly that. — Eventually, she’s ready for Tom and Adam to review her work and creates a Pull Request, a Github feature that allows her to explain the changes that her branch makes to the master branch. It's also where the co-authors can have a discussion about the branch’s content and request changes before the branch becomes part of the master trunk.

At this point, Sally can move to (or “check out”) the master branch on her local machine and pull down the very commits that were previously isolated in a feature branch. She’s now back at square one, with an updated master, and can proceed to create a new local branch for her next piece of work, helping poor Tom edit his chapter on Goblins. And so, the process would start all over again:

Create local branch

Write some commits within local branch

Push to Github

Create Pull Request explaining changes that branch contains

Merge to master

Pull down new master commits locally

Repeat

As you can see, this is a very smooth workflow, the perfect combination of working in isolation as well as collaboratively. Git — on your local machine — provides a brilliant way to create multiple versions of your work, via a rich annotated history, controlled and curated by you. Github — online — is a fantastic resource for not only storing and providing a clear visualisation of said history but also for collaborating and quality-control.

[progress-log]

what-is-git-and-github - extra *helpful Version Control is any system that allows you to understand the history of a file and how it has progressed. Back in my former life as a graphic designer I often had files that looked something like [v1, v2, etc.]. Git is a version control system specifically designed to work well with text files. Because ultimately, that’s all code really is: loads of text files hooked up together in some fashion. Git is a program you install which allows you to annotate the changes you make to create an easily navigable system history. Saving a file is a simplified version control system, but it’s not very good because it only allows you to move forward in time. The “undo” button lets you move backwards in the “life” of the file, but we all know it has certain limitations—ex. the file’s past is usually lost the minute you close the file. —

This is where Git shines. If Sally had been using Word combined with Git, she could’ve associated all those changes into one nice neat little summary of “Introduction of Elves into the series”. She could’ve seen the changes made across pages, chapters, files, and books, allowing her to truly understand the impact of the Elves on her fantasy series. This “neat little summary” is what we, in Git-land, refer to as a commit.

Recap. Git is a program that allows you to annotate the history of a system (or group) of files via commits, which are “snapshots” of the differences made to the system at a given point in time. Github takes that lovely commit history and hosts it online so that you can access it from any computer. You do this via pushing commits from your local machine (i.e.: the computer you’re currently using) up to Github, and then, from the new/different computer pulling those commits down. Sally author workflow ex. cont. "Sally starts her day working on her home desktop, writes some chapters, does some editing, etc. Along the way, she takes three “snapshots” (Git commits) of her work at strategic points... before she shuts down her home desktop computer, she makes sure to push her current Git commit history up online to Github. Once on Github, the commits live in a remote repository."

remote merely means online (vs local, which as we learned earlier means whichever computer you’re currently using). As for repository (often shorted to “repo”), it’s basically a folder with Git superpowers.

So, Github lets you store your work (which is annotated by Git commits) in a special online folder (repo). See? Simple.

[progress-log]

foundations-introduction-to-git Git is like a really epic save button for your files and directories. Officially, Git is a version control system. A save in a text editor records all of the words in a document as a single file. You are only ever given one record of the file, such as essay.doc, unless you make copies (which is difficult to remember to do and keep track of). A save in Git records differences in the files and folders AND keeps a historical record of each save. This feature is a game changer. As an individual developer, Git enables you to review how your project grows and to easily look at or restore file states from the past. Once connected to a network, Git allows you to push your project to GitHub or other alternatives for sharing and collaborating with other developers. While Git works on your local machine, GitHub is a remote storage facility on the web for all your coding projects. Almost all software development companies consider using Git to be an essential skill for modern web developers.

In this lesson, we will briefly explore the history of Git, what it is, and what it’s useful for. In the next lesson, we will go over the basic workflow for using Git, which should enhance your understanding and demonstrate why Git is so useful.

[progress-log]

shell-novice/02-filedir.html

General Syntax of a Shell Command

We have now encountered commands, options, and arguments, but it is perhaps useful to formalise some terminology. Consider the command below as a general example of a command, which we will dissect into its component parts:

ls is the command, with an option-F and an argument/. We’ve already encountered options which either start with a single dash (-), known as short options, or two dashes (--), known as long options. [Options] change the behavior of a command and Arguments tell the command what to operate on (e.g. files and directories). Sometimes options and arguments are referred to as parameters.  Sometimes options are referred to as switches or flags, especially options that take no argument. In this lesson we will stick with using the term option. Note that the sizes returned by ls -s are in blocks. These are defined differently for different operating systems. When options exist as both short and long options: - Use the short option when typing commands directly into the shell to minimize keystrokes and get your task done faster. - Use the long option in scripts to provide clarity. It will be read many times and typed once.

Nelle's Pipeline: Organizing Files

Nelle creates a directory called north-pacific-gyre (to remind herself where the data came from), which will contain her data files and data processing scripts. Each of her physical samples is labelled according to her lab’s convention with a unique ten-character ID, such as ‘NENE01729A’. This ID is what she used in her collection log to record the location, time, depth, and other characteristics of the sample, so she decides to use it within the filename of each data file. All 1520 files will go into the same directory.

If Nelle types part of a directory, then presses Tab, the shell will automatically complete the directory name for her using tab completion. Pressing Tab twice brings up a list of all the files. If Nelle then presses G and then presses Tab again, the shell will append ‘goo’ since all files that start with ‘g’ share the first three characters ‘goo’. To see all of those files, she can press Tab twice more.

Keypoints

The file system is responsible for managing information on the disk.

Information is stored in files, which are stored in directories (folders).

Directories can also store other directories, which then form a directory tree.

pwd prints the user’s current working directory.

ls [path] prints a listing of a specific file or directory; ls on its own lists the current working directory.

cd [path] changes the current working directory.

Most commands take options that begin with a single -.

Directory names in a path are separated with / on Unix, but \ on Windows.

/ on its own is the root directory of the whole file system.

An absolute path specifies a location from the root of the file system.

A relative path specifies a location starting from the current location.

. on its own means ‘the current directory’; .. means ‘the directory above the current one’.

[progress-log]

shell-novice/02-filedir.html

Exploring Other Directories

Let’s take a look at our Desktop directory by running ls -F Desktop, i.e., the command ls with the -F option and the argument Desktop.  cd stands for ‘change directory’, which is a bit misleading. The command doesn’t change the directory; it changes the shell’s current working directory—i.e. it changes the shell’s settings for what directory we are in. cd is akin to double-clicking a folder in a GUI to get into that folder. cd into the exercise-data directory we downloaded into Desktop. Many shell commands will not output anything to the screen when successfully executed. But if we run pwd after it, we can see that we are now in /Users/nelle/Desktop/shell-lesson-data/exercise-data. We now know how to go down the directory tree (i.e. how to go into a subdirectory), but how do we go up (i.e. how do we leave a directory and go into its parent directory) .. is a special directory name meaning “the directory containing this one”, or the parent of the current directory. If we run pwd after running cd .., we’re back in/Users/nelle/Desktop/shell-lesson-data -a stands for ‘show all’ (including hidden files); it forces ls to show us file and directory names that begin with ., such as .. (which, if we’re in /Users/nelle, refers to the /Users directory). As you can see, it also displays another special directory that’s just called ., which means ‘the current working directory’. It may seem redundant to have a name for it, but we’ll see some uses for it soon. In most command line tools, multiple options can be combined with a single - and no spaces between the options; ls -F -a is equivalent to ls -Fa. cd without an argument will return you to your home directory cd Desktop/shell-lesson-data/exercise-data works too to navigate through multiple directories at once. So far, when specifying directory names, or even a directory path (as above), we have been using relative paths. When you use a relative path with a command like ls or cd, it tries to find that location from where we are, rather than from the root of the file system. However, it is possible to specify the absolute path to a directory by including its entire path from the root directory, indicated by a leading slash. / tells the computer to follow the path from the root of the file system, so it always refers to exactly one directory, no matter where we are. To find the absolute path we’re looking for, we can use pwd and then extract the piece we need to move to shell-lesson-data. The shell interprets a tilde (~) character at the start of a path to mean “the current user’s home directory”. For example, if Nelle’s home directory is /Users/nelle, then ~/data is equivalent to /Users/nelle/data. This only works if it is the first character in the path;here/there/~/elsewhere is not here/there/Users/nelle/elsewhere. cd will translate - into the previous directory I was in, which is faster than having to remember, then type, the full path. This is a veryefficient way of moving back and forth between two directories – i.e. if you execute cd -twice, you end up back in the starting directory. The difference between cd .. and cd - is that the former brings you up, while the latter brings you back.

[progress-log]

shell-novice/02-filedir.html

The file system is the part of the operating system responsible for managing files and directories. It organizes our data into files, which hold information, and directories (also called ‘folders’), which hold files or other directories.

PWD

pwd (which stands for ‘print working directory’) shows you where you are. Directories are like places — at any time while we are using the shell, we are in exactly one place called our current working directory. Commands mostly read and write files in the current working directory, i.e. ‘here’, so knowing where you are before running a command is important.  Here, the computer’s response is /Users/nelle, which is Nelle’s home directory. Let’s have a look at how the file system as a whole is organized.

Like an upside down tree, the topmost directory is the root directory that holds everything else. We refer to it using a slash character, /, on its own; this character is the leading slash in /Users/nelle. Inside that directory are several other directories: bin (which is where some built-in programs are stored), data (for miscellaneous data files), Users (where users’ personal directories are located), tmp (for temporary files that don’t need to be stored long-term), and so on.

/ Notice that there are two meanings for the / character. When it appears at the front of a file or directory name, it refers to the root directory. When it appears inside a path, it’s just a separator. We can see what’s in our home directory by running ls: ls prints the names of the files and directories in the current directory. We can make its output more comprehensible by using the -F option which tells ls to classify the output by adding a marker to file and directory names to indicate what they are: - a trailing / indicates that this is a directory - @ indicates a link - * indicates an executable See that the home directory contains only sub-directories. Any names in the output without a classification symbol are files in the current working directory. Clear your terminal using the clear command. — ls has lots of other options—use $ls --help or $man ls to see the manual on macOS. You can use two options at the same time. To search in the man pages, use / followed by the search term, and N / Shift + N to move forward / backward. To quit the man pages, press Q. By default, ls lists the contents of a directory in alphabetical order by name. $ls -t lists items by time of last change instead of alphabetically. $ls -r lists the contents of a directory in reverse order. You may need to use the -l option to see the last changed dates.

[progress-log]

shell-novice On GUIs

The most widely used way to interact with personal computers is called a graphical user interface (GUI). With a GUI, we give instructions by clicking a mouse and using menu-driven interactions. While intuitive to learn, this way of delivering instructions to a computer scales poorly. Imagine: to copy the third line of one thousand text files in one thousand different directories and paste it into a single file [would take] hours with a GUI, and you could commit an error. The Unix shell is both a command-line interface (CLI) and a scripting language, allowing such repetitive tasks to be done automatically and fast. With the proper commands, the shell can repeat tasks with or without some modification as many times as we want, [and complete the task in the example] in seconds. While a GUI presents you with choices to select, CLI choices are not automatically presented to you, so you must learn a few commands. Unlike a spoken language, a small number of “words” (i.e. commands) gets you a long way. Sequences of commands can be written into a script, improving the reproducibility of workflows. As clusters and cloud computing systems become more popular for scientific data crunching, being able to interact with the shell is becoming a necessary skill. We can build on the command-line skills covered here to tackle a wide range of scientific questions and computational challenges. — Most popular shell environments by default put your user name and the host name before the $. Our first command, ls, which is short for listing. This command will list the contents of the current directory. If the shell can’t find a program whose name is the command you typed, it will print an error message such as: ks: command not found

Belle's Pipeline: A Typical Problem

Nelle Nemo has 1520 samples that she needs to run through an imaginary program called goostats.sh. In addition to this huge task, she has to write up results by the end of the month, so her paper can appear in a special issue of Aquatic Goo Letters. If Nelle chooses to run goostats.sh by hand using a GUI, she’ll have to select and open a file 1520 times. If goostats.sh takes 30 seconds to run each file, the whole process will take more than 12 hours of Nelle’s attention. With the shell, Nelle can instead assign her computer this mundane task while she focuses her attention on writing her paper. More specifically, the lessons explain how she can use a command shell to run the goostats.sh program, using loops to automate the repetitive steps of entering file names, so that her computer can work while she writes her paper. As a bonus, once she has put a processing pipeline together, she will be able to use it again whenever she collects more data. In order to achieve her task, Nelle needs to know how to: - navigate to a file/directory - create a file/directory - check the length of a file - chain commands together - retrieve a set of files - iterate over files - run a shell script containing her pipeline

Keypoints

A shell is a program whose primary purpose is to read commands and run other programs. Bash is the default shell in many implementations of Unix. The shell’s main advantages are its high action-to-keystroke ratio, its support for automating repetitive tasks, and its capacity to access networked machines.

[progress-log]

shell-novice

The Unix shell has been around longer than most of its users have been alive. It has survived because it’s a powerful tool that allows users to automate repetitive tasks and easily combine smaller tasks into larger, more powerful workflows. If you recognize the word “file” and either “directory” or “folder” (two common words for the same thing), you’re ready for this lesson. — The shell is a program that enables us to send commands to the computer and receive output, also referred to as the terminal or command line. Some computers include a default Unix Shell program.  For a Mac computer running macOS Mojave or earlier releases, the default Unix Shell is Bash. For a Mac computer running macOS Catalina or later releases, the default Unix Shell is Zsh. ```echo $SHELL``` to determine shell you're running (Zsh)

[progress-log]

foundations-command-line-basics

Terminal: On Linux and older Macs the line will end with $ and on newer Macs the line will end with %. This symbol - called the prompt - indicates that the terminal is waiting for you to enter a command. Type whoami and press Enter. Often guides and instructions for using the terminal will indicate commands by putting the symbol first, like $ whoami. This tells you to type the command in your terminal, but don’t enter the $. On a newer Mac, the % is the same as $ You will be making heavy use of the command line throughout this curriculum, and the upcoming installations project will need you to install many different software programs using the command line. Additionally, you will primarily be using Git within the command line. You may well be using the command line on a daily basis as a software developer, making it an indispensable skill in your toolset.

[progress-log]

foundations-text-editors

A text editor is by far the most used developer tool regardless of what type of developer you are. A good text editor can help you write better code with real-time code checking, syntax highlighting, and automatic formatting. Plugins, syntax highlighting, auto-closing of brackets and braces, and linting are just a few of the benefits of using a code editor. Visual Studio Code, or just VSCode as it’s commonly referred to, is an excellent free code editor. It has outstanding add-on support and great Git integration. VSCode is the most popular code editor among Odin’s students and moderators, so support is easy to find in the community. Keep in mind that this means you will not be able to get help if you are using a different text editor other than VSCode for the curriculum. [unrelated] As a reminder, if you’re using a virtual machine, you should install VSCode on your VM. You’re welcome to also install it on your host (i.e., your Windows main OS), but you’ll want to be sure that you have this critical tool inside your VM. —

foundations-command-line-basics

That blank screen or window with a prompt and blinking cursor is the command line interface (CLI), where you’re able to enter commands that your computer will run for you. While there’s no need for you to reenact the scene above, working with the command line is a critical skill for you to learn as a developer. The command line is like our base of operations, from which we can launch other programs and interact with them. It has a syntax of its own to learn, but since you’ll be entering the same commands dozens of times, you’ll quickly pick up the commands you need most. In this introductory lesson to the command line, you’ll learn how to navigate around your computer and how to manipulate files and directories (also known as folders) directly from the comfort of the command line. You’ll soon see that this isn’t as difficult as you may think. The commands you will learn in this lesson are very straightforward. So don’t let the prospect of using the command line for the first time intimidate you.

[progress-log]

foundations-installation-overview

Many online development courses use in-browser code editors or “sandboxes.” However, the best way to set yourself up for long-term success is to operate in a real development environment. We won’t lie to you: installing packages, editors, and even entire operating systems can be very frustrating. However, having the experience of setting up a development environment to run the code you’ll write is an invaluable, real-world skill... ... Dual-booting can be as easy as inserting a flash drive and clicking a few buttons. The benefits of dual-booting cannot be overstated. You will allow Linux to access the full capabilities of your hardware, have a clean and distraction-free environment for coding, and learn the platform used by many senior developers and servers around the world. Development Tools Are Built for Linux - Ruby (on Rails) and Node.js, popular backend technologies covered by The Odin Project and widely used in the larger web development community, are open source projects that explicitly expect to run on an open-source (UNIX-based) platform like Linux.

[progress-log]

MDN: how_the_Web_works#clients_and_servers

Clients and servers

Computers connected to the internet are called clients and servers.

Clients are the typical web user's internet-connected devices (for example, your computer connected to your Wi-Fi, or your phone connected to your mobile network) and web-accessing software available on those devices (usually a web browser like Firefox or Chrome).

Servers are computers that store webpages, sites, or apps. When a client device wants to access a webpage, a copy of the webpage is downloaded from the server onto the client machine to be displayed in the user's web browser.

The other parts of the toolbox

analogy: imagine web = road, client = house, server = store.

In addition to the client and the server, we also need to say hello to:

Your internet connection: Allows you to send and receive data on the web. Like the street between your house and the shop.

TCP/IP: Transmission Control Protocol and Internet Protocol are communication protocols that define how data should travel across the internet. This is like the transport mechanisms that let you place an order, go to the shop, and buy your goods. In our example, this is like a car or a bike (or however else you might get around).

DNS: Domain Name System is like an address book for websites. When you type a web address in your browser, the browser looks at the DNS to find the website's IP address before it can retrieve the website. The browser needs to find out which server the website lives on, so it can send HTTP messages to the right place (see below). This is like looking up the address of the shop so you can access it.

HTTP: Hypertext Transfer Protocol is an application protocol that defines a language for clients and servers to speak to each other. This is like the language you use to order your goods.

Component files: A website is made up of many different files, which are like the different parts of the goods you buy from the shop. These files come in two main types:

Code files: Websites are built primarily from HTML, CSS, and JavaScript, though you'll meet other technologies a bit later.

Assets: This is a collective name for all the other stuff that makes up a website, such as images, music, video, Word documents, and PDFs.

So what happens, exactly?

When you type a web address into your browser (for our analogy that's like walking to the shop):

The browser goes to the DNS server, and finds the real address of the server that the website lives on (you find the address of the shop).

The browser sends an HTTP request message to the server, asking it to send a copy of the website to the client (you go to the shop and order your goods). This message, and all other data sent between the client and the server, is sent across your internet connection using TCP/IP.

If the server approves the client's request, the server sends the client a "200 OK" message, which means "Of course you can look at that website! Here it is", and then starts sending the website's files to the browser as a series of small chunks called data packets (the shop gives you your goods, and you bring them back to your house).

The browser assembles the small chunks into a complete web page and displays it to you (the goods arrive at your door — new shiny stuff, awesome!).

Order in which component files are parsed

When browsers send requests to servers for HTML files, those HTML files often contain <link> elements referencing external CSS stylesheets and <script>elements referencing external JavaScript scripts. It's important to know the order in which those files are parsed by the browser as the browser loads the page:

The browser parses the HTML file first, and that leads to the browser recognizing any <link>-element references to external CSS stylesheets and any <script>-element references to scripts.

As the browser parses the HTML, it sends requests back to the server for any CSS files it has found from <link> elements, and any JavaScript files it has found from <script> elements, and from those, then parses the CSS and JavaScript.

The browser generates an in-memory DOM tree from the parsed HTML, generates an in-memory CSSOM structure from the parsed CSS, and compiles and executes the parsed JavaScript.

As the browser builds the DOM tree and applies the styles from the CSSOM tree and executes the JavaScript, a visual representation of the page is painted to the screen, and the user sees the page content and can begin to interact with it.

DNS explained

Real web addresses aren't the nice, memorable strings you type into your address bar to find your favorite websites. They are special numbers that look like this: 192.0.2.172.

This is called an IP address, and it represents a unique location on the web. However, it's not very easy to remember, is it? That's why the Domain Name System was invented. This system uses special servers that match up a web address you type into your browser (like "mozilla.org") to the website's real (IP) address.

Websites can be reached directly via their IP addresses. You can use a DNS lookup tool to find the IP address of a website.

Packets explained

Earlier we used the term "packets" to describe the format in which the data is sent from server to client. What do we mean here? Basically, when data is sent across the web, it is sent in thousands of small chunks. There are multiple reasons why data is sent in small packets. They are sometimes dropped or corrupted, and it's easier to replace small chunks when this happens. Additionally, the packets can be routed along different paths, making the exchange faster and allowing many different users to download the same website at the same time. If each website was sent as a single big chunk, only one user could download it at a time, which obviously would make the web very inefficient and not much fun to use.

[progress-log]

MDN: pages_sites_servers_and_search_engines web page: a document which can be displayed in a web browser. ^ can embed styles (look), scripts (interactivity), media (content) website: a group of web pages usually connected together

— web server: a computer that hosts a website on the Internet A web server is a computer hosting one or more websites. "Hosting" means that all the web pages and their supporting files are available on that computer. The web server will send any web page from the website it is hosting to any user's browser, per user request. Don't confuse websites and web servers. "My website is not responding", means the web server is not responding and therefore the website is not available. Since a web server can host multiple websites, the term web server is never used to designate a website, as it could cause great confusion. If we said, "My web server is not responding", that means multiple websites on that web server are not available. What_is_a_web_server: web server can refer to hardware or software

On the hardware side, a web server is a computer that stores web server software and a website's component files (for example, HTML documents, images, CSS stylesheets, and JavaScript files). A web server connects to the Internet and supports physical data interchange with other devices connected to the web.

On the software side, a web server includes several parts that control how web users access hosted files. At a minimum, this is an HTTP server. An HTTP server is software that understands URLs (web addresses) and HTTP(the protocol your browser uses to view webpages). An HTTP server can be accessed through the domain names of the websites it stores, and it delivers the content of these hosted websites to the end user's device.

— search engine: a web service that helps you find other web pages A browser is a piece of software that retrieves and displays web pages; a search engine is a website that helps people find web pages from other websites. The confusion arises because, the first time someone launches a browser, the browser displays a search engine's homepage. This makes sense, because, obviously, the first thing you want to do with a browser is to find a web page to display. Don't confuse the infrastructure (e.g., the browser) with the service (e.g., the search engine). analogy: public library as web server library sections as websites books at catalog numbers as webpages at unique URLs index as search engine