a-journey-of-a-thousand - Tumblr blog

[a-journey-of-a-thousand]

The Eight of Pentacles is a training montage you’ve ever seen, it is the acquisition of a skill set until you are the best at it, usually, it comes only after long, hard work (but sometimes you just get to download the information right into your brain) but after all the work you are rewarded by being a master of your craft.

[a-journey-of-a-thousand]

Francois Boucher. The Setting of the Sun, 1752.

[a-journey-of-a-thousand]

OIDC Scopes Analogy

Imagine a world without OIDC. When an app wants to access your google calendar, for example, it needs to request to access your login at google. Without OIDC, you would have to give the 3rd party app your actual login creds, that is, your email and password. That's not safe! With OIDC, it can generate a token for the 3rd party app to 'borrow' temporary access to your google account. It's like a one-time login 'password' (because a token is kind of like a password--it's a secret that's only between your logged-in session and the resource you want to access). Similar to how you get a unique access key-card when you check into a hotel. You don't own the hotel room, but you can borrow it for awhile. The awesome part about OIDC is that now you can have even better granular control over its access. You can have authentication scopes, for example. If you're a user, you don't have admin privileges. Or the 3rd party app can request to just view your google calendar, but not your google profile or display pic, for example. That's more secure! Scopes are kind of like, if you asked a cleaner to come clean your house, but you have some valuable jewellery in your bedroom. You give them a magic key that only opens the door to your kitchen and living room that you want cleaned, not the master key to every room in the house.

[a-journey-of-a-thousand]

Browser Cookie Analogy

I like to collect analogies. Here is one for browser cookies: The purpose of a cookie is to identify a user when they visit a site (as a shortcut to identification). A server sends a cookie if it wants the browser to save it. It is saved as name-value pairs. Your web browser will save it locally for the website to access later. It works similar to a coat-check. 🧥 1. You hand over your "coat" to coat-check which represents your pocket of data. This can be your account, your shopping cart, or any other info connected to you, from the perspective of the server. 2. You get a "ticket" that identifies your "coat." This is a cookie in some form of ID or unique identifier that links your identity to your pocket of data. 3. When you leave and return, you can get your "coat" with your "ticket." Your browser gives the website your cookie, and in exchange, it can reassemble the session data to some state before when you left it. source: https://www.kaspersky.com/resource-center/definitions/cookies

[a-journey-of-a-thousand]

Hard Startups

Some notes on startups:

- Instead of asking "what problem should I solve?" ask "what problem do I wish someone else would solve for me?"

- it’s often easier to succeed with a hard startup than an easy one.

- caveat: not all hard problems are worth solving

- your startup will rise to the level where it can no longer attract enough talented people.

- Another solution to this problem is to think about startups that can become quite successful with less than ten people.

- best situation is there are existing competitors but there seems to be something they all missed

http://www.paulgraham.com/schlep.html

https://blog.samaltman.com/hard-startups

https://www.youtube.com/watch?v=uvw-u99yj8w&ab_channel=YCombinator

[a-journey-of-a-thousand]

Relational Algebra

I am learning about how SQL is based on mathematical theory called relational algebra. It's a set of ideas, or a standard of what a good relational database design should be. it's quite formal and seems to draw mainly from set theory, but I also don't think it gets very mathematically abstract, and I don't want the following to give the impression that there's more abstraction going on than there really is in relational theory, but-- One thing I am excited about is this:

A join is the product of matrix multiplication

I've never seen matricies actually being used! It's absolutely incredible how there is a very real and tangible theory behind something to ubiquitous and practical as database design.

[a-journey-of-a-thousand]

[a-journey-of-a-thousand]

Why we need body-parser

A little embarrassing, but I'm building a REST API from scratch, and literally all I need to do is make one POST route and pass one piece of data from the client in a JSON body, and I'm pulling my hair out logging the request and searching for the body object. I could have sworn it was here. Was my entire experience as a developer an illusion? Little did I know, within this tiny statement, request.body that we so take for granted reveals an abstraction away from a whole lotta complicated server technology. It all started with streams. The data sent over the network is actually in a stream format. So what the body-parser (for example in node) does under the hood is collect and append all the data characters coming in as a stream, then tries to find a start and close bracket, and if it does, parses a json object.

A little something like this:

[a-journey-of-a-thousand]

Python Mocking and afterthoughts on meta-learning

Python's unittest library is a commonly used testing library and it comes with a useful mock ability. One of the mocks is @patch, which lets you mock out methods that your unit test might reference, but out of scope of the functionality you want to test.

An interesting caveat has to do with which namespace to replace the function from. You don't want to do it 'globally' but rather, relative to the code being run.

if there is module a that exports SomeClass

# a.py exports SomeClass

# b.py from a import SomeClass my_class instantiates SomeClass()

recall from modules that this import statement will save someClass as a global variable on module b

Here you want to patch on module b's SomeClass

@patch('b.SomeClass')

However if you import the other way:

# b.py import a a.SomeClass

You would patch it as

@patch('a.SomeClass')

Because even in module b it is referenced under the namespace 'a', recall that this import statement will instantiate and save the entire module a under the name 'a' in module b.

Thoughts on meta-learning

I've been feeling some pressure to try to learn a whole new stack as much as possible to become useful on a new project, and I've been experimenting and reflecting on the best way to go about this. My first approach was to do a deep dive into the inner workings of python, and the reason for this is that if I think long enough, I can conclude things like why the @patch mock library has this namespace caveat. This academic route is slow, deliberate, and even feeds my ego as I can be theoretical and sound smart and important. On the other hand, if I read up on mundane details about the @patch uses, (the quickguide actually), the benefit is that I don't have to spend brainpower coming up with these conclusions on my own. It's less impressive but it is a cognitive shortcut, especially when there are alot more difficult problems to solve on the job.

I thought it was nice motivation to get out of the safe and comfortable world of academia and get out into the exciting world of making real things. For the latter, doing a little bit of 'boring' gruntwork isn't a bad tradeoff.

[a-journey-of-a-thousand]

Subject Object "Self" in Python OOP

The difference between classes and instances are like programs vs. records. Programs have functions attached, but records store more basic data items. Ie. Employee class might have giveRaise() method defined on the class, but smaller details like name and address can be on the instance.

From the previous article on OOP, I noted that OOP inheritance works as search trees. Here the instance I1 is copied from class C1, which is itself a subclass of C2 and C3.

When an attribute is called, it resolves to the first namespace match it finds. For example, I2.name might resolve to C2.name, if that is the first place where it exists. Note that the search goes from left to right, so C2 has priority over C3.

Method calls work in the same way, if method I2.w is a reference call to method C3.w it means

"Call the C3.w function to process I2."

The method definition exists in memory only in the class, and the instance holds the data necessary that is relevant to the method call. When the method on C3 is executed, it's passed the entire I2 namespace to it as a first argument.

C3.w(I2)

This is where the self call would come from, by convention the variable is called self in python. The Instance is not explicit, it's implicitly defined.

It's called object-oriented because there is always a subject object when operations are run.

[a-journey-of-a-thousand]

Open Policy Agent (OPA)

On the topic of security at scale, one important function is to create and maintain authentication/authorization policy. This can be sophisticated and complex when coordinating many different services in a system. A way to decouple the management of the policy with the access of the policy is with a tool known as Open Policy Agent. can allow for authorization access, which users access what resources, what capabilities a container should have

decouples the policy setting from the policy enforcement

developers can set policy logic on the OPA server, and services can call the OPA, separating the two actions

A microservice can request some policy evaluation to OPA and receive an allowed/not allowed response

response data in JSON format

response is usually truthy/falsey but can also be a match result

two deployment modes: sidecar and admissions controller

1. Admissions Controller

whenever you execute command on kubctl (CLI) send request to API server, processes any requests

OPA takes requests from API server, process the request, send result back to API server

based on that the API server will decide if request will be allowed (admissions controller)

these results can be applied in CI/CD pipeline also

2. Sidecar Pattern

OPA is deployed alongside application in a microservice pod

Rego

syntax looks like golang

simple language, its purpose is to define auth and access policy logic

Allow: Composite objects sets (like higher order composite)

Set policy example

lets say we want a policy to let only employees to access endpoint GET /cars/id

input is a global variable that refers to the request object

can also use helper functions like is_employee to filter the request

the second part describes which request the filter applies to

[a-journey-of-a-thousand]

[a-journey-of-a-thousand]

Intro to Python OOP

Object oriented programming is different from using objects and code being object-based: pass objects around in scripts, use in expressions, called their methods, etc.

Object-oriented means objects will generally need to participate in an inheritance hierarchy.

Python class is a coding structure, device used to implement new kinds of objects in Python that support inheritance

Created with a new statement the class

The definition makes it look like a built-in type, and underlying it, it uses the more fundamental types

Designed to:

Create and manage new objects

Support inheritance (code customization and reuse)

When used properly can cut development time radically

Has two aspects, inheritance and composition

These ideas can apply to any application that can be decomposed into a set of objects

They are program units just like functions and modules, that package logic and data

They define new namespaces like modules

However have 3 critical distinctions that make them more useful when it comes to building objects:

Multiple Instances: are factories for generating objects. Every time we call a class, generate a new object with a distinct namespace, and each new object has access to the class’s attributes and gets a namespace of its own for its own data (similar to function closures)

Customization via inheritance: extend class and define attributes outside the class itself in subclasses. Can build up namespace hierarchies. This will come in handy for customization

Operator overloading: Python provides classes with special optional methods, for example classes can be sliced, concatenated, indexed. Similar options as built-in types

Classes are mostly "two bits of magic":

Special first argument in functions to receive *self*, the subject of the call (the class referencing itself)

Inheritance attribute search. When accessing a method or attribute using the .method syntax, python will automatically search up the class’s inheritance tree.

It’s not much, but OOP adds an extra layer of structure on top of flat procedural models

The difference between classes and instances:

Classes are instance factories

Instances represent the concrete items in a program’s domain

Inheritance can be represented as search trees, in fact this is close to how python classes are actually implemented under the hood.

In the diagram, an instance inherits attributes from its class, and a class inherits attributes from all classes above it in the tree.

[a-journey-of-a-thousand]

Enjoying the little things with Postman tools

One of the changes I've noticed as I gain experience as a developer is how much I appreciate putting the time in to learn the little things. By that I mean the things that aren't so exciting, new, or impressive, but simply serve to make my day-to-day more comfortable. Actually, it's equally important as anything else. This is how I've come to appreciate the tools I use and want to invest in learning more. I've started to get acclimated to the day-to-day experience of working on the back-end. There's a long way to go yet, but I'm excited to start exploring the back-end tools. One such tool is Postman. It's probably the best way to actually test out the APIs I'm working on, other than running integration tests.

I've used Postman before in very simple ways but now I'm interested in learning all its secrets. The first I've come across is the Collection Runner. It's an option on a collection menu to run all requests at the same time (or one after the other, but automatically).

I'm curious what it's best useful for. It almost seems like a pseudo-integration test, the way you can schedule, monitor, save results, and customize automated runs. I'm also delighted to find that there's so much more to Postman than I thought before. Since joining a back-end team I've noticed my notes have gotten shorter. I think this is mostly due to the breadth of knowledge I've had to absorb while being thrown into a new team. Previously I was studying at my own leisure and had time to dig into each subject deeply. I think this is a good adaptation however, I'll be able to cover more ground.

https://learning.postman.com/docs/running-collections/intro-to-collection-runs/

[a-journey-of-a-thousand]

Gateway Pattern

This is a design pattern for handling third party communication. In this pattern we keep a ‘gateway’ that encapsulates any external calls by the services in the app. This way there is a consistent interface. It's typically a simple wrapper around a foreign API, and all services will communicate through it, instead of trying to make independent calls to the foreign API directly.

Benefits:

Consistent

Great for dealing with awkward APIs

Easy to mock out for tests

Caveats:

Do not use when the endpoint is already well defined, then you just may be adding complexity

The impression I get is that if your app is better quality than the external service, the gateway will help insulate that, but if the external service is better quality than yours then don’t bother. Notes taken from Martin Fowler's post on gateway pattern

[a-journey-of-a-thousand]

Python modules

Modules in python are similar to Javascript where they are the top level grouping of code. When you make a ‘global’ variable in python it’s actually only up to the level of the module, which is usually loosely the same as one file, if you organize import/export by files.

When a module is imported in another file, during compilation it is saved as an object in your current file

# file b.py

import a

In file b you will have an object called a defined. If that module has any variables or methods they can be accessed through that object too.

# file a.py

def myFunc( … )

c = 10

In file b, there will be an object a, where you can access a.myFunc and a.c

Another way to import is by picking out specific methods/fields only:

# file b.py

From a import myFunc

myFunc()

Here, python will assign myFunc to the global level of the file, so you can access myFunc directly. It makes for less typing.

Assigning space in memory for a module object, running through all methods and fields and saving them can be an expensive computation for python, so it has an optimized way of doing it. It saves a cache file of the binary after it compiled with a timestamp, so it keeps track of any new changes in the module so it doesn’t recompile more times than it has to.

A result of this is that imports only happen once. You may find that once a import is run in a file, you are not able to run it again in the interactive session, or if you call it a second time in a file, the values don’t reset:

import a 

a.c = 11

import a

print (a.c)

# expected 11, got 10

Because it didn’t recompile.

(If you must recompile, you can use reload)

Sorry if I gave the impression that import runs at compile time. Another interesting thing is that similar to def, from and import are executable statements, not compile-time declarations. In general that means they run later. They are not run until python reaches that part of the code in execution. This means they can be nested in if statements, try catch statements, load modules conditionally, etc. Imported modules and names are not available until their associated import or from statements are run.

if (false):

import a

print a

>>> a not defined

I think the def one at least is familiar to javascript: in Javascript the function scope is run later than the global scope it is in, so you will initialize function definitions to your global object, but the variables inside the function are unknown before the function is executed.

[a-journey-of-a-thousand]

Python arguments

Python gives you alot of options when it comes to function arguments and parameters. So arguments can be defined in a function signature and also as parameters when calling the function, so those two are relevant here. You can have positional arguments, that are identified solely on the ordered position they are pass in, keyword arguments that are called using the form kwarg=value. Default arguments are the same, but are used when a function is defined:

def add(a=1, b=2)

print(a + b)

In contrast, keyword arguments are what we call the arguments we pass when we call a function

add(a=3, b=4)

>>> 7

They are pretty much the same, but they have different names depending on whether they are in the function definition or being passed as function call. Here, values a and b default to 1 and 2, but in the function call we pass in other values, so a is overridden to 3 and b to 4.

What they have in common is that they must come after positional arguments, this rule is similar to javascript.

def add(a=1, b=2, c)

# throws error because c is positional

What is interesting is the variable length arguments, using either the * to gather into a tuple, or a ** to gather into a dictionary. Can be useful for maintaining backwards compatibility, or if you want to loop through a dictionary in your function. They remind me of the … gather in javascript. 

def f(a, *pargs, **kwargs): print(a, args, kwargs)

f(1, 2, 3, x=1, y=2)

>>> 1, (2, 3) {‘y’: 2, ‘x’: 1}

These have the opposite effect if you are calling a function than if you are defining one:

 f(*(1, 2), **{‘d’: 4, ‘c’: 3})

# same as f(1, 2, c=3, d=4)

This is useful if you can’t predict the number of arguments that will be passed to a function ahead of time, and will make the call more generic.

To summarize, in thee header it collects any number of arguments, in the call it unpacks any number of arguments.