iamadevops

[iamadevops]

Problem with the SSL CA cert (path? access rights?)

If you encounter the error message in the title on Ubuntu / Debian, it means your ca-certs are outdated / corrupted.

The fix is to upgrade your ca-certs.

sudo apt-get update && sudo apt install -y ca-certificates

[iamadevops]

systemd-resolved get current DNS servers

When you try to fetch your current DNS server settings on your Linux machine and it is using systemd-resolve, the /etc/hosts won’t be useful, neither the /etc/systemd/resolved.conf as they’re empty by default.

Run the command below to get your current nameservers:

sudo resolvectl status

[iamadevops]

tfsec ignore multiple rules at once

source.tf

...

#tfsec:ignore:AWS002 tfsec:ignore:GEN001 resource "aws_s3_bucket" "my_bucket" { ... } ...

[iamadevops]

Clear Golang / Module cache

$ go clean -cache -modcache -i -r

[iamadevops]

How to convert RSA / OpenSSL / GPG keys to JSON format

You might using vault to store secrets and you need to pull them down to kubernetes or ECS cluster whatever. So you need to store your SSL certs and GPG keys / RSA keys in JSON format which is a huge pain.

So this is how you convert your key files into a JSON compatible one line:

``` $ awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' ssl.ca ```

[iamadevops]

Apache2 + ProxyPass + Certbot

You might want to have an apache2 as a reverse proxy and have a web application running in a docker container. Obviously we don’t want to let the certbot to write into the docker container (neither want to mount any directory to the container) so we want to proxypass every request except the URL for the http01 resolver. The solution is below. It will proxy all requests to the web app on localhost, except the acme-challenge so the cert-bot http01 challenge will succeed.

<VirtualHost *:80>  ServerName example.com  DocumentRoot "/var/www/example.com"  ProxyPreserveHost On

 ProxyPass /.well-known/acme-challenge/ !  ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/ </VirtualHost>

Also you want to redirect everything to https, except the acme challenge. So you need 2 block. One below to do the redirect (except the acme challenge), and another *:443 block with the ProxyPass.

<VirtualHost *:80>  ServerName example.com  DocumentRoot "/var/www/example.com"  RewriteEngine On

 RewriteCond %{HTTPS} !=on  RewriteCond %{REQUEST_URI} !^/\.well-known/  RewriteRule ^/?(.*) https://example.com/$1 [R,L]

</VirtualHost>

<VirtualHost *:443>  ServerName example.com  DocumentRoot "/var/www/example.com"  ProxyPreserveHost On  SSLEngine on  SSLCipherSuite AES256+EECDH:AES256+EDH  SSLProtocol All -SSLv2 -SSLv3  SSLHonorCipherOrder On  SSLCompression off  SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem  SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem  ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/   ErrorLog /var/log/apache2/example.com-error.log CustomLog /var/log/apache2/example.com-access.log combined </VirtualHost>

[iamadevops]

GPG key photo viewing

gpg --list-options show-photos --fingerprint 0xABCDEFGH1234

[iamadevops]

Switching to Chromebook - Part II.

http://blog.wooh.hu/post/switching-to-chromebook-part-2/

[iamadevops]

Switching to Chromebook - Part I.

http://blog.wooh.hu/post/switching-to-chromebook-part-1/

[iamadevops]

Howto fix jmap -F -dump:format=b on ArchLinux

When you're running ArchLinux and want to create a heapDump from your java application might encounter the following exception:

Caused by: sun.jvm.hotspot.utilities.AssertionFailure: Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got 

The full stacktrace is here:

# /opt/java7/bin/jmap -F -dump:format=b,file=/root/heap.dump 15739 Attaching to process ID 15739, please wait... Debugger attached successfully. Server compiler detected. JVM version is 23.25-b01 Dumping heap to /root/heap.dump ... Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.tools.jmap.JMap.runTool(JMap.java:197) at sun.tools.jmap.JMap.main(JMap.java:128) Caused by: sun.jvm.hotspot.utilities.AssertionFailure: Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32) at sun.jvm.hotspot.oops.ObjectHeap.collectLiveRegions(ObjectHeap.java:605) at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:244) at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:416) at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77) ... 6 more

The solution is to re-compile your java package with the debug symbols. Unfortunately the makepkg removes them when building the package. But you can override this by adding the following option to your PKGBUILD file:

options=(!strip)

This will leave the debug symbols in your code and you'll be able to create a java heapDump after installing the newly compiled package.

The related part from the makepkg.conf(5) man page:

OPTIONS=(strip docs libtool staticlibs emptydirs zipman purge !upx)

This array contains options that affect the default packaging. They are equivalent to options that can be placed in the PKGBUILD; the defaults are shown here. All options should always be left in the array; to enable or disable an option simply remove or place an “!” at the front of the option. If an option is specified multiple times, the final value takes precedence. Each option works as follows:

strip

Strip symbols from binaries and libraries. If you frequently use a debugger on programs or libraries, it may be helpful to disable this option.

[iamadevops]

Offset2lib: bypassing full ASLR on 64bit Linux

Awesome 'little' bug in implementation!

http://cybersecurity.upv.es/attacks/offset2lib/offset2lib.html

[iamadevops]

chef resource to be executed during compile time (1st phase)

Yesterday I had an issue with chef + vagrant. I was trying to use the following chef code:

package "make" chef_gem 'chef-handler-jenkins' require 'chef/handler/jenkins'

The "make" is required for chef-handler-jenkins because it wants to install libyajl2 which needs to compile something on the machine.

The problem is that the chef_gem resource is executed during the 1st phase, while the other resources like the package resource gets only executed in the 2nd phase, so chef_gem will fail due to the missing make command. The solution is quite simple. Force the package resource to run in the 1st phase and not in the 2nd phase.

So I changed the code as follows:

package "make" do action :nothing end.run_action(:install)

chef_gem 'chef-handler-jenkins'

require 'chef/handler/jenkins'

[iamadevops]

gem install libyajl2 failure in Vagrant

If you get an error from libyajl2 when installing it on a brand new vagrant ubuntu image, don't be surprised:

root@cloudbees-chef:/home/vagrant# /opt/chef/embedded/bin/gem install libyajl2 Building native extensions. This could take a while... ERROR: Error installing libyajl2: ERROR: Failed to build gem native extension.

/opt/chef/embedded/bin/ruby extconf.rb creating Makefile /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/libyajl2-1.2.0/ext/libyajl2 extconf.rb:104:in `makemakefiles': unhandled exception from extconf.rb:138:in `<main>'

Gem files will remain installed in /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/libyajl2-1.2.0 for inspection. Results logged to /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/libyajl2-1.2.0/ext/libyajl2/gem_make.out

The solution is to install make. This error message is a bit stupid, but it means you don't have make or gcc. In our case, a default ubuntu server does not have make installed, so do it via:

apt-get install make

[iamadevops]

Vagrant-berkshelf failure 4.0.1 when config.json exists

Yesterday I ran into a problem with the vagrant-berkshelf plugin. When I was running vagrant up, i got the following exception:

/Users/apapai/.vagrant.d/gems/gems/vagrant-berkshelf-4.0.1/lib/vagrant-berkshelf/action/upload.rb:73:in `read': wrong number of arguments (0 for 1..4) (ArgumentError)

This sounds bad. But luckily it's just a typo. Since nobody uses nowadays custom config for berkshelf, it was not caught by others.

I created a [patch] which is merged now, so just patch your local upload.rb until it get's released in 4.0.2.

[iamadevops]

ssh using loop in bash without exiting in the first iteration

Shell can be sometimes pretty tricky.

For example when you want to use a code like will only execute your ssh command on the first server and exit from the loop:

while read server; do ssh $server "do something"; done < server_list.txt

The solution is using the ssh with the option "-n".

The related snippet from the manual:

-n

Redirects stdin from /dev/null (actually, prevents reading from stdin).  This must be used when ssh is run in the background.  A common trick is to use this to run X11 programs on a remote machine.  For example, ssh -n shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi, and the X11 connection will be automatically forwarded over an encrypted channel.  The ssh program will be put in the background.  (This does not work if ssh needs to ask for a password or passphrase; see also the -f option.)

[iamadevops]

sudo: How to preserve existing environment variables

In some cases, you want to use sudo and preserve the environmental variables. Let's say you read options from the ENV in your scripts. Of course your scripts are executed via sudo due to security reasons.

First, you should read the sudo(8) manual page and you'll find the 1st solution there

-E, --preserve-env Indicates to the security policy that the user wishes to preserve their existing environment variables. The security policy may return an error if the user does not have permission to preserve the environment.

The last line indicates, that probably you have to change other settings as well. So you should next read the sudoers(5) manual.

So the 1st solution is to override the default env_reset, to !env_reset in your sudoers file, but this is a less secure approach.

Defaults !env_reset

A better way - the 2nd solution - is to define what environmental variables are allowed to preserve in your /etc/sudoers.d/<yourfile>. Like this:

Defaults env_keep += "VARIABLE_ONE VARIABLE_TWO"

This is a more secure and controllable solution than the previous one. So then you can execute your scripts like:

sudo VARIABLE_ONE="this" VARIABLE_TWO="that" /path/to/script

[iamadevops]

CCP: Changelog Client Python

Since the changelog server is a pretty good way to collect events from your distributed systems, I wrote a small python client to reduce the code duplication :) You can install it simply by:

pip install ccp

Pypi: http://pypi.python.org/pypi/ccp/ GitHub: https://github.com/woohgit/changelog-client-python