#JAVA_OPTS | Explore Tumblr posts and blogs

lupbiy · 8 years ago

Text

为什么优秀的程序员喜欢命令行？

优秀的程序员

要给优秀的程序员下一个明确的定义无疑是一件非常困难的事情。擅长抽象思维、动手能力强、追求效率、喜欢自动化、愿意持续学习、对��码质量有很高的追求等等，这些维度都有其合理性，不过又都略显抽象和主观。

（图片来自：http://t.cn/R6I1yhJ）

我对于一个程序员是否优秀，也有自己的标准，那就是TA对命令行的熟悉／喜爱程度。这个特点可以很好的看出TA是否是一个优秀的（或者潜在优秀的）程序员。我周围就有很多非常牛的程序员，无一例外都非常擅长在命令行中工作。那什么叫熟悉命令行呢？简单来说，就是90%的日常工作内容可以在命令行完成。

当然，喜欢／习惯使用命令行可能只是表象，其背后包含的实质才是优秀的程序员之所以优秀的原因。

自动化

Perl语言的发明者Larry Wall有一句名言：

The three chief virtues of a programmer are: Laziness, Impatience and Hubris. – Larry Wall

懒惰（Laziness）这个特点位于程序员的三大美德之首：唯有懒惰才会驱动程序员尽可能的将日常工作自动化起来，解放自己的双手，节省自己的时间。相比较而言，不得不说，GUI应用天然就是为了让自动化变得困难的一种设计（此处并非贬义，GUI有着自己完全不同的目标群体）。

（图片来自：http://t.cn/R6IBgYV）

GUI更强调的是与人类的直接交互：通过视觉手段将信息以多层次的方式呈现，使用视觉元素进行指引，最后系统在后台进行实际的处理，并将最终结果以视觉手段展现出来。

这种更强调交互过程的设计初衷使得自动化变得非常困难。另一方面，由于GUI是为交互而设计的，它的响应就不能太快，至少要留给操作者反应时间（甚至有些用户操作需要人为的加入一些延迟，以提升用户体验）。

程序员的日常工作

程序员除了写代码之外，还有很多事情要做，比如自动化测试、基础设施的配置和管理、持续集成/持续发布环境，甚至有些团队还需要做一些与运维相关的事情（线上问题监控，环境监控等）。

开发/测试

基础设施管理

持续集成/持续发布

运维（监控）工作

娱乐

而这一系列的工作背后，都隐含了一个自动化的需求。在做上述工作时，优秀的程序员会努力将其自动化，如果有工具就使用工具；如果没有，就开发一个新的工具。这种努力让一切都尽可能自动化起来的哲学起源于UNIX世界。

而UNIX哲学的实际体现则是通过命令行来完成的。

Where there is a shell, there is a way.

UNIX编程哲学

关于UNIX哲学，其实坊间有多个版本，这里有一个比较详细的清单。虽然有不同的版本，但是有很多一致的地方：

小即是美

让程序只做好一件事

尽可能早地创建原型(然后逐步演进)

数据应该保存为文本文件

避免使用可定制性低下的用户界面

审视这些条目，我们会发现它们事实上促成了自动化一切的可能性。这里列举一些小的例子，我们来看看命令行工具是如何通过应用这些哲学来简化工作、提高效率的。一旦你熟练掌握这些技能，就再也无法离开它，也再也忍受不了低效而复杂的各种GUI工具了。

命令行如何提升效率

一个高阶计算器

在我的编程生涯早期，读过的最为振奋的一本书是《UNIX编程环境》，和其他基本UNIX世界的大部头比起来，这本书其实还是比较小众的。我读大二的时候这本书已经出版了差不多22年(中文版也已经有7年了)，有一些内容已经过时了，比如没有返回值的main函数、外置的参数列表等等，不过在学习到HOC(High Order Calculator)的全部开发过程时，我依然被深深的震撼到了。

简而言之，这个HOC语言的开发过程需要这样几个组件：

词法分析器lex

语法分析器yacc

标准数学库stdlib

另外还有一些自定义的函数等，最后通过make连接在一起。我跟着书上的讲解，对着书把所有代码都敲了一遍。所有的操作都是在一台很老的IBM的ThinkPad T20上完成的，而且全部都在命令行中进行（当然，还在命令行里听着歌）。

这也是我第一次彻底被UNIX的哲学所折服的体验：

每个工具只做且做好一件事

工具可以协作起来

一切面向文本

下面是书中的Makefile脚本，通过简单的配置，就将一些各司其职的小工具协作起来，完成一个编程语言程序的预编译、编译、链接、二进制生成的动作。

YFLAGS = -d OBJS = hoc.o code.o init.o math.o symbol.o hoc5: $(OBJS) cc $(OBJS) -lm -o hoc5 hoc.o code.o init.o symbol.o: hoc.h code.o init.o symbol.o: x.tab.h x.tab.h: y.tab.h -cmp -s x.tab.h y.tab.h || cp y.tab.h x.tab.h pr: hoc.y hoc.h code.c init.c math.c symbol.c @pr $? @ touch pr clean: rm -f $(OBJS) [xy].tab.[ch]

虽然现在来看，这本书的很多内容已经过期（特别是离它第一次出版已经过去了近30年），有兴趣的朋友可以读一读。这里有一个Lex/Yacc的小例子的小例子，有兴趣的朋友可以看看。

当然，如果你使用现在最先进的IDE（典型的GUI工具），其背后做的事情也是同样的原理：生成一个Makefile，然后在幕后调用它。

基础设施自动化

开发过程中，工程师还需要关注的一个问题是：软件运行的环境。我在学生时代刚开始学习Linux的时候，会在Windows机器上装一个虚拟机软件VMWare，然后在VMWare中安装一个Redhat Linux 9。

（图片来自：http://t.cn/R6IBSAu）

这样当我不小心把Linux玩坏了之后，只需要重装一下就行了，不影响我的其他数据（比如课程作业、文档之类）。不过每次重装也挺麻烦，需要找到iso镜像文件，再挂载到本地的虚拟光驱上，然后再用VMWare来安装。

而且这些动作都是在GUI里完成的，每次都要做很多重复的事情：找镜像文件，使用虚拟光驱软件挂载，启动VMWare，安装Linux，配置个人偏好，配置用户名/密码等等。熟练之后，我可以在30 － 60分钟内安装和配置好一个新的环境。

Vagrant

后来我就发现了Vagrant，它支持开发者通过配置的方式将机器描述出来，然后通过命令行的方式来安装并启动，比如下面这个配置：

VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "precise64" config.vm.network "private_network", :ip => "192.168.2.100" end

它定义了一个虚拟机，使用Ubuntu Precise 64的镜像，然后为其配置一个网络地址192.168.2.100，定义好之后，我只需要执行：

$ vagrant up

我的机器就可以在几分钟内装好，因为这个动作是命令行里完成的，我可以在持续集成环境里做同样的事情 – 只需要一条命令。定义好的这个文件可以在团队内共享，可以放入版本管理，团队里的任何一个成员都可以在几分钟内得到一个和我一样的环境。

Ansible

一般，对于一个软件项目而言，一个全新的操作系统基本上没有任何用处。为了让应用跑起来，我们还需要很多东西。比如Web服务器、Java环境、cgi路径等，除了安装一些软件之外，还有大量的配置工作要做，比如apache httpd服务器的文档根路径，JAVA_HOME环境变量等等。

（图片来自：http://t.cn/R6IBZKm）

这些工作做好了，一个环境才算就绪。我记得在上一个项目上，不小心把测试环境的Tomcat目录给删除了，结果害的另外一位同事花了三四个小时才把环境恢复回来（包括重新安装Tomcat，配置一些JAVA_OPTS，应用的部署等）。

不过好在我们有很多工具可以帮助开发者完成环境的自动化准备，比如：Chef、 Puppet、Ansible。只需要一些简单的配置，然后结合一个命令行应用，整个过程就可以自动化起来了：

- name: setup custom repo apt: pkg=python-pycurl state=present - name: enable carbon copy: dest=/etc/default/graphite-carbon content='CARBON_CACHE_ENABLED=true' - name: install graphite and deps apt: name= state=present with_items: packages - name: install graphite and deps pip: name= state=present with_items: python_packages - name: setup apache copy: src=apache2-graphite.conf dest=/etc/apache2/sites-available/default notify: restart apache - name: configure wsgi file: path=/etc/apache2/wsgi state=directory

上边的配置描述了安装graphite-carbon、配置apahce等很多手工的劳动，开发者现在只需要执行：

$ ansible

就可以将整个过程自动化起来。现在如果我不小心把Tomcat删了，只需要几分钟就可以重新配置一个全新的，当然整个过程还是自动的。这在GUI下完全无法想象，特别是在有如此多的定制内容的场景下。

持续集成/持续发布

日常开发任务中，除了实际的编码和环境配置之外，另一大部分内容就是持续集成/持续发布了。借助于命令行，这个动作也可以非常高效和自动化。

Jenkins

持续集成/持续发布已经是很多企业IT的基本配置了。各个团队通过持续集成环境来编译代码、静态检查、执行单元测试、端到端测试、生成报告、打包、部署到测试环境等等。

比如在Jenkins环境中，在最前的版本中，要配置一个构建任务需要很多的GUI操作，不过在新版本中，大部分操作都已经可以写成脚本。

这样的方式，使得自动化变成了可能，要复制一个已有的pipline，或者要修改一些配置、命令、变量等等，再也不需要用鼠标点来点去了。而且这些代码可以纳入项目代码库中，和其他代码一起被管理、维护，变更历史也更容易追踪和回滚（在GUI上，特别是基于Web的，回滚操作基本上属于不可能）。

node { def mvnHome stage('Preparation') { // for display purposes git 'http://ift.tt/2jAK0Df' mvnHome = tool 'M3' } stage('Build') { sh "'${mvnHome}/bin/mvn' -Dmaven.test.failure.ignore clean package" } stage('Results') { junit '*/target/surefire-reports/TEST-.xml' archive 'target/*.jar' } }

上面这段groovy脚本定义了三个阶段，每个阶段中分别有自己的命令，这种以代码来控制的方式显然比GUI编辑的方式更加高效，自动化也变成了可能。

运维工作

自动化监控

Graphite是一个功能强大的监控工具，不过其背后的理念倒是很简单：

存储基于时间线的数据

将数据渲染成图，并定期刷新

用户只需要将数据按照一定格式定期发送给Graphite，剩下的事情就交给Graphite了，比如它可以消费这样的数据：

instance.prod.cpu.load 40 1484638635 instance.prod.cpu.load 35 1484638754 instance.prod.cpu.load 23 1484638812

第一个字段表示数据的名称，比如此处instance.prod.cpu.load表示prod实例的CPU负载，第二个字段表示数据的值，最后一个字段表示时间戳。

这样，Graphite就会将所有同一名称下的值按照时间顺序画成图。

（图片来自：http://t.cn/R6IxKYL）

默认地，Graphite会监听一个网络端口，用户通过网络将信息发送给这个端口，然后Graphite会将信息持久化起来，然后定期刷新。简而言之，只需要一条命令就可以做到发送数据：

echo "instance.prod.cpu.load 23 date +%s" | nc -q0 graphite.server 2003

date +%s会生成当前时间戳，然后通过echo命令将其拼成一个完整的字符串，比如：

instance.prod.cpu.load 23 1484638812

然后通过管道|将这个字符串通过网络发送给graphite.server这台机器的2003端口。这样数据就被记录在graphite.server上了。

定时任务

如果我们要自动的将数据每隔几秒就发送给graphite.server，只需要改造一下这行命令：

获取当前CPU的load

获取当前时间戳

拼成一个字符串

发送给graphite.server的2003端口

每隔5分钟做重复一下1-4

获取CPU的load在大多数系统中都很容易：

ps -A -o %cpu

��里的参数:

-A表示统计所有当前进程

-o %cpu表示仅显示%cpu列的数值

这样可以得到每个进程占用CPU负载的数字：

%CPU 12.0 8.2 1.2 ...

下一步是将这些数字加起来。通过awk命令，可以很容易做到这一点：

$ awk '{s+=$1} END {print s}'

比如要计算1 2 3的和：

$ echo "1\n2\n3" | awk '{s+=$1} END {print s}' 6

通过管道可以讲两者连起来：

$ ps -A -o %cpu | awk '{s+=$1} END {print s}'

我们测试一下效果：

$ ps -A -o %cpu | awk '{s+=$1} END {print s}' 28.6

看来还不错，有个这个脚本，通过crontab来定期调用即可：

#!/bin/bash SERVER=graphite.server PORT=2003 LOAD=ps -A -o %cpu | awk '{s+=$1} END {print s}' echo "instance.prod.cpu.load ${LOAD} date +%s" | nc -q0 ${SERVER} ${PORT}

当然，如果使用Grafana等强调UI的工具，可以很容易的做的更加酷炫：

（图片来自：http://t.cn/R6IxsFu）

想想用GUI应用如何做到这些工作。

娱乐

命令行的MP3播放器

最早的时候，有一个叫做mpg123的命令行工具，用来播放MP3文件。不过这个工具是商用的，于是就有人写了一个工具，叫mpg321，基本上是mpg123的开源克隆。不过后来mpg123自己也开源了，这是后话不提。

将我的所有mp3文件的路径保存成一个文件，相当于我的歌单：

$ ls /Users/jtqiu/Music/*.mp3 > favorites.list $ cat favorites.list ... /Users/jtqiu/Music/Rolling In The Deep-Adele.mp3 /Users/jtqiu/Music/Wavin' Flag-K'Naan.mp3 /Users/jtqiu/Music/蓝莲花-许巍.mp3 ...

然后我将这个歌单交给mpg321去在后台播放：

$ mpg321 -q --list favorites.list & [1] 10268

这样我就可以一边写代码一边听音乐，如果听烦了，只需要将这个后台任务切换到前台fg，然后就可以关掉了：

$ fg [1] + 10268 running mpg321 -q --list favorites.list

小结

综上，优秀的程序员借助命令行的特性，可以成倍（有时候是跨越数量级的）提高工作效率，从而有更多的时间进行思考、学习新的技能，或者开发新的工具帮助某项工作的自动化。这也是优秀的程序员之所以优秀的原因。而面向手工的、原始的图形界面会拖慢这个过程，很多原本可以自动化起来的工作被淹没在“简单的GUI”之中。

（图片来自：http://ift.tt/2np23iq

最后补充一点，本文的关键在于强调优秀的程序员与命令行的关系，而不在GUI程序和命令行的优劣对比。GUI程序当然有其使用场景，比如做3D建模、GIS系统、设计师��创作、图文并茂的字处理软件、电影播放器、网页浏览器等等。

应该说，命令行和优秀的程序员之间更多是关联关系，而不是因果关系。在程序员日常的工作中，涉及到的更多的是一些需要命令行工具来做支持的场景。如果走极端，在不适合的场景中强行使用命令行，而置效率于不顾，则未免有点矫枉过正，南辕北辙了。

为什么优秀的程序员喜欢命令行？，首发于文章 - 伯乐在线。

0 notes

letterbead93-blog · 6 years ago

Text

Pentaho Data Integration: WebSpoon on AWS Elastic Beanstalk and adding EBS or EFS Storage Volumes

This is a long overdue artilce on Hiromu’s WebSpoon. Hiromu has done a fantastic work on WebSpoon - finally bringing the familiar Spoon Desktop UI to the web browser.

For completeness sake I took the liberty of copying Hiromu’s instructions on how the set up the intial AWS Elastic Beanstalk environment. My main focus here is to provide simple approaches on how to add persistant storage options to your WebSpoon setup, some of which are fairly manual approaches (which should be later on replaced by a dedicated automatic setup). The article is more aimed toward users which are new to AWS.

Note: Once finished, always remember to terminate your AWS environment to stop occuring costs.

Overview

This guide will give you an example of how to deploy webSpoon to the cloud with the following requirements.

Deploy webSpoon to the cloud

Deploy webSpoon with plugins

Auto-scale and load-balance

Initial Quick Setup

To easily satisfy the requirements, this example uses the Docker image with the plugins and deploys it to AWS Elastic Beanstalk.

These are the rough steps using the new Amazon Configuration Interface:

Head to the AWS Console and sign into the console. Choose Beanstalk from the main menu. Then click Get Started or Create New Application:

In the Choose Environment dialog pick Web server environment.

In the Platform section, choose Multi-container Docker as a Preconfigure platform.

While we have a Dockerfile to provision our machine(s), we still need a method of orchestrating the setup of our cluster. This is where the Dockerrun.aws.json file comes in. In the Application code section, tick Upload your code and choose Dockerrun.aws.json as an Application code - contents copied below for convenience:

"AWSEBDockerrunVersion": 2, "containerDefinitions": [ "name": "webSpoon", "image": "hiromuhota/webspoon:latest-full", "essential": true, "memory": 1920, "environment": [ "name": "JAVA_OPTS", "value": "-Xms1024m -Xmx1920m" ], "portMappings": [ "hostPort": 80, "containerPort": 8080 ] ]

Click Configure more options instead of Create application.

On the next screen under Configuration presets you have two options. If you just want to run the container on one node choose Custom Configuration, otherwise choose High Availability.

On the same page, change EC2 instance type from t2.micro to t2.small or an instance type with 2GB+ memory. Click Save.

Optional: If you plan to ssh into the EC2 instance, edit the Security settings: Define your key pair.

Click Create Environment. Wait until your application and environment are created, which will take a few minutes:

High Availability option only: The screen will look like this one the environment is ready. Click on Configuration in side panel.

High Availability option only: Scroll down to find the Load Balancing section and click on the wheel icon:

High Availability option only: On the following screen enable session stickiness:

You can access the Spoon Web Interface via this URL:

http://<your-beanstalk-app-url>.com:8080/spoon/spoon

Your Beanstalk App URL is shown on the AWS Beanstalk application overview page.

Adding Volumes

The main aim of adding volumes is to persist the data outside of the Docker Container. We will have a look at various options:

Adding an EC2 Instance Docker Volume

Sources:

Note: Choosing this option does not really provide much benefit: We only map Docker container folders to local folders on the EC2 instance. So if you were to terminate your Beanstalk environment, the files would be gone as well. The main benefit here is that if the Docker Container gets terminated, the file at least survice on the EC2 instance.

Create a new project folder called beanstalk-with-ec2-instance-mapping.

Inside this folder create a Dockerrun.aws.json with following content:

"AWSEBDockerrunVersion": 2, "volumes": [ "name": "kettle", "host": "sourcePath": "/var/app/current/kettle" , "name": "pentaho", "host": "sourcePath": "/var/app/current/pentaho" ], "containerDefinitions": [ "name": "webSpoon", "image": "hiromuhota/webspoon:latest-full", "essential": true, "memory": 1920, "environment": [ "name": "JAVA_OPTS", "value": "-Xms1024m -Xmx1920m" ], "portMappings": [ "hostPort": 80, "containerPort": 8080 ], "mountPoints": [ "sourceVolume": "kettle", "containerPath": "/root/.kettle", "readOnly": false , "sourceVolume": "pentaho", "containerPath": "/root/.pentaho", "readOnly": false , "sourceVolume": "awseb-logs-webSpoon", "containerPath": "/usr/local/tomcat/logs" ] ]

First we create two volumes on the EC2 instance using the top level volumes JSON node: one for the .kettle files and one for the .pentaho files. Note that the sourcePath is the path on the host instance.

Note: This defines volumes on the hard drive of the EC2 instance you run your Docker container on. This is pretty much the same as if you were defining volumes on your laptop for a Docker container that you run. This does not set up magically any new EBS or EFS volumes.

Next, within the containerDefinitions for webSpoon, we add three mountPoints within the Docker Container. Here we map the container paths to the volumes we created earlier on (kettle and pentaho). The third mount point we define is for writing the logs out: This is a default requirement of the Beanstalk setup. For each container Beanstalk will create automatically a volume to store the logs. The volume name is made up of awseb-logs- plus the container name: In our case, this is: awseb-logs-webSpoon. And the logs we want to store are the Tomcat server logs.

The Beanstalk environment setup procedure is exactly the same as before, so go ahead and set up the environment.

Note: On the EC2 instance the directory /var/app/current/ is where the app files get stored (in our case this is only Dockerrun.aws.json). This folder does not required sudo privileges. If you ran the Docker container on your laptop you might have noticed that by default Docker stored name volumes in /var/lib/docker/volumes. On the EC2 instance this directory requires sudo privileges.

Once the environment is running, we can ssh into the EC2 instance.

Note: You can find the public DNS of your EC2 instance via the EC2 panel of the AWS console.

It is beyond the scope of this article to explain how to set up the required key pairs to ssh into an EC2 instance: Here is a good article describing the required steps. If you want to ssh into your instance, read this first. You also have to make sure that your Beanstalk environment knows which key you want to use. You can configure this via the main Configuration Panel under Security. This will restart the EC2 instance.

ssh -i <path-to-pem-file> ec2-user@<ec2-instance-public-dns>

We can double check now that the volume directories got created:

$ ls -l /var/app/current/ total 12 -rw-r--r-- 1 root root 1087 Dec 26 09:41 Dockerrun.aws.json drwxr-xr-x 2 root root 4096 Dec 26 09:42 kettle drwxr-xr-x 3 root root 4096 Dec 26 09:43 pentaho

Adding an EBS Volume

Sources:

Note: An EBS Drive is a device that will be mounted directly to your EC2 instance. It cannot be shared with any other EC2 instance. In other words, every EC2 instance will have their own (possibly set of) EBS Drive(s). This means that files cannot be shared across EC2 instances.

Two ESB Drives to Two Docker Volumes

For the next steps to work the volume mapping from Docker container to the EC2 instance has to be in place (as discussed in the previous section). We cover this below again.

Basically we have to create two layers of volume mapping:

Docker Container to EC2 instance

EC2 instance to EBS

There is no way to define an EBS volume in the Dockerrun.aws.json file: You have to create another file with a .config extension, which has to reside in the .ebextensions folder within your project’s folder. So the project’s folder structure should be like this:

. ├── Dockerrun.aws.json └── .ebextensions └── options.config

If you are not familiar with how mounting drivers on Linux works, read this article first.

Important: Configuration files must conform to YAML formatting requirements. Always use spaces to indent and don’t use the same key twice in the same file.

On the EC2 instance we will create a mount point under a new /data directory, which has less chance to interfere with any other process.

Let’s get started:

Create a new project folder called beanstalk-with-ebs-two-volumes.

Inside this folder create a Dockerrun.aws.json with following content:

"AWSEBDockerrunVersion": 2, "volumes": [ "name": "kettle", "host": "sourcePath": "/data/kettle" , "name": "pentaho", "host": "sourcePath": "/data/pentaho" ], "containerDefinitions": [ "name": "webSpoon", "image": "hiromuhota/webspoon:latest-full", "essential": true, "memory": 1920, "environment": [ "name": "JAVA_OPTS", "value": "-Xms1024m -Xmx1920m" ], "portMappings": [ "hostPort": 80, "containerPort": 8080 ], "mountPoints": [ "sourceVolume": "kettle", "containerPath": "/root/.kettle", "readOnly": false , "sourceVolume": "pentaho", "containerPath": "/root/.pentaho", "readOnly": false , "sourceVolume": "awseb-logs-webSpoon", "containerPath": "/usr/local/tomcat/logs" ] ]

Inside your project directory (beanstalk-with-ebs-two-volumes), create a subdirectory called .ebextensions.

In the .ebextensions directory create a new file called options.config and populate it with this content:

commands: 01mkfs: command: "mkfs -t ext3 /dev/sdh" 02mkdir: command: "mkdir -p /data/kettle" 03mount: command: "mount /dev/sdh /data/kettle" 04mkfs: command: "mkfs -t ext3 /dev/sdi" 05mkdir: command: "mkdir -p /data/pentaho" 06mount: command: "mount /dev/sdi /data/pentaho" option_settings: - namespace: aws:autoscaling:launchconfiguration option_name: BlockDeviceMappings value: /dev/sdh=:1,/dev/sdi=:1

These instructions basically format our two external volumes and then mount them. Note that at the very end under option_settings we specified that each EBS volume should be 1 GB big (which is very likely quite a bit too much for the pentaho volume, however, this is the minimum you can define).

Now we have to zip our files from within the project root directory:

[dsteiner@localhost beanstalk-with-ebs-two-volumes]$ zip ../webspoon-with-ebs.zip -r * .[^.]* adding: Dockerrun.aws.json (deflated 65%) adding: .ebextensions/ (stored 0%) adding: .ebextensions/options.config (deflated 43%)

Note: The zip file will be conveniently placed outside the project directory.

Next via the Web UI create a new Beanstalk environment. The approach is the same as before, just that instead of the Dockerrun.aws.json you have to upload the zip file now.

Important: You have to create the new Beanstalk environment in exactly the same Availability Zone within your Region as your EBS Drive resides in! Otherwise you can’t connect it! You can define the Availability Zone in the Capacity settings on the Configure env name page.

Once the environment is running, ssh into the EC2 instance.

Note: You can find the public DNS of your EC2 instance via the EC2 panel of the AWS console.

ssh -i <path-to-pem-file> ec2-user@<ec2-instance-public-dns>

We can check the mount points now:

$ ls -l /data total 8 drwxr-xr-x 3 root root 4096 Dec 26 16:12 kettle drwxr-xr-x 4 root root 4096 Dec 26 16:11 pentaho $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdh 202:112 0 1G 0 disk /data/kettle xvdi 202:128 0 1G 0 disk /data/pentaho

As you can see, now everything looks fine.

Once you have the new configuration running, you might want to check if the new volumes got created: You can do this by going to the EC2 section of the AWS console. On the side panel under Elastic Block Storage click on Volumes:

Note: Elastic Beanstalk will by default create also a volume /dev/xvdcz to store the Docker image.

One ESB Drive to Two Docker Volumes

Actually, what we did was a bit too complex (but might be required in some scenarios): We could have simply just mapped the /root folder of the Docker container to the /data folder on the EC2 instance and created a mount point /data that links the EC2 instance directory to the EBS volume. This way all the data is contained in one drive. Well, as it turns out, this is actually not a good idea, as we get loads of other files/folders as well:

[ec2-user@ip-xxx-xx-xx-xx ~]$ ls -la /data total 40 drwxr-xr-x 7 root root 4096 Dec 26 21:36 . drwxr-xr-x 26 root root 4096 Dec 26 21:24 .. drwxr-x--- 3 root root 4096 Dec 26 21:36 .java drwxr-x--- 3 root root 4096 Dec 26 21:37 .kettle drwx------ 2 root root 16384 Dec 26 21:24 lost+found drwxr-x--- 3 root root 4096 Dec 26 21:26 .m2 drwxr-x--- 3 root root 4096 Dec 26 21:36 .pentaho

Ok, so instead of this, we can leave the original Docker container to EC2 instance volume mapping in place:

Docker container path EC2 instance volume path /root/.kettle /data/kettle /root/.pentaho /data/pentaho

And just use one ESB volume, which we mount to /data.

Create a new project directory called beanstalk-with-ebs-one-volume.

Add a new Dockerrun.aws.json file to this folder, which looks like this (it’s exactly the same as when we added the volumes originally):

Inside your project directory (beanstalk-with-ebs-one-volume), create a subdirectory called .ebextensions. In the .ebextensions directory create a new file called options.config and populate it with this content:

commands: 01mkfs: command: "mkfs -t ext3 /dev/sdh" 02mkdir: command: "mkdir -p /data" 03mount: command: "mount /dev/sdh /data" option_settings: - namespace: aws:autoscaling:launchconfiguration option_name: BlockDeviceMappings value: /dev/sdh=:1

Now we have to zip our files from within the project root directory:

[dsteiner@localhost beanstalk-with-ebs-one-volume]$ zip ../webspoon-with-ebs.zip -r * .[^.]* adding: Dockerrun.aws.json (deflated 65%) adding: .ebextensions/ (stored 0%) adding: .ebextensions/options.config (deflated 43%)

Note: The zip file will be conveniently placed outside the project directory.

Next via the Web UI create a new Beanstalk environment. The approach is the same as before, just that instead of the Dockerrun.aws.json you have to upload the zip file now.

When we shh into our EC2 instance, we can see:

[ec2-user@ip-172-31-14-201 ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdh 202:112 0 1G 0 disk /data [ec2-user@ip-172-31-14-201 ~]$ ls -l /data/ total 24 drwxr-xr-x 3 root root 4096 Dec 26 22:33 kettle drwx------ 2 root root 16384 Dec 26 22:05 lost+found drwxr-xr-x 3 root root 4096 Dec 26 22:33 pentaho

As you can see, our /data directory looks way tidier now.

Avoid Conflicts

If we had specified /var/app/current/kettle and /var/app/current/pentaho as mount points we would have run into problems. Everything specified in .ebextensions gets executed before anything in Dockerrun.aws.json. So this approach would have mounted the EBS volumes first under /var/app/current and then later on when Dockerrun.aws.json would have tried to deploy our project, it would have seen that the /var/app/current already exists. In this case it would have renamed it to /var/app/current.old and deployed the app to a fresh new /var/app/current directory.

You can see this when you run the lbslk command to check how the devices were mounted:

[ec2-user@ip-xxx-xx-xx-xx ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdh 202:112 0 1G 0 disk /var/app/current.old/kettle

Conclusion: That’s why we need a different mount point! We want to specify a custom location that does not interfere with any other process.

Really Persisting the Data

Note: Here we only discuss a simple manual approach. This is only sensible if you run a single EC2 node with WebSpoon on it. For a more complex setup with load balancer and auto-scaling an automatic solution should be put in place.

So what is the point of this exercise really? Why did we do this? Our main intention was to have some form of persistent storage. Still, if we were to terminate the Beanstalk environment now, all our EBS volumes would disappear as well! However, via the EC2 panel under Elastic Block Storage there is a way to detach the volume:

The normal Detach Volume command might not work, because the volume is still used by our EC2 instance. You can, however, choose the Force Detach Volume command, which should succeed. Wait until the state of the drive shows available.

Next terminate your current Beanstalk environment. Once it’s terminated, you will see that your EBS Drive is still around. Start a new Beanstalk environment (this time just use the Dockerrun.aws.json file from this section, not the whole zip file - we do not want to create an new EBS drive).

Next, on the EC2 page in the AWS Console go to the EBS Volumes, mark our old EBS drive and right click: Choose Attach Volume. In a pop-up window you will be able to define to which instance you want to attach the EBS drive to.

Once it is attached, grab the public URL of your EC2 Instance from the EC2 area of the AWS console (click on Instances in the side panel). Ssh into your EC2 instance and then run the following:

$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdf 202:80 0 1G 0 disk xvdcz 202:26368 0 12G 0 disk

Originally I asked the EBS drive to be named sdf, but due to OS specifics it ended up being called xvdf as we can see from running the lsblk command. Note that the last letter remains the same. Also, you can see that it doesn’t have a mount point yet. So now we want to mount the EBS drive to the /data directory:

$ sudo mount /dev/xvdf /data

Next you have to restart the Docker container so that the changes can be picked up.

$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 18dfb174b88a hiromuhota/webspoon:latest-full "catalina.sh run" 15 minutes ago Up 15 minutes 0.0.0.0:80->8080/tcp ecs-awseb-Webspoon-env-xxx 0761525dd370 amazon/amazon-ecs-agent:latest "/agent" 16 minutes ago Up 16 minutes ecs-agent $ sudo docker restart 18dfb174b88a

Note: If you create the EBS Driver upfront separate from the Beanstalk environment, when you later shutdown the environment, the EBS Driver does not get terminated.

Alternative Approach: Manually create EBS Drive

Note: This approach is only really sensible if you were to run one EC2 instance only. The beauty of the previous approach is that every EC2 instance spun up via the auto-scaling process will have exactly the same setup (so e.g. one EC2 instance with one EBS drive). So for the approach outlined below, you do not need a load balancer and also not auto-scaling.

If you want to go down the manual road, you can as well create the EBS Drive manually:

Go to the EC2 area within the AWS Console and click on Volumes under Elastic Blockstorage (in the side panel).

Click on the Create Volume button. Fill out the required settings and confirm to create the volume.

Next go to the Elastic Beanstalk area of the AWS Console. Start a new Beanstalk environment: Use the Dockerrun.aws.json file from the beanstalk-with-ebs-one-volume project (if you skipped the previous sections, the instructions for setting up the Beanstalk environment are quite at the beginning of this article). This time also change edit the Capacity settings on the Configure env name page. For the Availability Zone define the same zone as your EBS Drive resides in.

Once the environment is up and running, you can Attach the Volume. On the EC2 page in the AWS Console go to the EBS Volumes:

Mark our EBS drive and right click: Choose Attach Volume. In a dialog you will be able to define to which instance you want to attach the EBS drive to.

Once it is attached, grab the public URL of your EC2 Instance from the EC2 area of the AWS console (click on Instances in the side panel). Ssh into your EC2 instance and then run the following:

$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdf 202:80 0 1G 0 disk xvdcz 202:26368 0 12G 0 disk

Originally I asked the EBS drive to be name sdf, but due to OS specifics it ended up being called xvdf as we can see from running the lsblk command. Note that the last letter remains the same. Also, you can see that it doesn’t have a mount point yet.

Since it is a fresh new EBS drive, we have to format it first:

$ sudo mkfs -t ext3 /dev/xvdf

Next we want to mount the EBS drive to the /data directory:

$ sudo mount /dev/xvdf /data

Next you have to restart the Docker container so that the files can be picked up:

Once you terminate your environment, the EBS Drive will still be available, so you can later on easily attach it to a new EC2 instance. It behaves this way because you created the EBS Drive separately from the Beanstalk environment.

Making use of Snapshots

If you don’t want to use WebSpoon nor the EBS drive for some time, you can take a snapshot of the data on the EBS Drive and store the snapshot on S3. Then you can get rid of the EBS Drive. Whenever you decide to get WebSpoon running again, you can restore the data from the snapshot onto a EBS Drive and attach it to the EC2 instance that is running WebSpoon and all is back to normal again.

Adding EFS Volume

Note: An EFS Volume is a network file storage (available in all AWS regions that support it) that can be shared between many EC2 instances at the same time. Because it is a network storage, it will be a bit slower than an EBS Volume (which is a device directly attached to an EC2 instance). Another advantage of an EFS Volume is that it grows or shrinks automatically according to your storage needs.

“You can create an Amazon EFS volume as part of your environment, or mount an Amazon EFS volume that you created independently of Elastic Beanstalk.”

Important: Before you start make sure that EFS is available in your region! If not, change the region selector in the top right hand corner of the AWS console.

Manual Approach: Via Web UI and Command Line

Note: This approach is only really suitable for smaller setups.

Go to the EFS Console and click on Create file system button. Provide the relevant details in the wizard dialog. This is really quite easy. Your network drive should be ready in a matter of minutes.

Next you can either ssh into your EC2 instances and run a few commands to mount the EFS Drive or add an additional config file to the beanstalk config files (and there are a few other options available as well). We will go with the first option for now.

Follow these steps:

Create a new Beanstalk environment with the Dockerrun.aws.json file from the beanstalk-with-ebs-one-volume project.

In the EFS Console, expand the details on your EFS Volume and you will find a link on how to mount the volume to an EC2 instance. The steps below are mostly a copy and paste of these instructions.

Check which security group is assigned to your EFS Volume Mount targets. Chances are that it is the default security group. We have to add the same security group to our EC2 instance(s), so that the instance can access the EFS mount target.

Head over to the EC2 console and click on Instance, then right click on your currently running instance: Networking > Change Security Groups.

Tick the default security group and confirm changes.

Next copy the Public DNS of your EC2 instance, then from the command line, ssh into your instance.

Once logged in, let’s install the NFS client: sudo yum install -y nfs-utils

In our case, the mount directory already exists (/data), hence we can directly mount the NFS Volume like so:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-9c912d55.efs.eu-west-1.amazonaws.com:/ /data

If you run the ls command on the /data directory you will see that it is empty now. Restart the Docker Container now so that these changes can be picked up.

Via Config

Important: Make sure that the EC2 instance has the same Security group assigned as the EFS Volume, otherwise you won’t be able to connect the two!

Elastic Beanstalk provides the option to:

create a new EFS (using the storage-efs-createfilesystem.config - Example). Note that this volume will be deleted once the environment is terminated since it is defined within the configuration detailed of this particular environment.

mount an existing EFS (using the storage-efs-mountfilesystem.config - Example).

These options are not exclusive: So you can either create a new EFS Volume and mount it or mount an existing EFS volume (which you already created separately at some point in the past - the configuration details are not part of the Beanstalk configuration details, in which case the EFS Volume will still exist after the Beanstalk environment was terminated).

If your plan is to not have the EFS Volume running all times (and WebSpoon for that matter), you can take a backup of the drive and reinstate it later on.

The configuration files have to be stored in the .ebextensions directory in your source code.

Note: “EFS requires a VPC. You can use the default VPC or create a custom VPC to use. Use the VPC management console to determine the number of available AZs and the default subnet for each.”

Mount Existing EFS

We will deploy new artifacts one by one to make sure that everything is working correctly:

Create a new project folder called beanstalk-with-efs-mount-existing-volume.

Copy and paste the Dockerrun.aws.json file from the beanstalk-with-ebs-one-volume project into this directory.

Create a new Beanstalk environment with this Dockerrun.aws.json file.

Next go to the EFS Console and create a new EFS Volume. Once the volume is ready, expand the details on your EFS Volume and copy the ID of Volume.

Head over to the EC2 console and click on Instance, then right click on your currently running instance: Networking > Change Security Groups.

Tick the default security group and confirm changes.

Inside this your project directory (beanstalk-with-efs-mount-existing-volume), create a subdirectory called .ebextensions.

In the .ebextensions directory create a new file called storage-efs-mountfilesystem.config and populate it with this content:

option_settings: aws:elasticbeanstalk:application:environment: # Use variable in conjunction with storage-efs-createfilesystem.config or # replace with EFS volume ID of an existing EFS volume # FILE_SYSTEM_ID: '`"Ref" : "FileSystem"`' FILE_SYSTEM_ID: 'fs-5d6fd394' # Replace with the required mount directory MOUNT_DIRECTORY: '/data' ############################################## #### Do not modify values below this line #### ############################################## REGION: '`"Ref": "AWS::Region"`' packages: yum: nfs-utils: [] jq: [] commands: 01_mount: command: "/tmp/mount-efs.sh" files: "/tmp/mount-efs.sh": mode: "000755" content : | #!/bin/bash EFS_REGION=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.REGION') EFS_MOUNT_DIR=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MOUNT_DIRECTORY') EFS_FILE_SYSTEM_ID=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.FILE_SYSTEM_ID') echo "Mounting EFS filesystem $EFS_DNS_NAME to directory $EFS_MOUNT_DIR ..." echo 'Stopping NFS ID Mapper...' service rpcidmapd status &> /dev/nulljq if [ $? -ne 0 ] ; then echo 'rpc.idmapd is already stopped!' else service rpcidmapd stop if [ $? -ne 0 ] ; then echo 'ERROR: Failed to stop NFS ID Mapper!' exit 1 fi fi # our mount point already exists, hence commented this section out # echo 'Checking if EFS mount directory exists...' # if [ ! -d $EFS_MOUNT_DIR ]; then # echo "Creating directory $EFS_MOUNT_DIR ..." # mkdir -p $EFS_MOUNT_DIR # if [ $? -ne 0 ]; then # echo 'ERROR: Directory creation failed!' # exit 1 # fi # else # echo "Directory $EFS_MOUNT_DIR already exists!" # fi mountpoint -q $EFS_MOUNT_DIR if [ $? -ne 0 ]; then echo "mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_FILE_SYSTEM_ID.efs.$EFS_REGION.amazonaws.com:/ $EFS_MOUNT_DIR" mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_FILE_SYSTEM_ID.efs.$EFS_REGION.amazonaws.com:/ $EFS_MOUNT_DIR if [ $? -ne 0 ] ; then echo 'ERROR: Mount command failed!' exit 1 fi chmod 777 $EFS_MOUNT_DIR runuser -l ec2-user -c "touch $EFS_MOUNT_DIR/it_works" if [[ $? -ne 0 ]]; then echo 'ERROR: Permission Error!' exit 1 else runuser -l ec2-user -c "rm -f $EFS_MOUNT_DIR/it_works" fi else echo "Directory $EFS_MOUNT_DIR is already a valid mountpoint!" fi echo 'EFS mount complete.'

Replace the FILE_SYSTEM_ID with the id of your EFS Volume Id. The config file I use here is fairly similar to this example, just that I commented out the section that creates the mount point, since this one already exists in our case: Remember that this directory gets created already by the instructions in our Dockerrun.aws.json (where we ask it to create the Docker volumes).

Next zip the two files up as we did before.

Go to the Dashboard page of your Beanstalk app and click on Upload and Deploy.

Add the zip file we just created and provide a version number. Confirm changes. It will attempt to deploy the new configuration: The good things now is that if this fails, it will automatically revert to the previous version, so at no point your are left with a broken environment.

Once the deployment succeeds, ssh into the EC2 instance and verify that the mounting was successful:

[ec2-user@ip-xxx-xx-xx-xxx ~]$ mount -t nfs4 xxxxxx.efs.eu-west-1.amazonaws.com:/ on /data type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx-xx-xx-xxx,local_lock=none,addr=xxx-xx-xx-xxx)

Create EFS and Mount It

This time we will create and mount the EFS volume via config settings.

The AWS Guide recommends deploying the storage-efs-createfilesystem.config with the source code code first (without any other changes), make sure that this deployment succeeds and only deploy everything with storage-efs-mountfilesystem.config. “Doing this in two separate deployments ensures that if the mount operation fails, the file system is left intact. If you do both in the same deployment, an issue with either step will cause the file system to terminate when the deployment fails.”

We actually have to do this in three major steps:

Create a new project folder called beanstalk-with-efs-create-and-mount-existing-volume.

Copy and paste the Dockerrun.aws.json file from the beanstalk-with-ebs-one-volume project into this directory.

Create a Beanstalk environment only with the Dockerrun.aws.json file.

Inside the your project directory (beanstalk-with-ebs-two-volumes), create a subdirectory called .ebextensions.

In the .ebextensions directory create a new file called storage-efs-createfilesystem.config with following content:

option_settings: - namespace: aws:elasticbeanstalk:customoption option_name: VPCId value: "vpc-9acf2ffc" - namespace: aws:elasticbeanstalk:customoption option_name: SubnetEUWest1a value: "subnet-9e9d92f9" - namespace: aws:elasticbeanstalk:customoption option_name: SubnetEUWest1b value: "subnet-b02536f9" - namespace: aws:elasticbeanstalk:customoption option_name: SubnetEUWest1c value: "subnet-d74e058c" Resources: FileSystem: Type: AWS::EFS::FileSystem Properties: FileSystemTags: - Key: Name Value: "EB-EFS-FileSystem" PerformanceMode: "generalPurpose" Encrypted: "false" ## Mount Target Resources # specify one mount target by availability zone MountTargetA: Type: AWS::EFS::MountTarget Properties: FileSystemId: Ref: FileSystem SecurityGroups: - Ref: MountTargetSecurityGroup SubnetId: Fn::GetOptionSetting: OptionName: SubnetEUWest1a MountTargetB: Type: AWS::EFS::MountTarget Properties: FileSystemId: Ref: FileSystem SecurityGroups: - Ref: MountTargetSecurityGroup SubnetId: Fn::GetOptionSetting: OptionName: SubnetEUWest1b MountTargetC: Type: AWS::EFS::MountTarget Properties: FileSystemId: Ref: FileSystem SecurityGroups: - Ref: MountTargetSecurityGroup SubnetId: Fn::GetOptionSetting: OptionName: SubnetEUWest1c ############################################## #### Do not modify values below this line #### ############################################## MountTargetSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for mount target SecurityGroupIngress: - FromPort: '2049' IpProtocol: tcp SourceSecurityGroupId: Fn::GetAtt: [AWSEBSecurityGroup, GroupId] ToPort: '2049' VpcId: Fn::GetOptionSetting: OptionName: VPCId

Note: There are various syntax options available: shorthand and longform expressions. To understand how they work, take a look at Option Settings. Also, validate the YAML file with one of the available online services, e.g. Code Beautify.

Once Beanstalk environment is created, the dedicated VPC is set up as well. Go to the VPN management console and get the VPC ID. Paste it into the relevant section within your storage-efs-createfilesystem.config file.

Next get the Subnet IDs. Paste them into the relevant section within your storage-efs-createfilesystem.config file.

Next zip up the two files as we did before.

Go to the Dashboard page of your Beanstalk app and click on Upload and Deploy.

Next add the storage-efs-mountfilesystem.config from the beanstalk-with-efs-mount-existing-volume project. We just have to make a small modification: Replace the hardcoded FILE_SYSTEM_ID with a variable:

option_settings: aws:elasticbeanstalk:application:environment: # Use variable in conjunction with storage-efs-createfilesystem.config or # replace with EFS volume ID of an existing EFS volume FILE_SYSTEM_ID: '`"Ref" : "FileSystem"`' # FILE_SYSTEM_ID: 'fs-5d6fd394' # Replace with the required mount directory MOUNT_DIRECTORY: '/data' ############################################## #### Do not modify values below this line #### ############################################## REGION: '`"Ref": "AWS::Region"`' packages: yum: nfs-utils: [] jq: [] commands: 01_mount: command: "/tmp/mount-efs.sh" files: "/tmp/mount-efs.sh": mode: "000755" content : | #!/bin/bash EFS_REGION=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.REGION') EFS_MOUNT_DIR=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MOUNT_DIRECTORY') EFS_FILE_SYSTEM_ID=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.FILE_SYSTEM_ID') echo "Mounting EFS filesystem $EFS_DNS_NAME to directory $EFS_MOUNT_DIR ..." echo 'Stopping NFS ID Mapper...' service rpcidmapd status &> /dev/nulljq if [ $? -ne 0 ] ; then echo 'rpc.idmapd is already stopped!' else service rpcidmapd stop if [ $? -ne 0 ] ; then echo 'ERROR: Failed to stop NFS ID Mapper!' exit 1 fi fi # our mount point already exists, hence commented this section out # echo 'Checking if EFS mount directory exists...' # if [ ! -d $EFS_MOUNT_DIR ]; then # echo "Creating directory $EFS_MOUNT_DIR ..." # mkdir -p $EFS_MOUNT_DIR # if [ $? -ne 0 ]; then # echo 'ERROR: Directory creation failed!' # exit 1 # fi # else # echo "Directory $EFS_MOUNT_DIR already exists!" # fi mountpoint -q $EFS_MOUNT_DIR if [ $? -ne 0 ]; then echo "mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_FILE_SYSTEM_ID.efs.$EFS_REGION.amazonaws.com:/ $EFS_MOUNT_DIR" mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_FILE_SYSTEM_ID.efs.$EFS_REGION.amazonaws.com:/ $EFS_MOUNT_DIR if [ $? -ne 0 ] ; then echo 'ERROR: Mount command failed!' exit 1 fi chmod 777 $EFS_MOUNT_DIR runuser -l ec2-user -c "touch $EFS_MOUNT_DIR/it_works" if [[ $? -ne 0 ]]; then echo 'ERROR: Permission Error!' exit 1 else runuser -l ec2-user -c "rm -f $EFS_MOUNT_DIR/it_works" fi else echo "Directory $EFS_MOUNT_DIR is already a valid mountpoint!" fi echo 'EFS mount complete.'

Zip everything up. Deploy this new archive to your Beanstalk environment.

After successful deployment, you can ssh into the EC2 instance and double check that the EFS volume is mounted:

[ec2-user@ip-xxx-xx-xx-xxx ~]$ mount -t nfs4 fs-cb48f402.efs.eu-west-1.amazonaws.com:/ on /data type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xxx-xx-xx-xxx,local_lock=none,addr=xxx-xx-xx-xxx)

Pentaho Server

You could also use Elastic Beanstalk to spin up a Pentaho Server with the Jackrabbit Repository and use the latter to store you files.

Source: http://diethardsteiner.github.io/pdi/2017/12/30/Webspoon-on-AWS.html

0 notes

musiccosmosru · 7 years ago

Link

Lately there have been a whole lot of changes to Java and its vibrant communities. Now shared between Oracle for Java SE and Eclipse Foundation for Jakarta EE (formerly Java EE), Java continues to be the leading programming language by developers and enterprises. As a matter of fact, it is now well-positioned to thrive in the cloud considering how modern application development is trending with over 12 million Java developers worldwide, and digital transformation being top of mind for many IT organizations.

With the sheer volume of Java apps in existence and soon to be developed, Java developers will benefit greatly from cloud services that will enable fast and secure application development while saving time and cost. Couple this with a vast geographic region coverage, it is a cloud solution every developer should experience.

Today, Microsoft is pleased to announce that Azure App Service now supports Java SE 8 based applications on Linux, now available in public preview. This and subsequent time released versions will be supported for an extended period, as well as upcoming LTS versions. Java web apps can now be built and deployed on a highly scalable, self-patching web hosting service where bug fixes and security updates will be maintained by Microsoft. Additional performance features include scaling to support millions of users with multiple instances, applications, and regions in a dynamic scaling intelligent configuration.

Java Web Apps benefits

Let Azure App Service do all the heavy lifting commonly associated with enterprise-grade infrastructure so developers can focus on productivity. Take advantage of Microsoft Azure’s battle tested infrastructure platform spanning from global security compliance to DevOps capabilities. Developer productivity benefits do not stop there. Java web apps provide the following benefits:

Fully managed enterprise platform – Log aggregation, email notifications, and Azure portal alerts. Version updates will soon include auto-patching.

Performance monitoring – Integrate with the Application Performance Management (APM) product of your choice to monitor and manage applications.

Global scale with high availability – 99.95% uptime with low latency, auto-scaling, or manual-scaling (up or out), anywhere in Microsoft’s global datacenters.

Security and compliance – App Service is ISO, SOC, PCI, and GDPR compliant.

Authentication and authorization – Built-in authentication with Azure Active Directory, governance with Roll-Based Access Control (RBAC) to manage IP address restrictions.

Build automation and CI/CD Support – Maven, Jenkins, and Visual Studio Team Services support will be available in the general availability release.

There are three ways of deploying Java Web Apps on Azure. You can create it from the Azure portal, use a template, or create and deploy from Maven. In this post, we will cover how to deploy a Spring Boot app using Maven.

Get started with Maven and Spring

To get started, clone your favorite Java Web app or use this sample: bash-3.2$ git clone

Add the Maven plugin for Azure Web Apps to the app project POM file and set server port to 80.

<build> <plugins> <plugin> <groupId>com.microsoft.azure</groupId> <artifactId>azure-webapp-maven-plugin</artifactId> <version>1.2.0</version> <configuration>  <resourceGroup>${RESOURCE_GROUP}</resourceGroup> <appName>${WEBAPP_NAME}</appName> <region>${REGION}</region> <pricingTier>S1</pricingTier>  <linuxRuntime>jre8</linuxRuntime> <deploymentType>ftp</deploymentType>  <resources> <resource> <directory>${project.basedir}/target</directory> <targetPath>/</targetPath> <includes> <include>app.jar</include> </includes> </resource> </resources> <appSettings> <property> <name>JAVA_OPTS</name> <value>-Djava.security.egd=file:/dev/./urandom</value> </property> </appSettings> </configuration> </plugin> </plugins> <finalName>app</finalName> </build>

Build, package, and deploy using Maven – like you would normally do.

bash-3.2$ mvn package azure-webapp:deploy [INFO] Scanning for projects... [INFO] [INFO] ---------------------------------------------------------------------- [INFO] Building petclinic 2.0.0 [INFO] ---------------------------------------------------------------------- [INFO] ... ... [INFO] --- azure-webapp-maven-plugin:1.2.0:deploy (default-cli) @ spring-petclinic --- [INFO] Start deploying to Web App myjavase-07262018aa... [INFO] Authenticate with Azure CLI 2.0 [INFO] Target Web App doesn't exist. Creating a new one... [INFO] Creating App Service Plan 'ServicePlan1af9c8f0-3f71-43a8'... [INFO] Successfully created App Service Plan. [INFO] Successfully created Web App. ... ... [INFO] Finished uploading directory: /Users/selvasingh/GitHub/selvasingh/spring-petclinic/target/azure-webapps/myjavase-07262018aa --> /site/wwwroot [INFO] Successfully uploaded files to FTP server: waws-prod-bay-081.ftp.azurewebsites.windows.net [INFO] Starting Web App after deploying artifacts... [INFO] Successfully started Web App. [INFO] Successfully deployed Web App at https://myjavase-07262018aa.azurewebsites.net [INFO] ---------------------------------------------------------------------- [INFO] BUILD SUCCESS [INFO] ---------------------------------------------------------------------- [INFO] Total time: 03:06 min [INFO] Finished at: 2018-07-13T18:03:22-07:00 [INFO] Final Memory: 139M/987M [INFO] ----------------------------------------------------------------------

Open the site in your favorite browser.

And here it is, your first Java web app on Azure App Service. Nice job!

More friendly Java tools on the way

We understand that development tools are not one size fit all. We are making sure we continue to enrich our tool suite with tools that enhance productivity and efficiency. In the meantime, expect to see support for Gradle and Jenkins next.

Next steps

Running Java Web Apps in the cloud cannot be any easier and faster. Give it try, create your first Java web app in Azure Service. Below are a few resources to help you get started.

If you don’t have an Azure subscription, create a free account today.

<a style="display:none" rel="follow" href="http://megatheme.ir/" title="قالب وردپرس">قالب وردپرس</a>

The post Azure App Service now supports Java SE on Linux | Blog appeared first on MusicCosmoS.

#beauty #healths #kids #royal espanol

0 notes

enterinit · 7 years ago

Text

Windows Server 2019 Insider Preview Build 17677 released

Windows Server 2019 Insider Preview Build 17677 released. Performance history for Storage Spaces Direct In this build, the performance history feature of Storage Spaces Direct gets even better: The Get-ClusterPerf cmdlet now includes self-diagnosis logic: if the cmdlet finds nothing to report, it now looks for common issues that would prevent performance history from working properly (for example, if its storage is missing) so that the cmdlet can provide clear error text. New cmdlets, Start-ClusterPerformanceHistory and Stop-ClusterPerformanceHistory, that are provided in this build make it easy to remediate such issues by cleaning up and/or re-provisioning performance history. New series, provided in this build, record how much Storage Spaces Direct data needs to repair/resync per server. Container Improvements In our continued effort to improve Windows Server Core and reduce its image size, we made the following changes: – we converted non-critical font components into optional components (OC) in Windows Server Core editions, and then removed these OCs from Windows Server Core container images. This change won’t affect the user experience of Windows Server Core, except that users now have the ability to enable or disable non-critical font components, like they can do for any other OC. For Server Core containers, only the default font, Arial, is supported; no other fonts are supported, and no others can be installed. Available Content Windows Server 2019 Preview is available in ISO format in 18 languages, and in VHDX format in English only. This build and all future pre-release builds will require use of activation keys during setup. The following keys allow for unlimited activations: Datacenter Edition 6XBNX-4JQGW-QX6QG-74P76-72V67 Standard Edition MFY9F-XBN2F-TYFMP-CCV49-RMYVH Windows Server vNext Semi-Annual Preview The Server Core Edition is available in English only, in ISO or VHDX format. The images are pre-keyed – no need to enter a key during setup. Symbols are available on the public symbol server – see Update on Microsoft’s Symbol Server blog post and Using the Microsoft Symbol Server. As before, matching Windows Server container images will be available via Docker Hub. This build will expire December 14th, 2018. How to Download To obtain the Insider software downloads, registered Insiders may navigate directly to the Windows Server Insider Preview download page. Known Issues In-place OS upgrade: Domain Controllers. Active Directory (AD) Domain Controllers (DC) might not be upgraded correctly by an in-place upgrade unless the NT Directory Service (NTDS) is stopped before initiating the upgrade. To ensure recoverability in the case of failure, back up any AD DC before performing an in-place OS upgrade. On recent preview releases of the operating system, the Failover Clustering feature might not install correctly, leaving the feature in “InstallPending” state. Drivers that are supplied with the operating system for network adapters from a specific manufacturer may fail to work on recent preview releases of the operating system. Creating or modifying environment variables by using setx on a system running Nano Server fails. On an affected system, setx requires a specific path in the registry, HKCU\Environment\, to exist by default. The Local Security Authority Subsystem Service (lsass.exe) may fail, causing the Active Directory server service to fail. When removal of a domain is attempted on an affected system, it reports an error: “The operation cannot continue because LDAP connect/bind operation failed.” The Virtual Machine Management Service (VMMS) may experience an error, APPLICATION_FAULT_INVALID_POINTER_READ, in Cluster Manager. When a system is running a recent preview release of the operating system, Windows Server Containers may experience an idle time of 240 milliseconds before Docker is started. A virtual machine that uses a Microsoft Virtual BIOS may fail to start and report an error: “Failed to restore with Error ‘Indicates two revision levels are incompatible.’ (0x8007051A)” Creation of a basic Linux system image in a Hyper-V container fails. On an affected system, Docker reports an error: “The command ‘/bin/sh -c /bin/echo echo test > test.txt’ returned a non-zero code.” Some third-party apps do not start when run in a Hyper-V container, possibly due to an incorrect character set. For affected apps, a workaround is to manually set the character set to UTF-8 in your JVM for the Container by running set JAVA_OPTS=-Dsun.jnu.encoding=UTF-8. Editing or creating policies for AppLocker can cause the MMC snap-in to crash when generated rules for a packaged app. After upgrading the operating system, the AppX database may have corrupted entries, which causes problems for components that use those entries. An administrator might notice this issue when generating packaged app rules or when managing an installed package, resulting in error messages such as “This wizard has encountered a fatal error and cannot continue” and “MMC has detected an error in a snap-in and will unload it.” Read the full article

#Windows #WindowsInsider #WindowsServer

0 notes

elisp-solves-problems · 8 years ago

Text

Connecting to Mongo with a self signed CA on a JVM in Kubernetes

At $WORK, we're creating an internal platform on top of Kubernetes for developers to deploy their apps. Our Ops people have graciously provided us with Mongo clusters that all use certificates signed by a self-signed certificate authority. So, all our clients need to know about the self-signed CA in order to connect to Mongo. For Node or Python, it's possible to pass the self-signed CA file in the code running in the application.

But, things are a little more complicated for Java or Scala apps, because configuration of certificate authorities is done at the JVM level, not at the code level. And for an extra level of fun, we want to do it in Kubernetes, transparently to our developers, so they don't have to worry about it on their own.

err, wha? telling the JVM about our CA

First off, we had to figure out how to tell the JVM to use our CA. And luckily since all the JVM languages use the same JVM, it's the same steps for Scala, or Clojure, or whatever other JVM language you prefer. The native MongoDB Java driver docs tell us exactly what we need to do: use keytool to import the cert into a keystore that the JVM wants, and then use system properties to tell the JVM to use that keystore. The keytool command in the docs is:

$ keytool -importcert -trustcacerts -file <path to certificate authority file> \ -keystore <path to trust store> -storepass <password>

The path to the existing keystore that the JVM uses by default is $JAVA_HOME/jre/lib/security/cacerts, and its default password is changeit. So if you wanted to add your self signed CA to the existing keystore, it'd be something like

$ keytool -importcert -trustcacerts -file ssca.cer \ -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit

(Even this very first step had complications. Our self signed CA was a Version 1 cert with v3 extensions, and while no other language cared, keytool refused to create a keystore with it. We ended up having to create a new self-signed CA with the appropriate version. Some lucky googling led us to that conclusion, but of particular use was using openssl to examine the CA and check its versions and extensions:)

$ openssl x509 -in ssca.cer -text -noout // Certificate: // Data: // Version: 3 (0x2) // Serial Number: ... // ... // X509v3 extensions: // X509v3 Subject Key Identifier: ... // X509v3 Key Usage: ... // X509v3 Basic Constraints: ...

Another useful command was examining the keystore before and after we imported our self signed CA:

$ keytool -list -keystore /path/to/keystore/file

as you can look for your self-signed CA in there to see if you ran the command correctly.

Anyway, once you've created a keystore for the JVM, the next step is to set the appropriate system properties, again as out lined in the docs:

$ java \ -Djavax.net.ssl.trustStore=/path/to/cacerts \ -Djavax.net.ssl.trustStorePassword=changeit \ -jar whatever.jar

Since the default password is changeit, you may want to change it... but if you don't change it, you wouldn't have to specify the trustStorePassword system property.

handling this in kubernetes

The above steps aren't too complicated on their own. We just need to make sure we add our CA to the existing ones, and point the JVM towards our new and improved file. But, since we'll eventually need to rotate the self-signed CA, we can't just run keytool once and copy it everywhere. So, an initContainer it is! keytool is a java utility, and it's handily available on the openjdk:8u121-alpine image, which means we can make a initContainer that runs keytool for us dynamically, as part of our Deployment.

Since seeing the entire manifest at once doesn't necessarily make it easy to see what's going on, I'm going to show the key bits piece by piece. All of the following chunks of yaml belong to in the spec.template.spec object of a Deployment or Statefulset.

spec: template: spec: volumes: - name: truststore emptyDir: {} - name: self-signed-ca secret: secretName: self-signed-ca

So, first things first, volumes: an empty volume called truststore which we'll put our new and improved keystore-with-our-ssca. Also, we'll need a volume for the self-signed CA itself. Our Ops provided it for us in a secret with a key ca.crt, but you can get it into your containers any way you want.

$ kubectl get secret self-signed-ca -o yaml --export apiVersion: v1 data: ca.crt: ... kind: Secret metadata: name: self-signed-ca type: Opaque

With the volumes in place, we need to set up init containers to do our keytool work. I assume (not actually sure) that we need to add our self-signed CA to the existing CAs, so we use one initContainer to copy the existing default cacerts file into our truststore volume, and another initContainer to run the keytool command. It's totally fine to combine these into one container, but I didn't feel like making a custom docker image with a shell script or having a super long command line. So:

spec: template: spec: initContainers: - name: copy image: openjdk:8u121-alpine command: [ cp, /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts, /ssca/truststore/cacerts ] volumeMounts: - name: truststore mountPath: /ssca/truststore - name: import image: openjdk:8u121-alpine command: [ keytool, -importcert, -v, -noprompt, -trustcacerts, -file, /ssca/ca/ca.crt, -keystore, /ssca/truststore/cacerts, -storepass, changeit ] volumeMounts: - name: truststore mountPath: /ssca/truststore - name: self-signed-ca mountPath: /ssca/ca

Mount the truststore volume in the copy initContainer, grab the file cacerts file, and put it in our truststore volume. Note that while we'd like to use $JAVA_HOME in the copy initContainer, I couldn't figure out how to use environment variables in the command. Also, since we're using a tagged docker image, there is a pretty good guarantee that the filepath shouldn't change underneath us, even though it's hardcoded.

Next, the import step! We need to mount the self-signed CA into this container as well. Run the keytool command as described above, referencing our copied cacerts file in our truststore volume and passing in our ssCA.

Two things to note here: the -noprompt argument to keytool is mandatory, or else keytool will prompt for interaction, but of course the initContainer isn't running in a shell for someone to hit yes in. Also, the mountPaths for these volumes should be separate folders! I know Kubernetes is happy to overwrite existing directories when a volume mountPath clashes with a directory on the image, and since we have different data in our volumes, they can't be in the same directory. (...probably, I didn't actually check)

The final step is telling the JVM where our new and improved trust store is. My first idea was just to add args to the manifest and set the system property in there, but if the Dockerfile ENTRYPOINT is something like

java -jar whatever.jar

then we'd get a command like

java -jar whatever.jar -Djavax.net.ssl.trustStore=...

which would pass the option to the jar instead of setting a system property. Plus, that wouldn't work at all if the ENTRYPOINT was a shell script or something that wasn't expecting arguments.

After some searching, StackOverflow taught us about the JAVA_OPTS and JAVA_TOOL_OPTIONS environment variables. We can append our trustStore to the existing value of these env vars, and we'd be good to go!

spec: template: spec: containers: - image: your-app-image env: # make sure not to overwrite this when composing the yaml - name: JAVA_OPTS value: -Djavax.net.ssl.trustStore=/ssca/truststore/cacerts volumeMounts: - name: truststore mountPath: /ssca/truststore

In our app that we use to construct the manifests, we check if the developer is already trying to set JAVA_OPTS to something, and make sure that we append to the existing value instead of overwriting it.

a conclusion of sorts

Uh, so that got kind of long, but the overall idea is more or less straightforward. Add our self-signed CA to the existing cacerts file, and tell the JVM to use it as the truststore. (Note that it's the trustStore option you want, not the keyStore!). The entire Deployment manifest all together is also available, if that sounds useful...

#kubernetes #mongo #jvm #ssca #self-signed #certificate-authority #self-signed-ca

0 notes