shell script to install spark 2.1.1

#! /bin/bash
echo download scala
wget http://downloads.typesafe.com/scala/2.12.2/scala-2.12.2.tgz
tar xvf scala-2.12.2.tgz
mv scala-2.12.2 /usr/lib
ln -s /usr/lib/scala-2.12.2 /usr/lib/scala
export PATH=$PATH:/usr/lib/scala/bin
scala -version

echo download spark
wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-hadoop2.7.tgz
tar xvf spark-2.1.1-bin-hadoop2.7.tgz

export SPARK_HOME=$HOME/spark-2.1.1-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin

firewall-cmd –permanent –zone=public –add-port=6066/tcp
firewall-cmd –permanent –zone=public –add-port=7077/tcp
firewall-cmd –permanent –zone=public –add-port=8080-8081/tcp
firewall-cmd –reload

echo ‘export PATH=$PATH:/usr/lib/scala/bin’ >> .bash_profile
echo ‘export SPARK_HOME=$HOME/spark-2.12.2-bin-hadoop2.7’ >> .bash_profile
echo ‘export PATH=$PATH:$SPARK_HOME/bin’ >> .bash_profile

How to kill results from ps & grep

from: https://unix.stackexchange.com/questions/30759/whats-a-good-example-of-piping-commands-together

Problem

Let’s say the command conky stopped responding on my desktop, and I want to kill it manually. I know a little bit of Unix, so I know that what I need to do is execute the command kill <PID>. In order to retrieve the PID, I can use ps or top or whatever tool my Unix distribution has given me. But how can I do this in one command?

Answer

$ ps aux | grep conky | grep -v grep | awk '{print $2}' | xargs kill

DISCLAIMER: This command only works in certain cases. Don’t copy/paste it in your terminal and start using it, it could kill processes unsuspectingly. Rather learn how to build it.

How it works

1- ps aux

This command will output the list of running processes and some info about them. The interesting info is that it’ll output the PID of each process in its 2nd column. Here’s an extract from the output of the command on my box:

$ ps aux
 rahmu     1925  0.0  0.1 129328  6112 ?        S    11:55   0:06 tint2
 rahmu     1931  0.0  0.3 154992 12108 ?        S    11:55   0:00 volumeicon
 rahmu     1933  0.1  0.2 134716  9460 ?        S    11:55   0:24 parcellite
 rahmu     1940  0.0  0.0  30416  3008 ?        S    11:55   0:10 xcompmgr -cC -t-5 -l-5 -r4.2 -o.55 -D6
 rahmu     1941  0.0  0.2 160336  8928 ?        Ss   11:55   0:00 xfce4-power-manager
 rahmu     1943  0.0  0.0  32792  1964 ?        S    11:55   0:00 /usr/lib/xfconf/xfconfd
 rahmu     1945  0.0  0.0  17584  1292 ?        S    11:55   0:00 /usr/lib/gamin/gam_server
 rahmu     1946  0.0  0.5 203016 19552 ?        S    11:55   0:00 python /usr/bin/system-config-printer-applet
 rahmu     1947  0.0  0.3 171840 12872 ?        S    11:55   0:00 nm-applet --sm-disable
 rahmu     1948  0.2  0.0 276000  3564 ?        Sl   11:55   0:38 conky -q

2- grep conky

I’m only interested in one process, so I use grep to find the entry corresponding to my program conky.

$ ps aux | grep conky
 rahmu     1948  0.2  0.0 276000  3564 ?        Sl   11:55   0:39 conky -q
 rahmu     3233  0.0  0.0   7592   840 pts/1    S+   16:55   0:00 grep conky

3- grep -v grep

As you can see in step 2, the command ps outputs the grep conky process in its list (it’s a running process after all). In order to filter it, I can run grep -v grep. The option -v tells grep to match all the lines excluding the ones containing the pattern.

$ ps aux | grep conky | grep -v grep
 rahmu     1948  0.2  0.0 276000  3564 ?        Sl   11:55   0:39 conky -q

NB: I would love to know a way to do steps 2 and 3 in a single grep call.

4- awk '{print $2}'

Now that I have isolated my target process. I want to retrieve its PID. In other words I want to retrieve the 2nd word of the output. Lucky for me, most (all?) modern unices will provide some version of awk, a scripting language that does wonders with tabular data. Our task becomes as easy as print $2.

$ ps aux | grep conky | grep -v grep | awk '{print $2}'
 1948

5- xargs kill

I have the PID. All I need is to pass it to kill. To do this, I will use xargs.

xargs kill will read from the input (in our case from the pipe), form a command consisting of kill <items> (<items> are whatever it read from the input), and then execute the command created. In our case it will execute kill 1948. Mission accomplished.

Create scheduled task in Spring Boot

Reference: https://spring.io/guides/gs/scheduling-tasks/

pom.xml:

<project xmlns=”http://maven.apache.org/POM/4.0.0&#8243; xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance&#8221;
xsi:schemaLocation=”http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd”&gt;
<modelVersion>4.0.0</modelVersion>

<groupId>com.sample</groupId>
<artifactId>gs-scheduling-tasks</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>

<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.2.RELEASE</version>
</parent>

<properties>
<java.version>1.8</java.version>
</properties>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

ScheduledTasks.java

package com.sample;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.text.SimpleDateFormat;
import java.time.LocalDate;
import java.util.Date;

/**
* Created by vagrant on 5/25/17.
*/
@Component
public class ScheduledTasks {

private static final Logger log = LoggerFactory.getLogger(ScheduledTasks.class);

private static final SimpleDateFormat dateFormat = new SimpleDateFormat(“HH:mm:ss”);

@Scheduled(fixedRate = 5000)
public void reportCurrentTime() {
log.info(“The time is now {}”, dateFormat.format(new Date()));
}
}

App.java

package com.sample;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;

/**
* Hello world!
*
*/
@SpringBootApplication
@EnableScheduling
public class App
{
public static void main( String[] args )
{
SpringApplication.run(App.class);
}
}

To run: mvn spring-boot:run

a script to install basic Java dev tools for Vagrant provision

#! /bin/bash
echo install jdk1.8
wget –no-cookies –no-check-certificate –header “Cookie: oraclelicense=accept-securebackup-cookie” “http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm&#8221;
yum localinstall jdk-8u131-linux-x64.rpm -y
touch /etc/profile.d/java.sh
cat <<EOT >> /etc/profile.d/java.sh
export JAVA_HOME=”/usr/java/jdk1.8.0_131″
export JAVA_PATH=”$JAVA_HOME”
export PATH=”$PATH:$JAVA_HOME”
EOT
rm jdk-8u131-linux-x64.rpm

echo install ant
wget http://www.us.apache.org/dist/ant/binaries/apache-ant-1.10.1-bin.tar.gz
tar xvfz apache-ant-1.10.1-bin.tar.gz -C /opt
ln -s /opt/apache-ant-1.10.1 /opt/ant
sh -c ‘echo ANT_HOME=/opt/ant>>/etc/environment’
ln -s /opt/ant/bin/ant /usr/bin/ant
rm apache-ant-1.10.1-bin.tar.gz

echo install python3.6
yum -y groupinstall development
yum -y install https://centos7.iuscommunity.org/ius-release.rpm
yum -y install python36u python36u-pip python36u-devel
# yum -y install python36u-pip
# yum -y install python36u-devel

echo install Ansible
yum -y install ansible

echo set basic vim style
touch /etc/vimrc
cat <<EOT >> /etc/vimrc
filetype plugin indent on
set tabstop=2
set shiftwidth=2
set expandtab
set softtabstop=2
set nowrap
set number
color blue
EOT

echo install maven
wget http://apache.mirror.gtcomm.net/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.tar.gz
tar xvf apache-maven-3.5.0-bin.tar.gz -C /usr/local
rm apache-maven-3.5.0-bin.tar.gz
ln -s /usr/local/apache-maven-3.5.0/ /usr/local/maven
touch /etc/profile.d/maven.sh
cat <<EOT >> /etc/profile.d/maven.sh
#! /bin/bash

export M2_HOME=/usr/local/maven
export PATH=$PATH:/usr/local/maven/bin
EOT

su -l vagrant -c “mkdir /home/vagrant/.m2”
touch /home/vagrant/.m2/settings.xml
cat <<EOT >> /home/vagrant/.m2/settings.xml
<settings>
<proxies>
<proxy>
<active>true</active>
<protocol>http</protocol>
<host>localhost</host>
<port>3128</port>
<nonProxyHosts>127.0.0.1</nonProxyHosts>
</proxy>
</proxies>
</settings>
EOT

chown vagrant /home/vagrant/.m2/settings.xml

echo install SDKMAN
yum install -y zip unzip
su -l vagrant -c ‘curl -s “https://get.sdkman.io&#8221; | bash’
su -l vagrant -c ‘source /home/vagrant/.sdkman/bin/sdkman-init.sh’

echo install Spring Boot
su -l vagrant -c ‘sdk install springboot’

echo install Intellij
sudo yum install -y xauth libXtst
wget http://download.jetbrains.com/idea/ideaIC-2017.1.3.tar.gz
tar xvf ideaIC-2017.1.3.tar.gz
mv idea-IC-171.4424.56 /usr/lib
ln -sf /usr/lib/idea-IC-171.4424.56 /usr/lib/idea
ln -sf /usr/lib/idea/bin/idea.sh /usr/bin/idea
sed -i ‘s/Xm\(s\|x\)[[:digit:]]*m/Xmx\12048m/g’ /usr/lib/idea/bin/idea.vmoptions /usr/lib/idea/bin/idea64.vmoptions
sed -i ‘s/ReservedCodeCacheSize=[[:digit:]]*m/ReservedCodeCacheSize=700m/g’ /usr/lib/idea/bin/idea.vmoptions /usr/lib/idea/bin/idea64.vmoptions
sed -i ‘4i -XX:MaxPermSize=512m’ /usr/lib/idea/bin/idea.vmoptions /usr/lib/idea/bin/idea64.vmoptions
cat <<EOT >> /etc
sysctl -p

cntlm on systemd

cntlm.services:

[Unit]
Description=cntlm service
After=syslog.target network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/cntlm.sh

[Install]
WantedBy=multi-user.target

cntlm.sh

#! /bin/bash
# script used by d$hhhhh.service

/usr/sbin/cntlm -c /etc/cntlm.conf

/etc/profile.d/cntlm.sh

export http_proxy=localhost:3128

export https_proxy=$http_proxy

export ftp_proxy=$http_proxy