Date manipulation with sed (Convert date to YYYY-MM-DD format)

In my previous post , I talked about a very useful script (with some useful functions) to manipulate dates.

For now, I’m writing a script for a customer, where I got to inject those manipulated dates in an SQL query.
The problem is that the script I’m talking in my previous post does not put the leading 0 when the day or the month is less than 10.

Example :

datecalc -a 1960 12 31 + 7
1961 1 7


In the sql query, the dates MUST be with format YYYY-MM-DD!
It’s pretty annoying when the dates returned are with format YYYY-M-D or YYYY-MM-D or YYYY-M-DD, isn’t it ?

As usual, I googled (Google knows all!) and found the easy answer!


#!/bin/sh
# by Jakukyo Friel under GPL v2.

sed -e 's/\(-\)\([0-9]-\)/-0\2/' | sed -e 's/\(-\)\([0-9]$\)/-0\2/'

Source

I’ve fought for hours with this line, cause it ain’t worked! The mistake was in the second sed substitution command, where the $ was included in the last range of characters (the last 2 digits representing the day), and it shouldn’t be there, but behind the mark!

Finally here is the correct version :


### This script convert date formats:
# YYYY-MM-DD => YYYY-MM-DD
# YYYY-M-DD => YYYY-MM-DD
# YYYY-MM-D => YYYY-MM-DD
# YYYY-M-D => YYYY-MM-DD

sed -e 's/\(-\)\([0-9]-\)/-0\2/' | sed -e 's/\(-\)\([0-9]\)$/-0\2/'


Let’s try the same command line :


echo "1967-1-7" | sed -e 's/\(-\)\([0-9]-\)/-0\2/' | sed -e 's/\(-\)\([0-9]\)$/-0\2/'
1967-01-07


YAY!!!

I’m angry at wasting so much time for a small misplaced $ sign. So I’m sharing with you all…
Unlike me, It may save you some time

Comments (0) - Posted in Shell script by wangee  

Date calculation in a unix shell script (ksh)

Hi!  

I had to implement date manipulation in an existing unix shell script (ksh).
The easiest way to add or substract days (or months, or years) from the current date (assuming that you are logged on a Linux host) is :

user@host:~$ date –date “now +3 days”
lundi 12 mars 2012, 09:55:17 (UTC+0100)

user@host:~$ date –date “now -3 days -4 month”
dimanche 6 novembre 2011, 09:58:20 (UTC+0100)

But this time I’m dealing with solaris 9 & 10 servers! “date –date” is not available.

So how to add X days or substract Y days from the current date ?

I’ve found this very useful script :

#! /usr/bin/ksh

# datecalc — Perderabo’s date calculator
#

USAGE=”\
datecalc -a year month day – year month day
datecalc -a year month day [-|+] n
datecalc -d year month day
datecalc -D year month day
datecalc -j year month day
datecalc -j n
datecalc -l year month
use \”datecalc -help\” use for more documentation”

DOCUMENTATION=”\
datecalc Version 1.1

datecalc does many manipulations with dates.
datecalc -a is for date arithmetic
datecalc -d or -D converts a date to the day of week
datecalc -j converts to date to or from julian day
datecalc -l outputs the last day of a month

All dates must be between the years 1860 and 3999.

datecalc -a followed by 7 parameters will calculate the
number of days between two dates. Parameters 2-4 and 6-8
must be dates in ymd form, and parameter 5 must be a minus
sign. The output is an integer. Example:

> datecalc -a 1960 12 31 – 1922 2 2
14212

datecalc -a followed by 5 parameters will calculate the
a new date offset from a given date, Parameters 2-4 must
be a date in ymd form, paramter 5 must be + or -, and
paramter 6 must be an integer. Output is a new date.
Example:

> datecalc -a 1960 12 31 + 7
1961 1 7

datecalc -d followed by 3 parameters will convert a date
to a day-of-week. Parameters 2-4 must be a date in ymd
form. Example:

> datecalc -d 1960 12 31
6

datecalc -D is like -d except it displays the name of
the day. Example:

> datecalc -D 1960 12 31
Saturday

datecalc -j followed by 3 parameters will convert a date
to Modified Julian Day number. Example:
> datecalc -j 1960 12 31
37299

datecalc -j followed by a single parameter will convert
a Modified Julian Day number to a date. Example:
> datecalc -j 37299
1960 12 31

datecalc -l followed by year and month will output the last
day of that month. Note that by checking the last day of
February you can test for leap year. Example:
> datecalc -l 2002 2
28″

lastday() {
integer year month leap
# ja fe ma ap ma jn jl ag se oc no de
set -A mlength xx 31 28 31 30 31 30 31 31 30 31 30 31

year=$1
if ((year<1860 || year> 3999)) ; then
print -u2 year out of range
return 1
fi
month=$2
if ((month<1 || month> 12)) ; then
print -u2 month out of range
return 1
fi

if ((month != 2)) ; then
print ${mlength[month]}
return 0
fi

leap=0
if ((!(year%100))); then
((!(year%400))) && leap=1
else
((!(year%4))) && leap=1
fi

feblength=28
((leap)) && feblength=29
print $feblength
return 0
}

date2jd() {
integer ijd day month year mnjd jd lday

year=$1
month=$2
day=$3
lday=$(lastday $year $month) || exit $?

if ((day<1 || day> lday)) ; then
print -u2 day out of range
return 1
fi

((standard_jd = day – 32075
+ 1461 * (year + 4800 – (14 – month)/12)/4
+ 367 * (month – 2 + (14 – month)/12*12)/12
– 3 * ((year + 4900 – (14 – month)/12)/100)/4))
((jd = standard_jd-2400001))

print $jd
return 0
}

jd2dow()
{
integer jd dow numeric_mode
set +A days Sunday Monday Tuesday Wednesday Thursday Friday Saturday

numeric_mode=0
if [[ $1 = -n ]] ; then
numeric_mode=1
shift
fi

jd=$1
if ((jd<1 || jd>782028)) ; then
print -u2 julian day out of range
return 1
fi

((dow=(jd+3)%7))

if ((numeric_mode)) ; then
print $dow
else
print ${days[dow]}
fi
return
}

jd2date()
{
integer standard_jd temp1 temp2 jd year month day

jd=$1
if ((jd<1 || jd>782028)) ; then
print julian day out of range
return 1
fi
((standard_jd=jd+2400001))
((temp1 = standard_jd + 68569))
((temp2 = 4*temp1/146097))
((temp1 = temp1 – (146097 * temp2 + 3) / 4))
((year = 4000 * (temp1 + 1) / 1461001))
((temp1 = temp1 – 1461 * year/4 + 31))
((month = 80 * temp1 / 2447))
((day = temp1 – 2447 * month / 80))
((temp1 = month / 11))
((month = month + 2 – 12 * temp1))
((year = 100 * (temp2 – 49) + year + temp1))
print $year $month $day
return 0
}

#
# Parse parameters and get to work.
case $1 in
-a) if (($# == 8)) ; then
if [[ $5 != - ]] ; then
print -u2 – “$USAGE”
exit 1
fi
jd1=$(date2jd $2 $3 $4) || exit $?
jd2=$(date2jd $6 $7 $8) || exit $?
((jd3=jd1-jd2))
print $jd3
exit 0
elif (($# == 6)) ; then
jd1=$(date2jd $2 $3 $4) || exit $?
case $5 in
-|+) eval ‘((‘jd2=${jd1}${5}${6}’))’
jd2date $jd2
exit $?
;;
*)
print -u2 – “$USAGE”
exit 1
;;
esac

fi
;;

-d|-D) if (($# != 4)) ; then
print -u2 – “$USAGE”
exit 1
fi
jd1=$(date2jd $2 $3 $4) || exit $?
numeric=-n
[[ $1 = -D ]] && numeric=”"
eval jd2dow $numeric $jd1
exit $?
;;

-j) if (($# == 4)) ; then
date2jd $2 $3 $4
exit $?
elif (($# == 2)) ; then
jd2date $2 $3 $4
exit $?
else
print -u2 – “$USAGE”
exit 1
fi
;;

-l) if (($# == 3)) ; then
lastday $2 $3
exit $?
else
print -u2 – “$USAGE”
exit 1
fi
;;

-help) print – “$USAGE”
print “”
print – “$DOCUMENTATION”
exit 0
;;

*) print -u2 – “$USAGE”
exit 0
;;

esac

#not reached
exit 7

Source : http://www.unix.com/unix-dummies-questions-answers/4870-days-elapsed-between-2-dates.html?postid=16559#post16559

I took my inspiration from it, and I could make it easily!

Have a nice day,
Wangee

Comments (0) - Posted in Shell script by wangee  

HTTPS + SSH + OpenVPN + XMPP + Tinc on the same port ? YES WE CAN!

Most of the networks (enterprise or public) are filtering and most of the time, only the ports 80 (HTTP) and 443 (HTTPS) are open. The consequence is obviously that you cannot connect to another port and this is a pity!

The dream for the geeks would be to use a single port for several protocols. Yves Rutschle (aka WhiteRabbit) started from a simple observation :

• When you’re browsing on a secured website, the https client (your web browser) talks the first.
• A ssh client (putty for example) is waiting for a response from the SSH server.  The server talks first (login prompt)

Imagine a daemon that listens on a TCP port (ex : port 443), and it waits.

• When no packet is received and after 2 seconds, it means it’s a ssh connection and it forwards to the ssh server.
• When it gets an incoming packet, and depending of the header, it’s either a https, openvpn, xmpp or even a tinc request and it forwards to the right server!

This amazing proxy exists and is called sslh.   sslh is a SSL/SSH multiplexer pretty easy to use and install.

For Fedora Core 16, there is no rpm package yet, that’s why I made one, and you can download it : sslh-1.10-1.fc16.i686.rpm

Please note that this package is unofficial for now, I’ve submitted it for being part of the official Fedora repositories.

Another IMPORTANT note: For now, sslh is launched under the user root, and it’s not a good thing at all on a security point of view.  I’m looking forward to solve this as soon as possible.

How to install sslh on Fedora Cora 16 ?

• Install the package

rpm -ivh sslh-1.10-1.fc16.i686.rpm

• check the file /etc/sysconfig/sslh and the ports where SSH and https (SSL) will listen. I want sslh to listen on port 443, ssh listen on its original port (22) and https to listen on 8443. Mine file looks like this :

LISTEN=ifname:443
SSH=localhost:22
SSL=localhost:8443

• check the file /etc/init.d/sslh. Locate the line OPTIONS= and set up the ports for your needs. Eventually replace –user=root by a restricted existing user name. This will workaround the root security flaw. Mine looks like this :

OPTIONS="-v --user=root -p 0.0.0.0:443 --ssl 127.0.0.1:8443 --ssh 127.0.0.1:22"

• configure the web server. Edit the file /etc/httpd/conf.d/ssl.conf and locate every line containing the string “443″ and replace by “8443″
• configure the ssh server : Edit the file /etc/ssh/sshd_config and modify the line starting by “Port 22″ according to your needs. Mine didn’t require any change.
• restart the syslog daemon, ssh and httpd

systemctl restart rsyslog.service
systemctl restart httpd.service
systemctl restart sshd.service

• Finally start sslh

systemctl start sslh.service

• Enjoy!

If you have applied this mini how-to, please leave a comment if you found something wrong or if you want to propose improvements.

Comments (0) - Posted in Sys Admin by wangee  

Better than top : htop!

I don’t know how to name it.  A command ? A tool ? No matter!  Every Unix SysAdmin knows “top” and find it very useful when troubleshooting the performance of an Unix system.

But do you know HTOP ? Htop is similar to top but with nice features.   It requires ncurses.

Instead of just showing numbers, htop will show you in a graphic way the CPU Usage, Memory Usage, Swap Usage.

Among those nice features (non-exhaustive list) :

• Horizontal and vertical Scrolling.
• Easier to kill a task.  No need to type the PID of the process you want to kill.   Just scroll to the process you want to kill and press “k” or F9, then the signal you want to send.
• Same if you want to renice a process : Press F7 to lower or F8 to raise the priority level.
• You want to see the parents/child processes ?  No problem, press F5 and you got a nice treeview.
• Sorting the processes by CPU%, MEM%, PID ?  F6 will do it!
• Finding the open files by a process ? pressing “l” will make it!
• Setting CPU Affinity (“a” key)

Tree View

Support for a large number of processors

 

Setting up CPU Affinity

 

htop running on a machine with 128 cores and 1TB of RAM

The father of “htop” is Hisham Muhammad (now you know where the “h” comes from ^^), and is hosted on sourceforge.

Packages available for Fedora, Debian and Ubuntu.

Comments (0) - Posted in Sys Admin by wangee  

GFS : The Grandfather – Father – Son backup tape rotation

Next episode  in those obvious things a backup administrator *must* know is about the tape rotation schemes.

I’m filling my wiki slowly and today I wrote a page about the GFS (Grandfather Father Son) tape rotation.  This tape rotation scheme is very often described as an industry standards.  Right or Wrong, I don’t know?  What I’m sure of, is that I used it many times.   And this GFS scheme is very interesting and can be adapted to most business need.  Depending of the constraints of data retention you might have, depending on the number of tapes you have, depending on the amount of data to backup, GFS is the first scheme to look at.

Click here to read the wiki page about the GFS Tape rotation scheme

Comments (0) - Posted in Backup by wangee  

Resuming on Amanda (Advanced Maryland Automatic Network Disk Archiver)

Comments (0) - Posted in Backup by wangee  

Yay! Starting up with my new blog

Hi guys!

This is my first post, and I’m starting up a technical blog.  The goal of this blog is simple : Keeping my readers updated on some technical stuffs I wrote.

What I plan is a new takeover of the skills I had in the past.  I think to do all that more in a spirit of re-learning, with details on how I made it.  I hope the readers of this technical blog will find relevant information for their private or business use.

In addition to this blog, I’ve set up a wiki that you can reach at the following URL : http://wiki.opsyx.com/

Thanks for reading, and you can already bookmark this website

Wangee

Comments (0) - Posted in Uncategorized by wangee