UNIX useful cmds

http://freeengineer.org/learnUNIXin10minutes.html

http://www.theunixschool.com/2010/11/5-important-things-to-follow-to-be-fast.html

http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/

awk -F'|' '{print $63}' ACCOUNTS_DAILY_20150601.txt|grep '10'|uniq|head

awk '$63=="10"{$63="GTB"}1' FS='|' OFS='|' ACCOUNTS_DAILY_20140101.txt>temp.txt

awk '$63=="0"{$63="AML"}1' FS='|' OFS='|' ACCOUNTS_DAILY_20140101.txt

awk '$63=="1010"{$63="PFS"}1' FS='|' OFS='|' ACCOUNTS_DAILY_20140101.txt

awk '$63=="1000"{$63="0"}1' FS='|' OFS='|' ACCOUNTS_DAILY_20140101.txt

• In vi editor open file , then shift : /wordof record

It will go to that specific record

O extract a .class file http://www.molvis.indiana.edu/app_guide/unix_commands.html

Command purpose

rm <file name> To remove files,.txt….

rmdir <your Directory name> To Remove Directory name

rm –rf to remove nonempty dir

alias Enquire or set alias

awk Deal with tabular data

cat file* concatenate file

cd dir change working directory

chmod opts file* Change attributes and permissions

cp src* dst copy source files to destination

diff Difference between files.

display file* Display images

file file* List possible type/origin of files.

find node types find files from node matching types.

ftp host file transfer protocol

gcc GNU C Compiler (System build tool)

grep string file* find lines containing string in files

gzip file* compress files

history Command history

host str DNS lookup utility

kill N end task number N

less file Look at file

ls {dir..} list files in directories

ln {options} link files

lp file* print files

make Update series of dependent files

man topic Online help on topic

mdir floppy disk directory

mcopy floppy disk copy

more file Scroll through file or stdin

mount mount filesystems

od Look at non ascii file

ping host test connection with host

ps process status.

pwd print working directory

rxvt Open a console on X windows.

sed Batch stream editor

strings file Look at ASCII strings in a file

su {name} Swap user to root, or given name

tar opts dst file* bundle or unbundle files.

telnet host intereactive connection to another computer

touch file* Change timestamps of files

vi file* edit files. ':n' in vi gives next file

wc file* Word count files

wget options download web pages or ftp server files

xset options X-windows behaviour

xterm Set up X-terminal

1. Basic Cursor Movement for vi editor

From Command Mode

k Up one line

j Down one line

h Left one character

l Right one character (or use <Spacebar>)

w Right one word

b Left one word

use 6k,6j…………….etc

NOTE: Many vi commands can take a leading count (e. g., 6k, 7e).

http://www.linuxnix.com/2012/07/23-awesome-less-known-linuxunix-command-chaining-examples.html

Extract a .gz file:
gzip -d file.gz

To see gzipped file without actually having to gunzip it

gunzip -c NORKOM_US_20140827.xml

or

zcat 12_TRANSACTIONS_DAILY_20141027.txt.gz|head -1

Extract a tar.gz file:
tar xvzf file.tar.gz

tar xvf file.tar

Compress into .Z format:

compress 1_TRANSACTIONS_DAILY_20120525.txt

here above .txt converts into .Z format

gzip filename.xml

it converts filename.xml.gz

Find Java class inside a folder of JARS

for a in *.jar
do
echo $a; unzip -t $a | grep -i CLassFileName
done

Sending email:

mail -s 'test email from umix ' kiran3kumar@hsbc.co.in

or

mailx -s 'test email from umix ' kiran3kumar@hsbc.co.in<mail

Extract jar file:

Jar xf xxx.jar

Extract specific class from jar use below command:

jar xf norkomextract.jar | grep -i ExtractNorkomData.class

find cmd

http://www.cyberciti.biz/faq/howto-find-a-file-under-unix/

http://www.bsd.org/unixcmds.html

-- to find path of command in linux

whereis java

To find the path the operating system uses to execute a command when you enter it on the command line, use the which command instead, for example:

which lpr

(for ref………..http://kb.iu.edu/data/acec.html)

--find command is recommend because of speed and ability to deal with filenames that contain spaces.cd /path/to/dir
find . -type f -exec grep -l "word" {} +
find . -type f -exec grep -l "seting" {} +
find . -type f -exec grep -l "foo" {} +

find *.xml -type f -exec grep -l "table" {} +

find / -type f -name "*.sh" -exec grep -il 'KYC_PATH=' {} +

find . -type f -name kyc_common.sh

find . -type f -name "*.sh" -exec grep -il 'KYC_PATH=' {} +

Ex:

norkomsm@usvh3euap42:/opt/norkomSMDE1/norkom_home/bin> find mon_alert Start_GTB_Scenario_Manager.sh

--Retriving specific lines of a file based on matching

Grep <matching name> <filename>

grep '<parameter name="filePattern">' scheduleconfig_dataload.xml

grep '<parameter name="fileSpecPath">' scheduleconfig_dataload.xml

cat output.log | grep FAIL

I want some more information along with this. Like the 2-3 lines above this line with FAIL

grep -C 3 FAIL output.log

-to remove non-printable binary characters (garbage) from a Unix text file

tr -cd '\11\12\15\40-\176' < file-with-binary-chars > clean-file

--

(mv)Rename command syntax

rename oldname newname *.files

For example rename all *.bak file as *.txt, enter:$ rename .bak .txt *.bak

rename .txt .txt.bak *.txt

rename .txt .txt.1 *20140114.txt

---- Find a File by Name in UNIX, Solaris, or Linux

Using the find command, one can locate a file by name.

To find a file such as filename.txt anywhere on the system:

find / -name filename.txt -print

Recent operating system versions do not require the print option because this is the default. To limit the search to a specific directory such as /usr:

find /usr -name filename.txt –print

ex: find /opt/norkomSMDE4/norkom_home/conf_PFS/monitoring/ -name processingconfig_PFS_PostProcess.xml –print

find /opt/norkomSMDE4/norkom_home/ -name processingconfig_PFS_PostProcess.xml -print

--df

grep -P "[\x80-\xFF]" ACCOUNTS_UPDT_20121209.txt

LANG=C sed -i 's/[\x80-\xFF]//g' ACCOUNTS_UPDT_20121209.txt

nohup /opt/norkomSM/HSBC/scripts/shell_scripts/nrkm_monthend_mvzip.sh &

 df –h

(disk freespace)

The above command is one of the most commonly used commands as it displays the sizes in an easy to read format as shown in the below example.

Filesystem	Size	Used	Avail	Use%	Mounted on
/dev/hda2	28G	7.6G	19G	29%	/
tmpfs	252M	0	252M	0%	/lib/init/rw
- grep usage http://www.cyberciti.biz/faq/howto-use-grep-command-in-linux-unix/ -- Display all environment variables You can use the commands env, set, and printenv display all environment variables -to remove non printable chars in a file. perl -pe's/[[:^ascii:]]//g' < ACCOUNTS_DAILY_20121206.txt > newfile_accounts_ascii.txt in the above example, ACCOUNTS_DAILY_20121206.txt has non printable chars.and newfile_accounts_ascii.txt is the file after removing non printable chars. To check non printable chars in a file, grep -P "[\x80-\xFF]" *.* to see non printable ^ charactors: cat -v NORKOM_US_20141127.xml | grep -n '\^' to remove ^ chars: cat -v NORKOM_US_20141127.xml | sed 's/\^.//g' > NORKOM_US_20141127.xml_removed grep '[^[:print:]]' NORKOM_US_20141127.xml --- http://www.aliencoders.com/content/how-remove-m-and-other-non-printable-characters-file-linux ---Removing one record/line of file

http://en.kioskea.net/faq/1451-sed-delete-one-or-more-lines-from-a-file

---Remove spaces in a file name

cp '1_ACCOUNTS_ DAILY_20130701.txt' `ls '1_ACCOUNTS_ DAILY_20130701.txt' | tr -d ' '`

--add header trailer to a file

sed '1i\HR|t|4' mytest.txt | sed '2i\hr2|r|0' | sed '$a\#TR|End' > mytest.txt.tmp

sed '1i\HR|t|4' mytest.txt | sed '4i\ram nest|r|0' | sed '$a\#TR|End' > mytest.txt.tmp.1

prog:

#!/bin/sh

FILES=$(echo *.in)

for file in $FILES
do
echo "Modifying file : $file"

sed '1i\
h1|t|4' $file | sed '2i\
h2|r|0' | sed '$a\
#End' > $file.tmp

mv $file.tmp $file
done

http://www.dwbiconcepts.com/tutorial/24-interview-questions/50-top-25-unix-interview-questions-with-answers-part-i.html

Remove HR,TR of a file,

sed '1d;$d' file > newfile

or

below is better one

sed -i'' -e '1d' -e '$d' yourfile

sed -i'' -e '1d' -e '$d' *20140624.txtnon

1d deletes first line $d deletes last line.

-cmd to see the folder structure of a directory

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'

- fist column of file not NRKM( | separated file)

cut -d'|' -f1 NRKM_CASES_RBWMCMB_DAILY_20131127.txt.1 | grep -v 'NRKM'

• Files in human readable format like kb.mb,gb…etc

ls –lh

->to see control-M charactors in a file (^M)

Cat –v file.sh

perl -p -i -e "s/\r//g" file.sh

->to get 3^rd fields of file whose records are separated by pipe

cut -d'|' -f3 <file nameNGTInvalidCustNullCust.bad> | uniq

- grep '23039451\|23039443\|23027887\|7104084178\|7104084111' ODS_CRM_ACCTS_DAILY_20140125.xml

In solaries ,

egrep '23039451|23039443|23027887|7104084178|7104084111' CRM_ACCOUNT_DAILY_20140129.xml

above one will work in all os’s

-- lines before and after grep match

---------------------------------

grep -B5 "the living" gettysburg-address.txt # show all matches, and five lines before each match

grep -A10 "the living" gettysburg-address.txt # show all matches, and ten lines after each match

grep -B5 -A5 "the living" gettysburg-address.txt # five lines before and ten lines after

http://alvinalexander.com/unix/edu/examples/grep.shtml

http://alvinalexander.com/unix/edu/examples/grep.shtml

http://www.thegeekstuff.com/2011/10/grep-or-and-not-operators/

Ok, Assuming that your file is a text file, having the fields seperated by comma seperator ','. You would also know which field 'transactionid' is in terms of its position. Assuming that your 'transactionid' field is 7th field.

awk -F ',' '{print $7}' text_file | sort | uniq -c

awk -F '|' '{print $4}' NRKM_CASE_REL_DAILY_20140624.txt | sort | uniq -c |wc -l

specific fields with poipe separated:

awk -F'|' '{print $1"|"$1"|P|"}' CUSTOMERS_DAILY_20150317.txt|grep -v 'PSEUDO_'|head

sort and uniq records:

http://www.theunixschool.com/2012/09/howto-remove-delete-duplicate-records.html

sort file | uniq

Unix - How to append text to every line of a file except for the header line/first line

Add 2 pipes to end of each line

sed ' s/$/ |||/' mytest.txt>new.txt

sed -e 's/$/test added/' InFile >OutFile

Here are three options:

• awk and its variants (gawk, mawk etc.):
• awk '{if(NR==1){print $0,"| Place"} else{print $0,"| Paris"}}' file.txt
• Perl:
• perl -lne '$.==1 ? print "$_ | Place" : print "$_ | Paris"' file.txt
• sed
• sed '1 s/$/ | Place/; 1! s/$/ | Paris/' file.txt

-----Remove Control-M chars

Perhaps you have ctrl-M (carriage returns) at the end of each line in your file.

To View: cat -v /data/upgrade/upgrade_db.ksh

To Fix: perl -p -i -e "s/\r//g" /data/upgrade/upgrade_db.ksh

Unix Sed Tutorial: Find and Replace Text Inside a File Using RegEx

sed -i 's#trunc(systimestamp) - 18#trunc(systimestamp) - 11#g' filename

sed -i 's#trunc(systimestamp - 12)#trunc(systimestamp - 5)#g'

sed -i 's#QA2#QA4#g' filename

http://www.thegeekstuff.com/2009/09/unix-sed-tutorial-replace-text-inside-a-file-using-substitute-command/

get no.of pipes of a record in a file

awk -F"|" '{print NF-1}' NGT_CUSTINFO_20141021.txt |sort -n | uniq –c

-send email from unix/linux

echo "something" | mailx -s "subject" kiran3kumar@hsbc.co.in

mailx -s "subject" kiran3kumar@hsbc.co.in <nrkm_sftp_test.txt

uuencode nrkm_sftp_test.txt nrkm_sftp_test.txt| mail kiran3kumar@hsbc.co.in -s "subject"

uuencode nrkm_sftp_test.txt nrkm_sftp_test.txt| mailx kiran3kumar@hsbc.co.in -s "llsubject"

Send an Email with Attachment and Body

( cat nrkm_body_test.txt; uuencode nrkm_sftp_test.txt nrkm_sftp_test.txt )| mail -s "Sub:Email With Body Text and Attachment" kiran3kumar@hsbc.co.in

http://www.thegeekstuff.com/2009/12/how-to-send-an-email-with-attachment-and-body-from-linux/

Display common lines in both files:

perl -ne 'print if ($seen{$_} .= @ARGV) =~ /10$/' file1 file2

or

comm -1 -2 /path/to/file1/ /path/to/file2

here file1 and file2 are in sorted order data.

For sorting use below

/path/to/file1 | sort > /path/to/file1_sorted
/path/to/file2 | sort > /path/to/file2_sorted
comm -1 -2 /path/to/file1_sorted/ /path/to/file2_sorted

Try

Only file1 elements

comm -2 -3 /path/to/file1_sorted/ /path/to/file2_sorted

only file2 elements

comm -1 -3 file1.txt file2.txt

-to see last line of multiple files

tail -n1 file1.txt file2.txt --last one line of files

tail -n2 file1.txt file2.txt --last 2 lines of files

--File operations

http://www.webforefront.com/bigdata/xsvfiles.html

Data Masking in unix cmds:

b86471@usvh3euap42:/home/b86471> cat feedbeforemask.txt

1111|hsgsfr|tgfr|6hhgvcx|okhjg|hhhhgfgh||

After masking 3^rd fields,

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{$3="9999"}1' feedbeforemask.txt

1111|hsgsfr|9999|6hhgvcx|okhjg|hhhhgfgh||

After masking 1^st,4^th,3^rd fields oin a file,

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{$1="xxxxx"} {$4="9999"} {$3="9999"}1' feedbeforemask.txt

xxxxx|hsgsfr|9999|9999|okhjg|hhhhgfgh||

placing masked file into other file,

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{$1="xxxxx"} {$4="9999"} {$3="9999"}1' feedbeforemask.txt>feedaftermasking.txt

b86471@usvh3euap42:/home/b86471> cat feedaftermasking.txt

xxxxx|hsgsfr|9999|9999|okhjg|hhhhgfgh||

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{$2=$4=$3="9999"}1' feedbeforemask.txt

1111|9999|9999|9999|okhjg|hhhhgfgh||

Apply function of 2^nd field

awk -F'|' -v OFS='|' '{$2=substr($2,2,3)}1' feedbeforemask.txt

b86471@usvh3euap42:/home/b86471> cat feedbeforemask.txt

1111|hsgsfr|tgfr|6hhgvcx|okhjg|hhhhgfgh||

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{sub(/h/, "cc", $2)}1' feedbeforemask.txt

1111|ccsgsfr|tgfr|6hhgvcx|okhjg|hhhhgfgh||

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{gsub(/1/, "2", $1)}1' feedbeforemask.txt

2222|hsgsfr|tgfr|6hhgvcx|okhjg|hhhhgfgh||

b86471@usvh3euap42:/home/b86471> awk -F'|' -v OFS='|' '{gsub("h", "z",$2)}1' feedbeforemask.txt

1111|zsgsfr|tgfr|6hhgvcx|okhjg|hhhhgfgh||

Replacing first and second column of feed

awk -F'|' -v OFS='|' '{gsub("1", "2", $1)}{gsub("h", "z",$2)}1' feedbeforemask.txt

2222|zsgsfr|tgfr|6hhgvcx|okhjg|hhhhgfgh||

----awk -F'|' '{print $1"|"$1"|P|"}' file >temp file

 awk -F'|' '{print $1"|"substr($2,1,3)"|"$3"|"$4"|"$5"|"$6"|"}' CUSTOMER_ASSOC_INFO_DAILY_20150601.txt|head

------awk '$70==""{$70="PFS"}1' FS='|' OFS='|' testfile.txt

-awk '{ if (length($2) > max) max = length($2) } END { print max }' CUSTOMER_ASSOC_INFO_DAILY_20150601.txt

-----get specific line on huge file

# print line number 52

sed -n '52p' # method 1

sed '52!d' # method 2

sed '52q;d' # method 3, efficient on large files

sed '52q;d' CUSTOMERS_DAILY_20150127.txt.bkp

Assuming you need lines 20 to 40,

sed -n '20,40p' file_name

get no.of pipes of a record in a file

awk -F"|" '{print NF-1}' NGT_CUSTINFO_20141021.txt |sort -n | uniq –c

get the records which are having no.of pipes not equal to 15 (try NF-1 also)

awk -F'|' 'NF!=16 {print "Record",NR,"Fields count:",NF}' ASSOCIATED_NAME_DAILY_20150127.txt

get the records which are having no.of pipes equal to 15:

awk -F'|' 'NF==16 {print "Record",NR,"Fields count:",NF}' ASSOCIATED_NAME_DAILY_20150127.txt|wc -l

get recs which are not having no.of pipes 112

awk -F'|' 'NF-1!=112 {print "Record",NR,"Fields count:",NF-1}' CUSTOMERS_DAILY_20150127.txt.bkp

print record no and record of file which are having no.of pipes 3

awk -F'|' 'NF-1==3 {print NR,$0}' CUSTOMERS_DAILY_20150127.txt.bkp

or

awk -F'|' 'NF==4 {print NR,$0}' CUSTOMERS_DAILY_20150127.txt.bkp

NF=no.of fileds

NR=no.of Rows

Add Characters at the BEGINNING of Each Line

http://www.shellhacks.com/en/Add-Character-to-the-Beginning-or-to-the-End-of-Each-Line-using-AWK-and-SED

Use one of the following commands to remove blank lines from a file.

1. Using the grep command :
$ grep -v "^$" file.txt
2. Using the sed command :
$ sed '/^$/d' file.txt
3. Using the awk command :
$ awk '/./' file.txt
4. Using the tr command :
$ tr -s '\n' < file.txt

You also can pipe the output of each command to a new file, like follows :
$ grep -v "^$" input.txt > output.txt

Print Lines Between Two Patterns with SED:

http://www.shellhacks.com/en/Using-SED-and-AWK-to-Print-Lines-Between-Two-Patterns

sed -n '/StartPattern/,/EndPattern/p' FileName

-change specific filed sof big file

I have a very big file (more than 10000 columns). I would like to change 3 entries in the second column and keep anything else the same, including the field separator

awk '$2==120{$2=1201}1' FS='\t' OFS='\t' file

awk '$1=="eCRM"{$1="CMB"}1' FS='|' OFS='|' testfile.txt

For multiple conditions just add more structures i.e:

awk '$2==120{$2=1201}$3==130{$3==1301}1'

awk -F'|' '$20=" "{print $0}' OFS='|' ACCOUNTS_DAILY_20150601.txt|wc -l

To see longest line of file:

wc -L testfile.txt

it will show longest line length only

below shows all

$ awk '{ print length(), NR, $0 | "sort -rn" }' tmp.txt

10 3 abracadabr

8 4 mu mu mu

7 2 tatatat

6 1 lalala

To show just the first line:

$ awk '{ print length(), NR, $0 | "sort -rn" }' tmp.txt | head -n 1

10 3 abracadabr

-The example mentioned below will print the lines 120, 145, 1050 from the syslog.

$ sed -n -e 120p -e 145p -e 1050p /var/log/syslog

http://www.thegeekstuff.com/2009/08/10-awesome-examples-for-viewing-huge-log-files-in-unix/

-split big file with 1 GB(1024mb) files set

> split -b 1024m <file> <split-name>

> split -b 1024m reallybigfile smallfile
should give you files smallfileaa, smallfileab, smallfileac and smallfilead

split -b 10m file1.txt slitfile.txt (spilt into 10mb files set)

--password protected folder/file

zip -P password -r zzzz.zip zzzz.txt

zip -P password -r zzzz.zip zzzz (here zzzz is folder)

zip -P happyhsbc -r UAT4.zip UAT4