Hungred Dot Com

getting ï»¿ infront of json api call

After i migrated a server from apache to docker nginx, i notice that all the request coming from this nginx setup has a ï»¿ infront that is not visible to the browser but when you do a script call, your script will complain about invalid json format.

What is ï»¿

Well, first of all, this characters that we can't see is actually UTF-8 BOM or Byte order mark with the byte sequence 0xEF,0xBB,0xBF at the front of the file.

What to do

Lucky, There is 2 solutions for you. You can either change all the file format to UTF-8 and i meant all of it in the folder since any file included with UTF-8 BOM will cause this problem. Hence, all the file will need to be convert to UTF-8 or remove the byte sequence 0xEF,0xBB,0xBF

You can use the following script to run recursively on your root folder which will convert all the file from utf8 BOM to utf8.

find . -type f -exec sed -i.bak -e '1s/^\xEF\xBB\xBF//' {} \; -exec rm '{}.bak' \;

Personally, this works best but you'll need to work a little bit before its completed depending on how big your folder is.

The other solution is to remove BOM on the receiving end. so once you've grab your api content, remove the BOM with the following script,

        function remove_utf8_bom($text)
        {
                $bom = pack('H*','EFBBBF');
                $text = preg_replace("/^$bom/", '', $text);
                return $text;
        }

or you can just do this

                $raw_body = str_replace("\xEF\xBB\xBF",'',$raw_body);

both works pretty much the same. Then you should be able to parse your json_decode normally.

For more details file replacement you can visit muzso blog. Hope it helps!

easy resize kvm without lvm

Basically i have a chance to resize a kvm without lvm / lvm2. Adding size to a kvm is pretty straight forward, all you need to do is the following,

qemu-img resize vmdisk.img +40G

and if you boot up your machine, you'll see 10G if you hit the following command,

fdisk -l

now, we need to increase this partition so that our existing partition will increase from 60GB to 100GB.

[[email protected] ~]# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-2097151, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): 
Using default value 2097151

Command (m for help): p

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2dbb9f13

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     2097151     1047552   83  Linux

Command (m for help): w
The partition table has been altered!

Once you've done that you should have a bigger partition of 100GB

Resize your filesystem with resize2fs

now just do the following and your size should increase to 100GB

[email protected]:~# resize2fs /dev/sda1
resize2fs 1.43.5 (04-Aug-2017)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 8, new_desc_blocks = 13
The filesystem on /dev/sda1 is now 26214144 (4k) blocks long.

pretty straight forward i must say!

List of Useful Proxmox Command

It's been some time since i wrote something in this blog. I've been pretty busy with product building till i lose track of time. Time sure pass quickly. Recently i've a chance to work on Proxmox again. It is still as powerful as ever and many more version came up and so are the problems. Hence, i figured to list down those command that i come across that might be useful one day. I'll bring it into a few sections for easy navigation in the future.

Proxmox Command

Get a quick overview on how fast your system is: pveperf
Verify the subscription status of your hardware node: pvesubscription get
Start a backup of machine 101: vzdump 101 -compress lzo
PVE Cluster Manager - see "man pvecm" for details.
Restart every single Proxmox services: service pve-cluster restart && service pvedaemon restart && service pvestatd restart && service pveproxy restart
Proxmox VE version info - Print version information for Proxmox VE packages. : pveversion
Find next free VM ID: pvesh get /cluster/nextid
View sum of memory allocated to VMs and CTs: grep -R memory /etc/pve/local | awk '{sum += $NF } END {print sum;}'
View sorted list of VMs like vmid proxmox_host type: cat /etc/pve/.vmlist | grep node | tr -d '":,'| awk '{print $1" "$4" "$6 }' | sort -n | column -t
View sorted list of vmid: cat /etc/pve/.vmlist | grep node | cut -d '"' -f2 | sort -n

KVM Command

List all your KVM machines: qm list
See how much memory your machine 101 has: qm config 101 | grep ^memory
List the memory setting of a kvm: qm config 101 | grep ^memory
restore KVM vzdump backups - see "man qmrestore"
backup utility for virtual machine - see "man vzdump"
unlock kvm: qm unlock 101
Restore a QemuServer VM to VM 601: qmrestore /mnt/backup/vzdump-qemu-888.vma 601

LXC Command

forcefully start lxc: lxc-start -n 101 -F
mount lxc virtual disk: pct mount 101
unmount lxc virtual disk: pct unmount 101
repair virtual disk: pct fsck 101
check configuration of lxc: pct config 101
Remove container: pct destroy 101
Restore a container to a new CT 600: pct restore 600 /mnt/backup/vzdump-lxc-777.tar

OpenVZ Command

utility to control an OpenVZ container - see "man vzctl"
vzctl wrapper to manage OpenVZ containers - see "man pvectl"
display top CPU processes: vztop
cat /proc/user_beancounters
vzlist
backup utility for virtual machine - see "man vzdump"
restore OpenVZ vzdump backups - see "man vzrestore"

if you got anything to share or have any awesome command useful for your day to day Proxmox management, do let me know!

Awesome Useful cPanel Commands

Here are a list of useful commands for everyone under the sun to do their work easily with cPanel

cPanel Resource Usage Stats

To view cPanel’s stats you can run this command via SSH:

/usr/local/cpanel/bin/dcpumonview

This will show all processes, users, etc.

Get cPanel Resource Stats for X Days

If you want to get the stats for a user for say the past 5 days or so, run this command in SSH:

domain="thedomain.com"; for i in `seq 1 7 `; do let i=$i+1 ; let  k=$i-1 ; let s="$(date +%s) - (k-1)*86400"; let t="$(date +%s) - (k-2)*86400"; echo `date -Idate -d @$s`; /usr/local/cpanel/bin/dcpumonview `date -d @$s +%s` `date -d @$t +%s` | sed -r -e '[email protected]^<tr bgcolor=#[[:xdigit:]]+><td>(.*)</td><td>(.*)</td><td>(.*)</td><td>(.*)</td><td>(.*)</td></tr>[email protected]: \1\tDomain: \2\tCPU: \3\tMem: \4\tMySQL: \[email protected]' -e '[email protected]^<tr><td>Top Process</td><td>(.*)</td><td colspan=3>(.*)</td></tr>[email protected]\1 - \[email protected]' | grep $domain -A3 ; done

Script to find cPanel account and its corresponding IP address

cat /etc/userdatadomains | perl -pi -e "s/^.*? //," | perl -pi -e "s/==.*==6/ 6/," | perl -pi -e "s/:80==//," | sort | uniq

cPanel script to assign IP via shell: /usr/local/cpanel/bin/setsiteip -u username IPaddress

Courtesy of https://sites.google.com/site/pleskylinuxcom/bash-scripting

If you want to just monitor a specific user and not access the logs you can do so with these commands:

Monitor specific user using TOP

top -c d2 -u username

Monitor all users using TOP

top -c d2

Alternately you can use htop instead of top if you have it installed.

Script to delete Big file

#!/bin/bash
find /home -name '*.DS_Store' -type f -delete &
find /home -name '*.swp' -type f -delete &
find /home -name '*.swo' -type f -delete &
find /home -name 'error_log' -size +10M -type f -delete &
find /home -type f -name '*' -size +500M -exec rm -if {} \; &

In case you are wondering, anything bigger than 500M

find spammer script in cpanel

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F "cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

the above will look for script that is spamming your cpanel server

find 10 biggest disk user in cpanel

find /home -type d -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

the above will search for the 10 biggest folder used by your user

Clear Exim Mail Queue

exiqgrep -zi|xargs exim -Mrm

this will clear all your exim queue to sparkling clean.

Delete cPanel email more than 2 years

find -P /home/*/mail/*/*/cur -mtime '+729';find -P /home/*/mail/*/*/new -mtime '+729'

firing above will delete all email that is bigger than 729 day

check for all unique ip connected to your server

netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

useful when you are getting DDOS

Setup HA + Floating IP + BGP on Vultr on Ubuntu 17.10

ok, recently i am trying to setup a low cost HA on the cloud. And i decide to go with Vultr since they provide $2.5 machine for me to test and play around (although i can only spam 2 of such machine) Here i will drop down how i did it so i won't forget and you can copy it if you want.

Initial Cloud VPS Setup

ok here, you will need at least 3 machines, i'll call them the following with the spec i've used to create this setup

Ubuntu 17.10 x64 1024 MB Server - 192.168.23.10 [Worker1]
Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker2]
Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker3]
Floating IP - 192.168.10.10

i name them worker1, worker2 and worker3 as show above. Now, the reason why i choose Ubuntu 17 instead of CoreOS is purely because CoreOS is stuck at Docker 1.12 the time i am writing and you need to install its Alpha version to get Docker 17.09-ce

Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.

Installing lastest
Docker on Ubuntu 17.10

Fire the following comment on each machine,

sudo apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce -y

and if it doesn't work, most likely docker doesn't support your newest Ubuntu and you have to switch it to a lower one as show below,

vi /etc/apt/sources.list

change artful to zesty

deb [arch=amd64] https://download.docker.com/linux/ubuntu zesty stable

then go ahead and install again.
Once its done, check the version. You should get something like this below,

[email protected]:/etc/bird# docker -v
Docker version 17.09.0-ce, build afdb6d4

once this is done, we are going to install blue for BGP

Installing and Configuring BGP using Bird in Ubuntu 17.10

First, do yourself a favor and add its repo

sudo add-apt-repository ppa:cz.nic-labs/bird
apt-get update -y

once this is done, you'll get an error because it doesn't support Ubuntu 17.10 yet. Hence, gonna go downwards again.

vi /etc/apt/sources.list.d/cz_nic-labs-ubuntu-bird-artful.list

change it to the below,

deb http://ppa.launchpad.net/cz.nic-labs/bird/ubuntu xenial main

save it and try to install bird below,

sudo apt-get install bird

And we have bird installed. Now to configure it, we will paste the following into /etc/bird/bird.conf

log "/var/log/bird" all;

router id 192.168.23.10; # instance ip

protocol device
{
scan time 60;
}

protocol direct
{
interface "dummy1";
}

protocol bgp vultr
{
local as 11512; #instances ASN number
source address 192.168.23.10; #instance ip
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.252 as 64019; #vultr IP and ASN number
password "xC4c2EGaXuP7a74!"; #vulr password
}

then proceed to restart

service bird restart
service bird status
● bird.service - BIRD Internet Routing Daemon (IPv4)
   Loaded: loaded (/lib/systemd/system/bird.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-10-30 10:30:51 UTC; 19min ago
  Process: 5140 ExecStartPre=/usr/sbin/bird -p (code=exited, status=0/SUCCESS)
  Process: 5130 ExecStartPre=/usr/lib/bird/prepare-environment (code=exited, status=0/SUCCESS)
 Main PID: 5145 (bird)
    Tasks: 1 (limit: 4915)
   Memory: 640.0K
      CPU: 53ms
   CGroup: /system.slice/bird.service
           └─5145 /usr/sbin/bird -f -u bird -g bird

Oct 30 10:30:51 worker_a systemd[1]: Starting BIRD Internet Routing Daemon (IPv4)...
Oct 30 10:30:51 worker_a systemd[1]: Started BIRD Internet Routing Daemon (IPv4).

now make sure you have the above, if not, its most likely because your log permission is set as root, if that's the case, try the below,

chown bird:bird /var/log/bird

and restart bird again. Ensure your firewall isn't blocking port 179 for BGP to communicate with your machine. Once this is done, you can test it out externally via telnet

telnet 192.168.23.10 179
Trying 192.168.23.10...
Connected to 192.168.23.10.vultr.com.
Escape character is '^]'.
Connection closed by foreign host.

you should get something like this. Now, the most important part if not you will get the following

birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   start  13:49:30    Idle          Received: Bad peer AS
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 0 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              0          0          0        ---          0
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Idle
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Error wait:       10/300
    Last error:       Received: Bad peer AS

Setup Dummy IP Configuration

run the following to create a dummy IP configuration for your Bird to forward to

ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.168.10.10

where 192.168.10.10 is the floating IP. Now you can test whether the above setup is correct with the following

ip addr show dev dummy1
6: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 4e:79:76:36:6d:b0 brd ff:ff:ff:ff:ff:ff
    inet 45.77.110.78/32 scope global dummy1
       valid_lft forever preferred_lft forever
    inet6 fe80::4c79:76ff:fe36:6db0/64 scope link
       valid_lft forever preferred_lft forever

Now, restart your Bird so that it gets the above dummy. To set it persistently, do the following

vi /etc/rc.local
ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.168.10.10/32

then restart or reboot your machine

service bird restart

Once this is done, you should get the following,

birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   up     14:44:15    Established
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 1 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              1          0          0        ---          1
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Established
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Neighbor ID:      208.167.241.119
    Neighbor caps:    refresh restart-aware AS4 add-path-rx
    Session:          external multihop AS4
    Source address:   45.77.78.111
    Hold timer:       132/180
    Keepalive timer:  29/60

Once you get this, all should be good. Try it out.