cman gets stucked on unmounting configfs

well, once again i got stuck when my cluster suddenly doesn't work and i need to figure out why without restarting my server. restarting cman always throw me a stuck "Unmounting configfs..."

/etc/init.d/cman restart
Stopping cluster:
   Stopping dlm_controld... [  OK  ]
   Stopping fenced... [  OK  ]
   Stopping cman... [  OK  ]
   Unloading kernel modules... [  OK  ]
   Unmounting configfs...

running service cman status i get the following

service cman status
Found stale pid file

now i tried to stop all services that may caused this but apparently it was because the cluster is still running, hence, it wasn't able to unmount configfs

/etc/init.d/pve-cluster stop

once i've stopped my pve cluster. i tried restarting cman again

/etc/init.d/cman restart
Stopping cluster:
   Stopping dlm_controld... [  OK  ]
   Stopping fenced... [  OK  ]
   Stopping cman... [  OK  ]
   Unloading kernel modules... [  OK  ]
   Unmounting configfs... [  OK  ]
Starting cluster:
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Unfencing self... [  OK  ]

now my cman starts running again i can start my cluster again

/etc/init.d/pve-cluster restart
Restarting pve cluster filesystem: pve-cluster.

just to make sure everything works, you can run the following

service cman status
cluster is running.

now i don't have to restart my server in order to get my cluster running again. Hope this helps!

SMTP Auth – SMTP Relay

If you are getting an error with the following error

SMTP error from remote mail server after RCPT TO:<admin @domain.com>: 550 smtp auth requried

from the script that is running on your server but the domain.com isn't locate on the same physical machine. You are most likely doing a SMTP Relay and your exim isn't really happy with not having authentication credential being provided.

In this case, you can just add the following to your exim.conf file, assuming 123.123.123.123 is your script server,

domainlist local_domains = dsearch;/etc/exim4/domains/
domainlist relay_to_domains = dsearch;/etc/exim4/domains/
hostlist relay_from_hosts = 127.0.0.1 : 123.123.123.123
hostlist whitelist = net-iplsearch;/etc/exim4/white-blocks.conf
hostlist spammers = net-iplsearch;/etc/exim4/spam-blocks.conf

you can also placed 123.123.123.123 into the file /etc/exim4/white-blocks.conf to whitelist the host on your server.

you might also need to add auth_advertise_hosts = * as show below,

host_lookup = *
auth_advertise_hosts = 123.123.123.123
rfc1413_hosts = *

which expand to all host other than localhost (of course, you might want to change it to ip instead of *)

this should allows your script to sent email using your smtp server as a relay without the need for authentication.

WordPress libgomp: Thread creation failed: Resource temporarily unavailable

Another fresh issue, pretty straight forward but if you are facing this issue with the error in apache saying

libgomp: Thread creation failed: Resource temporarily unavailable, referer: https://xxxxx.org/wp-admin/media-new.php

this is most likely due to your server limit has been reached either by user level or root level. The quickest way to resolve this temporary is to increase your soft limit as shown below,

ulimit -s 999999

once you've done that. you should try upload file in WordPress and you shouldn't see the HTTP Error message. but to make this permanent after you've reboot.

Open the file located at vi /etc/security/limits.d/90-nproc.conf

*          soft    nproc     999999
root       soft    nproc     unlimited

update and change the value u see above and it should do the trick. If this doesn't do the trick, you might want to try adding the following to your .htaccess

SetEnv MAGICK_THREAD_LIMIT 1

This happens when the full installation of ImageMagick cannot be done which causes the HTTP Error to show.

Fix Getting $_FILES ERROR 3

Alright. its been a year of resting and facing the normal issue which I could easily google and get the answer out of everyone (as usual), but recently I faced this issue where my customer upload fails 'sometimes' and the error they are getting from $_FILES is 3 which is "The uploaded file was only partially uploaded."

Now, this is all good and great and should be easily be resolved but in the end, you get stuck for a long long time. Some of the solutions you can try are the following,

Ensure your PHP setup are correct. Such as the below config

upload_max_filesize=80M
post_max_size=80M
file_uploads = On
upload_tmp_dir = /tmp
max_file_uploads = 20
memory_limit=-1

and if these are correctly configured and you are still getting an issue and you are suspecting maybe its AWS load balancer or you might have a load balancer, you can forget that idea since its 'partial upload' error meaning, its uploaded but it gets 'terminated' suddenly. So the next possible candidate here would be the web server.

And since the customer was using an apache2 server. Do check the following to ensure they are not set to '0'.

Timeout 3000
KeepAlive On
MaxKeepAliveRequests 1000

My issue was that someone set Timeout to 0 and it causes this issue consistently. The other issue that you may face and get stuck would be that Timeout or MaxkeepAliveRequests is too short which cause error 3 to show up due to the web server cutting your users off during upload but not everyone gets cut off. Hence, you guys were unable to reproduce this consistently.

Hope this helps!

Setup HA + Floating IP + BGP on Vultr on Ubuntu 17.10

ok, recently i am trying to setup a low cost HA on the cloud. And i decide to go with Vultr since they provide $2.5 machine for me to test and play around (although i can only spam 2 of such machine) Here i will drop down how i did it so i won't forget and you can copy it if you want.

Initial Cloud VPS Setup

ok here, you will need at least 3 machines, i'll call them the following with the spec i've used to create this setup

  • Ubuntu 17.10 x64 1024 MB Server - 192.168.23.10 [Worker1]
  • Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker2]
  • Ubuntu 17.10 x64 512 MB Server - 192.168.23.11 [Worker3]
  • Floating IP - 192.168.10.10

i name them worker1, worker2 and worker3 as show above. Now, the reason why i choose Ubuntu 17 instead of CoreOS is purely because CoreOS is stuck at Docker 1.12 the time i am writing and you need to install its Alpha version to get Docker 17.09-ce

Note: You should not attach the floating IP to any particular instance via your control panel. If an IP is attached via the control panel, high availability will not function properly.

Installing lastest
Docker on Ubuntu 17.10

Fire the following comment on each machine,

sudo apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce -y

and if it doesn't work, most likely docker doesn't support your newest Ubuntu and you have to switch it to a lower one as show below,

vi /etc/apt/sources.list

change artful to zesty

deb [arch=amd64] https://download.docker.com/linux/ubuntu zesty stable

then go ahead and install again.
Once its done, check the version. You should get something like this below,

[email protected]:/etc/bird# docker -v
Docker version 17.09.0-ce, build afdb6d4

once this is done, we are going to install blue for BGP

Installing and Configuring BGP using Bird in Ubuntu 17.10

First, do yourself a favor and add its repo

sudo add-apt-repository ppa:cz.nic-labs/bird
apt-get update -y

once this is done, you'll get an error because it doesn't support Ubuntu 17.10 yet. Hence, gonna go downwards again.

vi /etc/apt/sources.list.d/cz_nic-labs-ubuntu-bird-artful.list

change it to the below,

deb http://ppa.launchpad.net/cz.nic-labs/bird/ubuntu xenial main

save it and try to install bird below,

sudo apt-get install bird

And we have bird installed. Now to configure it, we will paste the following into /etc/bird/bird.conf

log "/var/log/bird" all;

router id 192.168.23.10; # instance ip

protocol device
{
scan time 60;
}

protocol direct
{
interface "dummy1";
}

protocol bgp vultr
{
local as 11512; #instances ASN number
source address 192.168.23.10; #instance ip
import none;
export all;
graceful restart on;
next hop self;
multihop 2;
neighbor 169.254.169.252 as 64019; #vultr IP and ASN number
password "xC4c2EGaXuP7a74!"; #vulr password
}

then proceed to restart

service bird restart
service bird status
● bird.service - BIRD Internet Routing Daemon (IPv4)
   Loaded: loaded (/lib/systemd/system/bird.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-10-30 10:30:51 UTC; 19min ago
  Process: 5140 ExecStartPre=/usr/sbin/bird -p (code=exited, status=0/SUCCESS)
  Process: 5130 ExecStartPre=/usr/lib/bird/prepare-environment (code=exited, status=0/SUCCESS)
 Main PID: 5145 (bird)
    Tasks: 1 (limit: 4915)
   Memory: 640.0K
      CPU: 53ms
   CGroup: /system.slice/bird.service
           └─5145 /usr/sbin/bird -f -u bird -g bird

Oct 30 10:30:51 worker_a systemd[1]: Starting BIRD Internet Routing Daemon (IPv4)...
Oct 30 10:30:51 worker_a systemd[1]: Started BIRD Internet Routing Daemon (IPv4).

now make sure you have the above, if not, its most likely because your log permission is set as root, if that's the case, try the below,

chown bird:bird /var/log/bird

and restart bird again. Ensure your firewall isn't blocking port 179 for BGP to communicate with your machine. Once this is done, you can test it out externally via telnet

telnet 192.168.23.10 179
Trying 192.168.23.10...
Connected to 192.168.23.10.vultr.com.
Escape character is '^]'.
Connection closed by foreign host.

you should get something like this. Now, the most important part if not you will get the following

birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   start  13:49:30    Idle          Received: Bad peer AS
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 0 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              0          0          0        ---          0
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Idle
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Error wait:       10/300
    Last error:       Received: Bad peer AS

Setup Dummy IP Configuration

run the following to create a dummy IP configuration for your Bird to forward to

ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.168.10.10

where 192.168.10.10 is the floating IP. Now you can test whether the above setup is correct with the following

ip addr show dev dummy1
6: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 4e:79:76:36:6d:b0 brd ff:ff:ff:ff:ff:ff
    inet 45.77.110.78/32 scope global dummy1
       valid_lft forever preferred_lft forever
    inet6 fe80::4c79:76ff:fe36:6db0/64 scope link
       valid_lft forever preferred_lft forever

Now, restart your Bird so that it gets the above dummy. To set it persistently, do the following

vi /etc/rc.local
ip link add dev dummy1 type dummy
ip link set dummy1 up
ip addr add dev dummy1 192.168.10.10/32

then restart or reboot your machine

service bird restart

Once this is done, you should get the following,

birdc show proto all vultr
BIRD 1.6.3 ready.
name     proto    table    state  since       info
vultr    BGP      master   up     14:44:15    Established
  Preference:     100
  Input filter:   REJECT
  Output filter:  ACCEPT
  Routes:         0 imported, 1 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              0          0          0          0          0
    Import withdraws:            0          0        ---          0          0
    Export updates:              1          0          0        ---          1
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Established
    Neighbor address: 169.254.169.254
    Neighbor AS:      64515
    Neighbor ID:      208.167.241.119
    Neighbor caps:    refresh restart-aware AS4 add-path-rx
    Session:          external multihop AS4
    Source address:   45.77.78.111
    Hold timer:       132/180
    Keepalive timer:  29/60

Once you get this, all should be good. Try it out.