Fully Automated Web Platform

Introduction

On my resume, you can read that this website is on a Fully Automated Web Platform. I have decided to explain this architecture.

Schema

Architecture

I have 2 KVM Hypervisors on CentOS 7 on online.net which contains the following resources.

Resources

kvm-001

Name CPU RAM Disk system Disk datas OS Utilities
kvm-001 4 16 30 470 CentOS 7 KVM Hypervisors
fw-001 1 2 10 5 CentOS 7 Firewall
rproxy-001 1 1 10 5 CentOS 7 Reverse Proxy
mail-001 1 1 10 30 CentOS 7 Mail Server
www-001 1 2 10 20 CentOS 7 Web Server
gitlab-001 1 3 30 70 CentOS 7 GitLab Server
jenkins-001 1 4 30 70 CentOS 7 Jenkins Server

I know ! I have overallocated my CPU resources ! But it's enough for my needs ;)

kvm-002

Name CPU RAM Disk system Disk datas OS Utilities
kvm-002 4 16 30 870 CentOS 7 KVM Hypervisors
fw-002 1 2 10 5 CentOS 7 Firewall
rproxy-002 1 1 10 5 CentOS 7 Reverse Proxy
monitoring-002 1 5 20 230 CentOS 7 Monitoring Server
logs-002 1 4 20 180 CentOS 7 Logs Server

Technologies

The following technologies are used :

  • CentOS
  • KVM
  • IPTables
  • OpenVPN
  • Postfix
  • Dovecot
  • Amavisd
  • Spamassassin
  • Varnish
  • Haproxy
  • Nginx
  • Pelican
  • Sphinx
  • Nagios
  • Grafana
  • InfluxDB
  • Telegraf
  • Filebeat
  • ELK
  • GitLab
  • Jenkins
  • Ansible
  • Capistrano

There is a lot of technologies ! I'll try to explain the choice of each one.

CentOS

Well, I'll try to not feed the troll. I've made this choice to try to make a coherent and efficient system :

  • CentOS is a very stable system and has a (very) long time support
  • I want to use KVM, especially because one of my server has a software RAID (At that time, VMWare has some troubles with this !). RedHat has a strong affinity with KVM.
  • I really love Ansible and RedHat has also a strong affinity with Ansible.
  • Finally, all the software I used (Jenkins, GitLab, etc ...) has a dedicated CentOS repository.

KVM

I was searching for a virtualization solution (not containerization solution), among them :

  • VMWare : On my resume, you can see that I have worked a lot with VMWare and I love this solution, but, a0s mentioned above, one of my server has a software RAID, so this choice was not possible because I want to stay homogeneous.
  • Openstack : I have a limited number of servers... and I think that a dedicated network (vs a mutual one) is a prerequisites to have.
  • Proxmox : If I have to choose for a KVM solution, I prefer to have a baremetal one.
  • KVM : Meet my needs !

IPTables

As I want to use only one OS, there is no BSD systems on my platform, otherwise Pfsense, Opnsense, even a simple pf configuration could be my choice.

I have think of using IPFire, but finally I prefer to create my own Iptables scripts (in order to improve my knowledge).

OpenVPN

I need to setup a VPN beetween my two KVM servers.

I may have to choose IPSEC with Racoon and Strongswan, but I also need a Mobile VPN and I think that OpenVPN is well adapted to this case !

Postfix - Dovecot - Amavisd - Spamassassin

For my Mail Server, I have choose the classic way :

  • Postfix : to send and receive mails
  • Dovecot : the IMAP part
  • Amavisd : to check viruses
  • Spamassassin : to check SPAM

Varnish

Well, I have made the choice to use only statics websites. I would describe later the technologies I use.

As my websites are only statics, it makes sense to cache them with Varnish. My ratio of cache hits is :

Varnish Hit Ratio

Not 100% ? Yes, I try to update this website and my resume as far as I can, so it happens that I have to clear some items of the cache !

Haproxy

As you can see, this website use the HTTPS protocol :

Blog Nicolas VION - SSL

Varnish doesn't support SSL, so some choices were available to me to offload SSL :

In my work experience, I have sometimes seen Pound used and it ... works, as ... nginx. However, I really love Haproxy which is a wonderful product with a lot of features ! It is also very easy to monitor !

Nginx

All my websites are statics, so I use Nginx to serve them. I don't want to use heavyweight framework based on PHP, ... Also, I really want to put all my websites in the Varnish cache !

To generate my websites, I use :

  • Pelican
  • Sphinx

Pelican

Pelican is a static site generator written in Python.

A lot of these tools exists, like Hugo but I prefer to use Python than Go to build my website. It also use Markdown which is a nice language to write informations and documentation (It's a pure matter of taste) !

Sphinx

Sphinx is a documentation generator in Python too.

Again, a lot of tools exists but I want to keep the Python base.

Nagios - Grafana - InfluxDB - Telegraf

Nagios is the leader product in opensource monitoring.

Grafana is a quite new tool, but very powerful and we can make really awesome dashboard ! I have also thought to use Centreon, especially the given ISO, but I want to separated tools :

  • Nagios : Alerting
  • Grafana : Graphing

To send datas to Grafana, I use the following tools :

  • InfluxDB : A powerful storage metrics engine for Grafana (and others)
  • Telegraf : An agent for Collecting & Reporting Metrics & Data.

ELK - Filebeat

To store and analyze my logs, I use :

  • Elasticsearch : a powerful analysis engine.
  • Logstash : a tool to collect, parse and transform logs.
  • Kibana : an open source data visualization plugin for Elasticsearch

I have used the ELK Stack a lot in my job and this is clearly the best opensource suite to collect and analyze logs.

Gitlab

All my servers and websites are generated by code, so I must have a powerful versioning tool. I have chosen Gitlab because I want to use Git (No SVN, ...).

I already use Github for some stuffs I want public, however, I don't want (yet !) to publish all my codes and I want to keep some control on it !

Jenkins

I have think to use GitLab-CI which is a really nice tool, but ... I don't know a lot about Jenkins, so I have installed it to improve my knowledge.

I have multiples types of jobs :

  • Deploy my infrastructures code : I use Ansible.
  • Deploy my websites code : I use Capistrano.
  • Create an exhaustive inventory : I use Ansible-CMDB.
  • Renew my SSL Certificate : I use Let's Encrypt.
  • Launch sequential security scans : I use Clamav.
  • Building my CV : I use Latex to create my CV.

Ansible

There is a lot of tools that give the possibility to deploy Infrastructures As Code :

  • Ansible : The one I choose. It is written in Python and supported by RedHat. It also doesn't need any agent to be setup !
  • Chef : I use this one at work ! It is a powerful tool, however, we need to install an agent on every server, and the creation of the Chef Server Cluster is really huge !
  • Puppet : Also a good product, same problems as Chef.
  • Salt Stack : Another good product, never used.

Capistrano

At work, we use this tool to deploy websites code. It's a good tool, so I want to use it too !

I recently see that there is some python tool named Fabistrano, maybe I will use it a day :) !

Conclusions

I'm quite satisfied with my platform and I learned a lot :) ! It is now Fully Automated and I can focus on the content ! However, the following part can be improved :

  • Automatic updates : Actually, I use yum-cron to run updates on my servers. I am thinking of using my Jenkins server + Ansible to run them and add some quality process (health check, advanced reporting, pipelines ...)
  • Adding a failover part : Actually, this platform isn't a High Availability Platform (because of cost).
  • Implements more security tools : Like Ossec, Nessus ...
  • Convert all my servers as containers and use Kubernetes : At work, I manage some customer projects with Kubernetes + Docker and I really like it !