In my last post, I set up ELK in a docker container to see if it would meet my needs, but I found that unless I wanted to go very deep with docker, I’d need to do a raw installation of ELK. The main motivation was that I wanted to install an SSL certificate in Kibana using LetsEncrypt from my pfSense box, and building a job that builds docker every 90 days seemed brittle.
One thing I realized was that you do do a lot without installing the ‘L’ in ELK. LogStash and ElasticSearch both provide means to ingest logs. When you install filebeat on your client, you can opt to output to LogStash or to ElasticSearch. I went direct to ElasticSearch for now, though I will likely revisit that later on. Here are two resources that discuss that here and here.
Install Elastic Search
There are very good instructions for the basic setup at elastic.com that I used, which I’ll transcribe.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list apt-get update && sudo apt-get install elasticsearch
Then I set up my storage on my ZFS partition (I’ll be able to set a quota on this if I need, thanks to ZFS):
zfs create storage/elastic mkdir /storage/elastic/data mkdir /storage/elastic/logs chown -R elasticsearch:elasticsearch /storage/elastic
And finally I made several changes to the elastic configuration:
vi /etc/elasticsearch/elasticsearch.yml
And set the following (ref here for the network options):
path.data: /storage/elastic/data
path.logs: /storage/elastic/logs
network.host: _site_,_local_
Finally, in order to prevent filling up your system logs with elastic logging what it is being sent, you need to follow the instructions found here to stop logging.
vi /usr/lib/systemd/system/elasticsearch.service
And set:
StandardOutput=null
StandardError=null
Then
systemctl daemon-reload
As before, in order for filebeat to send data to elastic search, you need to in stall the geoip plugin (found here):
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
Once this is done, you can
systemctl enable elasticsearch service elasticsearch start
Remember to open firewall ports as described in my earlier post.
Install Kibana
As before, there are excellent instructions at elastic.com.
apt-get install kibana
Then edit the kibana config to allow it to listen on all interfaces:
vi /etc/kibana/kibana.yml
and set
elasticsearch.url to your server name:9200
server.host: "0.0.0.0"
Systemctl enable kibana service kibana start
You can then go to your server at https://server:5601 and view Kibana.
Filebeat UTC timezone correction
Since I had already configured filebeat, I expected to see data come in, but I was mystified that the system was empty. I drove myself crazy for a while until I found that filebeat was sending all the data over in UTC, putting it in Kibana in the past. Fixing this took 2 steps:
On the host where filebeat is enabled, you need to edit the configuration file for the system module
vi /etc/filebeat/modules.d/system.yml
and un-comment both #var.convert_timezone lines and set it to
var.convert_timezone: true
Restart filebeat:
service filebeat restart
Then, you need to delete the ingestion pipeline on the elastic search server:
curl -XDELETE 'http://elasticsearch:9200/_ingest/pipeline/filebeat-*'
Based on this post.
Once that was done, I was able to see data come in to Kibana happily.
Adding SSL to kibana
Finally, I wanted to add support to my LetsEncrypt job on pfSense to send a certificate to my Kibana instance. Kibana’s certificates are PEM formatted, so you generate them from a p12 file with:
sudo openssl pkcs12 -in kibana.p12 -nokeys -clcerts -passin pass:test1234 | openssl x509 -outform pem -out kibana.crt.pem sudo openssl pkcs12 -in kibana.p12 -nocerts -passin pass:test1234 -passout pass:123456 | openssl pkcs8 -topk8 -inform PEM -passin pass:123456 -outform pem -nocrypt -out kibana_key.pem
Then I created a folder in /usr/share/kibana for certificates:
mkdir /usr/share/kibana/cert
And moved the certs there (and added the following to my cert installation script).
sudo mv unifi.crt.pem /usr/share/kibana/cert/ sudo mv unifi_key.pem /usr/share/kibana/cert/ sudo chown -R kibana:kibana /usr/share/kibana/cert sudo service kibana restart> /dev/null 2>&1
Finally, edit the kibana config to set the server to use certificates:
vi /etc/kibana/kibana.yml
and update the following lines:
server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/cert/kibana.crt.pem
server.ssl.key: /usr/share/kibana/cert/kibana_key.pem
At this point, my installUnifiCertificate.sh script looks like (my unifi andunifi video are all on the same server):
#!/bin/bash #Unifi COntroller Stuff sudo keytool -delete -alias unifi -keystore /var/lib/unifi/keystore -storepass aircontrolenterprise > /dev/null 2>&1 sudo keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/unifi/keystore -srckeystore ~/unifi.p12 -srcstoretype PKCS12 -alias unifi -srcstorepass test1234 > /dev/null 2>&1 #end unifi controller # Video Stuff sudo openssl pkcs12 -in unifi.p12 -nokeys -clcerts -passin pass:test1234 | openssl x509 -outform der -out unifi.crt.der sudo openssl pkcs12 -in unifi.p12 -nocerts -passin pass:test1234 -passout pass:123456 | openssl pkcs8 -topk8 -inform PEM -passin pass:123456 -outform DER -nocrypt -out unifi_key.der sudo mv unifi.crt.der /usr/lib/unifi-video/data/certificates/ufv-server.cert.der sudo mv unifi_key.der /usr/lib/unifi-video/data/certificates/ufv-server.key.der sudo chown -R unifi-video:unifi-video /usr/lib/unifi-video/data/certificates #end video stuff sudo service unifi-video restart >/dev/null 2>&1 sudo service unifi restart> /dev/null 2>&1 #kibana sudo openssl pkcs12 -in unifi.p12 -nokeys -clcerts -passin pass:test1234 | openssl x509 -outform pem -out unifi.crt.pem sudo openssl pkcs12 -in unifi.p12 -nocerts -passin pass:test1234 -passout pass:123456 | openssl pkcs8 -topk8 -inform PEM -passin pass:123456 -outform pem -nocrypt -out unifi_key.pem sudo mv unifi.crt.pem /usr/share/kibana/cert/ sudo mv unifi_key.pem /usr/share/kibana/cert/ sudo chown -R kibana:kibana /usr/share/kibana/cert sudo service kibana restart> /dev/null 2>&1 echo 'Success'
Conclusion
One thing I could not get working was SSL for elasticsearch. I’ll try it again later on, but I suspect it may require a license.
Anyhow, now I have a nice Kibana server that I can use when I set up a little honeypot to test how easily it is to get hacked.
What I’m listening to as I do this: Slayer’s Seasons In The Abyss. I’m reading Louder Than Hell, an oral history of Heavy Metal, which my wife got me last Christmas, and they have a lot on Slayer, which is a band that I never really got into when I was introduced to Heavy Metal. So I’m doing some research.