glassfish – add certificate to trust store

We changed our SMTP server recently, which is used by apps to send notification emails. The problem was that the certificate it was using was issued by itself, hence when adding it to a mail client, i.e. thunderbird, it asks you to confirm the exception before it could send any mail (it requires STARTTLS).

So, in order for our apps running on a Glasfish 4.1 Server to send emails, we need to add the certificate to the trust store.

When attempting to send an email with the following code

public void sendMail(String to, String subject, String body) {
        try {
            // sets SMTP server properties
            Properties properties = new Properties();
            properties.put("mail.smtp.host", SMTP_HOST);
            properties.put("mail.smtp.port", SMTP_PORT);
            properties.put("mail.smtp.auth", "true");
            properties.put("mail.smtp.starttls.enable", "true");
            Authenticator auth = new Authenticator() {
                public PasswordAuthentication getPasswordAuthentication() {
                    return new PasswordAuthentication(SMTP_USER, SMTP_PASSWORD);
                }
            };
            Session session = Session.getInstance(properties, auth);
            MimeMessage msg = new MimeMessage(session);

            msg.setFrom(new InternetAddress(SMTP_USER));
            InternetAddress[] toAddresses = {new InternetAddress(to)};
            msg.setRecipients(Message.RecipientType.TO, toAddresses);
            msg.setSubject(subject);
            msg.setSentDate(new Date());
            msg.setText(body, "utf-8", "html");

            Transport.send(msg);
        } catch (MessagingException e) {
            String error = String.format("Ocurrió un problema al enviar un CORREO a: %s", to);
            log.error(error, e);
        } catch (Exception e) {
            String error = String.format("Ocurrió un problema al enviar un CORREO a: %s", to);
            log.error(error, e);
        }
    }

The error logged on Glassfish was

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

First get the certificate of the SMTP server. We got it through the web client accessing with Chrome and clicking on the ‘Not Secure’ label, then Certificate | Details | Export… /tmp/smtp.cer

Then we need to import it with the keytool.

To get the path where the trusted certs are stored I ran the command

ps aux | grep glass

and identified the line

-Djavax.net.ssl.trustStore={glassfish}/domains/domain1/config/cacerts.jks

Now it’s time to import the certificate. First do a backup.

cp {glassfish}/domains/domain1/config/cacerts.jks {glassfish}/domains/domain1/config/cacerts.jks.bkp

Then import the certificate

keytool -import -trustcacerts -alias smtp -file /tmp/smtp.cer -keystore {glassfish}/domains/domain1/config/cacerts.jks

done! now you can send emails through this SMTP server.

Anuncios

postgres – btree level 1 not found in index

We have a jira instance which uses the postgres database to store its information. The server had roughly 50 Gb of disk, so it ran out of space before we noticed. We extended the partition to 100 Gb but apparently it was too late.
Jira started complaining about it with an 500 internal server error (Estado HTTP 500 – Could not determine database type. (Conexión rechazada. Verifique que el nombre del Host y el puerto sean correctos y que postmaster este aceptando conexiones TCP/IP.)). Viewing the logs we found Caused by: java.net.ConnectException: Connection refused

We checked our database and database settings for external connections and couldn’t connect.

Then we tried to start the postgres service

sudo service postgresql start

2018-12-04 09:01:59.268 CST [21281] LOG:  database system was interrupted while in recovery at 2018-12-04 08:59:20 CST
2018-12-04 09:01:59.268 CST [21281] HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2018-12-04 09:01:59.631 CST [21281] LOG:  database system was not properly shut down; automatic recovery in progress
2018-12-04 09:01:59.636 CST [21281] LOG:  redo starts at 7/CC020C68
2018-12-04 09:01:59.639 CST [21281] LOG:  record with zero length at 7/CC07E998
2018-12-04 09:01:59.639 CST [21281] LOG:  redo done at 7/CC07E968
2018-12-04 09:01:59.639 CST [21281] LOG:  last completed transaction was at log time 2018-12-03 10:29:49.99078-06
2018-12-04 09:01:59.641 CST [21281] FATAL:  btree level 1 not found in index "19881"

VITALLY IMPORTANT FIRST RESPONSE

I did a backup of the postgres data files

ssh admin@jira.server
sudo tar cvf postgres.bkp.tar /var/lib/postgresql/

Copied it to my local machine

rsync -av admin@jira.server:postgres.bkp.tar .

then installed the same version (series) that is on the server into my local machine.
Since I have ubuntu 18.04 I had to add the postgres repository to install postgres version 9.3.

echo “deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main” | sudo tee /etc/apt/sources.list.d/pgdg.list
wget –quiet -O – https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add –
sudo apt update
sudo apt install postgresql-9.3 postgresql-contrib-9.3

Then I tried to run postgres locally with the backup data

cd /tmp
tar xvf postgres.bkp.tar
sudo su postgres
/usr/lib/postgresql/9.3/bin/postgres -D /tmp/var/lib/postgresql/9.3/main/

it complained about not finding the configuration file so I just copied it from the default dirs

cp /etc/postgresql/10/main/postgresql.conf /tmp/var/lib/postgresql/9.3/main/
cp -R /etc/postgresql/10/main/conf.d/ /tmp/postgres/var/lib/postgresql/9.3/main/

edit the postgresql.conf file and change the data dir appropiately

data_directory = ‘/tmp/var/lib/postgresql/9.3/main’

then I tried again (but this time in single mode) and it reproduced exactly the same error as on server. Great!

postgres@elite-tsj:/tmp$ /usr/lib/postgresql/9.3/bin/postgres --single -P -d 1 -D /tmp/var/lib/postgresql/9.3/main/
2018-12-04 09:01:59.268 CST [21281] LOG:  database system was interrupted while in recovery at 2018-12-04 08:59:20 CST
2018-12-04 09:01:59.268 CST [21281] HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2018-12-04 09:01:59.631 CST [21281] DEBUG:  checkpoint record is at 7/CC025C50
2018-12-04 09:01:59.631 CST [21281] DEBUG:  redo record is at 7/CC020C68; shutdown FALSE
2018-12-04 09:01:59.631 CST [21281] DEBUG:  next transaction ID: 0/9547804; next OID: 227497
2018-12-04 09:01:59.631 CST [21281] DEBUG:  next MultiXactId: 3; next MultiXactOffset: 5
2018-12-04 09:01:59.631 CST [21281] DEBUG:  oldest unfrozen transaction ID: 676, in database 1
2018-12-04 09:01:59.631 CST [21281] DEBUG:  oldest MultiXactId: 1, in database 1
2018-12-04 09:01:59.631 CST [21281] DEBUG:  transaction ID wrap limit is 2147484323, limited by database with OID 1
2018-12-04 09:01:59.631 CST [21281] DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 1
2018-12-04 09:01:59.631 CST [21281] LOG:  database system was not properly shut down; automatic recovery in progress
2018-12-04 09:01:59.634 CST [21281] DEBUG:  resetting unlogged relations: cleanup 1 init 0
2018-12-04 09:01:59.636 CST [21281] LOG:  redo starts at 7/CC020C68
2018-12-04 09:01:59.639 CST [21281] LOG:  record with zero length at 7/CC07E998
2018-12-04 09:01:59.639 CST [21281] LOG:  redo done at 7/CC07E968
2018-12-04 09:01:59.639 CST [21281] LOG:  last completed transaction was at log time 2018-12-03 10:29:49.99078-06
2018-12-04 09:01:59.639 CST [21281] DEBUG:  resetting unlogged relations: cleanup 0 init 1
2018-12-04 09:01:59.641 CST [21281] FATAL:  btree level 1 not found in index "19881"

since this data is a copy i just executed pg_resetxlog as shown here.
Be aware that use of the pg_resetxlog utility should be done as an absolute last resort, and there are still some things you should try first..

/usr/lib/postgresql/9.3/bin/pg_resetxlog /tmp/var/lib/postgresql/9.3/main/

it complained I had to force it, there you go

/usr/lib/postgresql/9.3/bin/pg_resetxlog -f /tmp/var/lib/postgresql/9.3/main/

done! Now let’s see if it’s working again

sudo su postgres
/usr/lib/postgresql/9.3/bin/postgres -D /tmp/var/lib/postgresql/9.3/main/

boom! it’s working.

Now it’s time to do a backup with sql dump (pg_dump) or better yet a full sql dump (pg_dumpall).

In another terminal (or tab)

sudo su postgres
pg_dumpall -p 5433 > /tmp/postgres-full.dmp

in my case I had to specify the port (-p 5433).

Now that I have a full backup I can try the same on production.

/usr/lib/postgresql/9.3/bin/pg_resetxlog -f /var/lib/postgresql/9.3/main/
sudo service postgresql start

thank the gods! (to recall the option pointed here)

enable ssh session using keys instead of password in dokku

ssh

We have a Dokku server installation. You can run remote commands to ease up a little bit the administration tasks.
So, to avoid the need to logon the server and run some dokku commands I needed to configure ssh keys.
First, let’s create our keys -if you haven’t yet-

ssh-keygen -t rsa

let’s copy our public key to the clipboard.

cat ~/.ssh/id_rsa.pub | xclip -selection clipboard

you can add the second command to an alias in the file ~/.bash_aliases if you wish

alias cb=’xclip -selection clipboard’

so the previous command would be

cat ~/.ssh/id_rsa.pub | cb

then we can paste our public key to the authorized clients on the server

ssh joe@dokku.me
sudo su
cd /home/dokku/.ssh
cat >> authorized_keys
press CTRL-SHIFT-V
press CTRL-C

you must prepend your public key (your key starts with the type, ssh-rsa in this case) in the authorized_keys file with the following configuration (this should have been done when installing dokku)

command=”FINGERPRINT=SHA256:qriqw51R00ha3WHe5Zo987APhwxolya3Sv5Dvwm/fqE NAME=\”admin1\” `cat /home/dokku/.sshcommand` $SSH_ORIGINAL_COMMAND”,no-agent-forwarding,no-user-rc,no-X11-forwarding,no-port-forwarding ssh-rsa AAAAB3Nza…

done!

now you can exit the server and run a remote command from your machine

ssh -t dokku@dokku.me apps:list

ubuntu – from 16.04 to 18.04

ubuntu bionic beaver logo

I used Ubuntu 16.04 for as long as I could. It turns out that Dropbox sent me a notification about dropping support for file systems different than ext4.

your Dropbox folder will stop syncing because it’s on a file system or partition that no longer meets the requirements

I thougth I was using ext4 but, since I had enabled encryption on my home folder, in reality I had ecryptfs. So I needed to use another partition if I’d like to continue using Dropbox.
I thought it wouldn’t be so hard to resize my primary partition -since I used LVM- and create another partition with ext4. Well I tried system-config-lvm and kvpm but they both required to boot from a liveusb or livecd distribution since the partition I wanted to resize was my root partition.

So I decided it would be easier, and perhaps better, for me to just upgrade to ubuntu 18.04 LTS.

I wasn’t wrong, the new version feels smoother, faster and a little bit cleaner.

The only hiccups I had were

  • no fullscreen apps (hide menu bar when maximized)
    Although you can install this gnome extension to enable this behavior.
  • HUD is gone
    I used it with gimp a lot -crop, resize, any menu command available just pressing ALT-. Well, there’s also a gnome extension for this.
  • hot corners (to change between applications pointing the mouse to a corner or CMD+W)
    You can use a gnome extension called custom corner as explained in this article.
  • adding custom launchers to dock
  • it doesn’t have a default backup manager (you can install deja-dup as it was the default before)
    > sudo apt install deja-dup
  • i needed support for exfat (it’s where I placed my last backup)
    > sudo apt install exfat-fuse exfat-utils

well, so far I’m very happy with the upgrade.

ubuntu 16.04 – switch from yarn back to npm again

Recently I’ve upgraded jhipster using yarn, but it caused me some issues running a newly created application. I don’t think the upgrade caused the problem but yarn and the way I installed node.

Time ago I installed node, building it from source to have the latest LTS version. Afterwards, I installed yarn following their instructions and adding their repo. In between creating and upgrading projects, something must have gone wrong and I ended up with different incompatible versions of npm installed. Which led to the issue described above.

if i did a

node -v
v8.12.0

but

npm -v
2.5.2

I checked where did it came from

which npm
~/.config/yarn/global/node_modules/npm/bin/npm

but i never installed (so i remember) npm through yarn.

I tried removing it with

yarn global remove npm
error This module isn’t specified in a package.json file.

I decided to clean up all the mess and start over again.
First I had to uninstall node (remember I built it from source)

cd ~/Projects/opensource/node
sudo make uninstall

Then remove yarn

sudo apt remove yarn

optionally you could delete all the cache and configuration files

rm -rf ~/.config/yarn

Now we’re ready to install again, but this time it’s easier to add their repos and install and update from them.

install node (LTS)

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash –
sudo apt-get install -y nodejs
node -v
v8.12.0
npm -v
6.4.1
which npm
/usr/bin/npm

Since jhipster now recommends npm as default, that’s it.

There’s a lot of discussion about npm or yarn, like this on reddit or this article

optionally install yarn

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add –
echo “deb https://dl.yarnpkg.com/debian/ stable main” | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn

yarn global upgrade jhipster doesn’t update

I was trying to upgrade jhipster to its latest version in order to update an application that was generated by a previous version of the generator.

Since I use yarn for package management (although it seems i will be getting back to npm), I just tried

yarn global upgrade generator-jhipster

which let me with the same version I had before

jhipster -v
4.10.2

I found the answer here

Keep in mind that upgrade respects semver range specified in package.json. I believe the default location on OSX is ~/.config/yarn/global/package.json

So yarn global add some-package would add a carret-range, something like ^1.0.0 so if a 2.0.0 comes out, yarn global upgrade would not upgrade to that because it doesn’t fit the range. You would specify the –latest/-L flag to ignore the semver range and get the latest as tagged in the registry.

so, in order to update jhipster run the following command

yarn global upgrade generator-jhipster –latest

dokku – move wordpress site with duplicator

wp-dokku

about dokku

Docker powered mini-Heroku. The smallest PaaS implementation you’ve ever seen.

Dokku allow us to run an app inside docker containers and scale easily -just like heroku does-

We have a microsite made with wordpress which we wanted to move inside dokku. The first thing you need is to install duplicator in your original website, create a package and build it, download the installer.php and the zip file.

We’ll use the dokku-wordpress script to generate the required files. It has the following requirements

A Dokku server. Tested on 0.7.2+
The dokku-mysql plugin
make installed locally
curl or wget installed locally

To install the dokku-mysql plugin just ssh into the dokku server and run

sudo dokku plugin:install https://github.com/dokku/dokku-mysql.git mysql

Then clone the dokku-wordpress repository to your local machine, run the command and follow the instructions that are printed in the command line from the last command and take note of the DATABASE_URL when you link the database container

git clone https://github.com/dokku-community/dokku-wordpress.git
cd dokku-wordpress
make build APP_NAME=testapp SERVER_NAME=<ip or name>

# run the following commands on the server to setup the app:

dokku apps:create testapp

# setup plugins persistent storage

mkdir -p /var/lib/dokku/data/storage/testapp-plugins
chown 32767:32767 /var/lib/dokku/data/storage/testapp-plugins
dokku storage:mount testapp /var/lib/dokku/data/storage/testapp-plugins:/app/wp-content/plugins

# setup upload persistent storage

mkdir -p /var/lib/dokku/data/storage/testapp-uploads
chown 32767:32767 /var/lib/dokku/data/storage/testapp-uploads
dokku storage:mount testapp /var/lib/dokku/data/storage/testapp-uploads:/app/wp-content/uploads

# setup your mysql database and link it to your app

export MYSQL_IMAGE_VERSION="5.6"
dokku mysql:create testapp-database
dokku mysql:link testapp-database testapp

# you will also need to set the proper environment variables for keys and salts
# the following were generated using the wordpress salt api: https://api.wordpress.org/secret-key/1.1/salt/
# and use the following commands to set them up:

dokku config:set testapp AUTH_KEY='Q<z4;VJkx#Jd;tV^d?ex^jwM^z/|b.$w,f4x[/vgCSVXKhZ|aO1p,'
dokku config:set testapp SECURE_AUTH_KEY='>zR1uL02;5+pDRA%HEz!~ty])hV^@;-s-`-rk`W+)zfdeWQ&e=yZM.j)M0Q4}xk1'
dokku config:set testapp LOGGED_IN_KEY=':5W#<&PO*_B7<Cj?7aLrbq|8PoF?2ZR.?_DAki*e3Oh!+RM@Ga@q6:L~(p=!D,SyHwa$B5.Fk'
dokku config:set testapp LOGGED_IN_SALT='xv$Lq]/6bowHaMF@Eff5fL/ab,-pu,P,Jh4U{uiIE~>mog<.O9gDW2+!IOo$M*='
dokku config:set testapp NONCE_SALT='dj8^5?t=V-,nV7}-{*bnz7b7vb.=N1:~BrMC`-D{GLl(k[+}!hh#D[tk#TcG;7|'

# now, on your local machine, change directory to your new wordpress app, and push it up

cd testapp
git push dokku master

then you'll have a vanilla wordpress deployed in dokku.

You should have a directory generated in the cloned folder named after your app, get into it and delete all the files except the following (notice there are hidden files)

.buildpacks
composer.json
composer.lock
.git

copy the files downloaded from duplicator inside this folder and deploy doing a git push dokku

open your app's url pointing to installer.php file and follow the instructions. Use the values from the DATABASE_URL generated earlier, if you didn't take note, you can get it by running the command

ssh -t dokku@<ip or name> config <appname>

use that values for the DB configuration step and continue with the setup.
for example if you have the following DATABASE_URL

mysql://mysql:7ab9b1b45db2232c@dokku-mysql-testapp-database:3306/testapp-database
the corresponding values would be

user=mysql
pass=7ab9b1b45db2232c
server=dokku-mysql-testapp-database
database=testapp-database

done! you should have cloned successfully your wordpress site into dokku.