Firewall deployment

File location

When starting a standalone firewall (in bridge, or float mode), the corda-firewall.jar file defaults to reading the firewall’s configuration from a firewall.conf file in the directory from which the command to launch the process is executed. The syntax is:

corda-firewall [-hvV] [--install-shell-extensions]
               [--logging-level=<loggingLevel>] [-b=<baseDirectory>]
               [-f=<_configFile>]

Where:

• --config-file, -f: Allows you to specify a configuration file with a different name, or at a different file location. Paths are relative to the current working directory
• --base-directory, -b: Allows you to specify the firewall’s workspace location. A firewall.conf configuration file is then expected in the root of this workspace
• --verbose, --log-to-console, -v: If set, prints logging to the console as well as to a file.
• --logging-level=<loggingLevel>: Enable logging at this level and higher. Possible values: ERROR, WARN, INFO, DEBUG, TRACE. Default: INFO.
• --install-shell-extensions: Install corda-firewall alias and auto completion for bash and zsh. See cli-application-shell-extensions for more info.
• --help, -h: Show this help message and exit.
• --version, -V: Print version information and exit.

Format

The firewall configuration file uses the HOCON format which is superset of JSON. Please visit https://github.com/typesafehub/config/blob/master/HOCON.md for further details.

Defaults

A set of default configuration options are loaded from the built-in resource file. Any options you do not specify in your own firewall.conf file will use these defaults:

healthCheck = true
keyStorePassword = "cordacadevpass"
trustStorePassword = "trustpass"
enableAMQPPacketTrace = false
artemisReconnectionIntervalMin = 5000
artemisReconnectionIntervalMax = 60000
politeShutdownPeriod = 1000
p2pConfirmationWindowSize = 1048576
auditServiceConfiguration : {
  loggingIntervalSec = 60
}
silencedIPs = []
useProxyForCrls = true

firewalldefault_latest.conf

Firewall operating modes

• SenderReceiver: selects a single process firewall solution to isolate the node and Artemis broker from direct Internet contact. It is still assumed that the firewall process is behind a firewall, but both the message sending and receiving paths will pass via the bridge. In this mode the outboundConfig and inboundConfig configuration sections of firewall.conf must be provided, the bridgeInnerConfig and floatOuterConfig sections should not be present.

• BridgeInner: mode runs this instance of the corda-firewall.jar as the trusted portion of the peer-to-peer firewall float. Specifically, this process runs the complete outbound message processing. For the inbound path it operates only the filtering and durable storing portions of the message processing. The process expects to connect through a firewall to a matched FloatOuter instance running in the DMZ as the actual TLS 1.2/AMQP 1.0 termination point.

• FloatOuter: causes this instance of the corda-firewall.jar to run as a protocol break proxy for inbound message path. The process will initialise a TLS control port and await connection from the BridgeInner. Once the control connection is successful the BridgeInner will securely provision the TLS socket server key and certificates into the FloatOuter. The process will then start listening for inbound connection from peer nodes.

Fields

The configuration fields are listed in Corda Enterprise Firewall configuration fields ).

Complete example

As an example to show all features, the following is a walk-through of the configuration steps to set-up a pair of HA hot-cold nodes for two separate legal identities, connected to by a HA hot-warm set of BridgeInner and FloatOuter that use some simple certificates to secure the control tunnel and a SOCKS5 proxy for outgoing connectivity (see diagram). This is also the recommended full enterprise deployment pattern, although there are plenty of alternative deployment options.

Conceptually deployment will be done as follows:

The following diagram illustrates physical deployment of the example setup discussed above:

The Float instances run on vmFloat1 and vmFloat2 which are located in the DMZ.

The SOCKS5 proxy is running on vmSocks which also resides in the DMZ.

Each of the vmInfra1 and vmInfra2 computers host: ZooKeeper cluster participant, Bridge instance and Artemis cluster participant:

Keystores generation

A special tool was created to simplify generation of the keystores. For more information please see HA Utilities. This section explains how to generate a number of internally used keystores. Commands below can be executed on any machine as long as it will be easy enough to copy results to the other machines including DMZ hosts.

It is also advisable to create an application user account (say corda) and use it instead of using own personal user.

Tunnel keystore generation

For Float and Bridge to communicate a tunnel keystore has to be created as follows:

java -jar corda-tools-ha-utilities-4.9.jar generate-internal-tunnel-ssl-keystores -p tunnelStorePass -e tunnelPrivateKeyPassword -t tunnelTrustpass

This should produce files: tunnel/float.jks, tunnel/tunnel-truststore.jks and tunnel/bridge.jks which will be used later on.

Artemis keystore generation

Bridge communicates to Artemis which requires a separate keystore.

Due to Artemis limitations the password for the keystore should be the same as the password for the private keys in the store. The tool below caters for these arrangements. Artemis trust password can and should be different.

The tool should be used as follows:

java -jar corda-tools-ha-utilities-4.9.jar generate-internal-artemis-ssl-keystores -p artemisStorePass -t artemisTrustpass

This should produce files: artemis/artemis-truststore.jks, artemis/artemis.jks which will be used later on.

Node VMs setup

As shown on the Physical deployment diagram above there will be two separate machines in two distinct data centres hosting Corda Nodes for Legal Entity A and Legal Entity B. For this setup, each machine is powerful enough to host nodes for both entities with all the CorDapps and two datacentres are used for High Availability purposes.

Prerequisites

Corda Network connectivity

Before nodes can be configured, Corda Network administrator will need to provide:

• Network root trust store file: network-root-truststore.jks and password for it in this example assumed to be trustpass;
• Corda Network URL for Doorman e.g.: http://r3-doorman:10001;
• Corda Network URL for NetworkMap e.g.: http://r3-netman:10001

Nodes inbound connectivity provisions

In order for the nodes for both legal entities Entity A and Entity B to be reached from the outside of the organisation by other nodes, a single TCP endpoint address is being exposed.

From infrastructure point of view this can be address of a load balancer which will be routing network traffic to vmFloat1 and vmFloat2 hosts in the DMZ.

Out of vmFloat1 and vmFloat2 there will be at most one active host which will be accepting incoming communication. Therefore, load balancer will route inbound traffic to vmFloat1 or vmFloat2.

Databases setup

Each legal entity is supposed to have it is own database(DB) schema in order to store Corda transaction data. Therefore Entity A and Entity B should have different DB connectivity URLs.

For nodes’ High Availability(HA) functionality to work properly, databases the nodes connect to should be remote databases with transactional guarantees. Please see Hot-cold high availability deployment. I.e. HA nodes cannot be using local H2 database.

In the example below we will be using Azure SQL DB, however it can be any database Corda Enterprise supports.

Two empty schemas should be created for Entity A and Entity B and upon first startup of the node the necessary tables will be created automatically.

Base directory setup

Initially, the nodes configuration is performed on vmNodesPrimary host and then there is a special paragraph that details vmNodesSecondary setup.

Files artemis/artemis.jks and artemis/artemis-truststore.jks should be copied from Artemis keystore generation stage.

Corda FAT Jar corda.jar from Corda Enterprise distribution should also be copied into base directory.

Any CorDapps the node is meant to be working with should be installed into cordapps directory.

Creating node configuration files

Since there will be two distinct nodes serving two different legal entities they are meant to have two difference X.500 names, please see myLegalName field in the config files below.

Also these two separate node may have different passwords to protected their keystore (keyStorePassword) and their trust store (trustStorePassword).

Suggested configuration for node serving Entity A on vmNodesPrimary would be a entityA/node.conf files containing:

myLegalName = "O=Entity A,L=London,C=GB"
p2pAddress = "banka.com:10005" // Host and port exposed by Internet facing firewall/load balancer in front of float servers in DMZ.
messagingServerAddress = "vmInfra1:11005" // Specifying endpoints of remote Artemis instances, Note: SWAP1
messagingServerExternal = true // Specifying that it is an external instance
// Public keystore settings
keyStorePassword = "entityAStorePass"
trustStorePassword = "entityATrustPass"
// RPC settings
rpcSettings {
    address = "0.0.0.0:10006"
    adminAddress = "0.0.0.0:10026"
}
dataSourceProperties { // Point at clustered Azure SQL Server
    dataSourceClassName = "com.microsoft.sqlserver.jdbc.SQLServerDataSource"
    dataSource.url = "jdbc:sqlserver://entityAdb.database.windows.net:1433;databaseName=corda;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30"
    dataSource.user = Corda
    dataSource.password = password
}
database {
    schema = dbo
}
security {
    authService {
        dataSource {
            type = INMEMORY
            users = [
                {
                    password = password
                    permissions = [
                        ALL
                    ]
                    username=user
                }
            ]
        }
    }
}
useTestClock = false
enterpriseConfiguration = {
    externalBridge = true // Ensure node doesn't run P2P AMQP bridge, instead delegate to the BridgeInner.
    messagingServerConnectionConfiguration = "CONTINUOUS_RETRY"
    messagingServerBackupAddresses = ["vmInfra2:11005"] // See "messagingServerAddress" above, Note: SWAP1
    mutualExclusionConfiguration = { // Enable the protective heartbeat logic so that only one node instance is ever running.
        on = true
        updateInterval = 20000
        waitInterval = 40000
    }
    messagingServerSslConfiguration = {
                sslKeystore = artemis/artemis.jks
                keyStorePassword = artemisStorePass
                trustStoreFile = artemis/artemis-truststore.jks
                trustStorePassword = artemisTrustpass
    }
}
networkServices {
    doormanURL = "http://r3-doorman:10001"
    networkMapURL = "http://r3-netman:10001"
}
devMode = false // Turn off things like key autogeneration and require proper doorman registration.
detectPublicIp = false // Do not perform any public IP lookup on the host.
sshd {
    port = 2222
}

For “sibling” node serving Entity B on vmNodesPrimary would be a entityB/node.conf file containing:

myLegalName = "O=Entity B,L=London,C=GB"
p2pAddress = "banka.com:10005" // Host and port exposed by Internet facing firewall/load balancer in front of float servers in DMZ.
messagingServerAddress = "vmInfra1:11005" // Specifying endpoints of remote Artemis instances, Note: SWAP1
messagingServerExternal = true // Specifying that it is an external instance
// Public keystore settings
keyStorePassword = "entityBStorePass"
trustStorePassword = "entityBTrustPass"
// RPC settings
rpcSettings {
    address = "0.0.0.0:10106"
    adminAddress = "0.0.0.0:10126"
}
dataSourceProperties { // Point at clustered Azure SQL Server
    dataSourceClassName = "com.microsoft.sqlserver.jdbc.SQLServerDataSource"
    dataSource.url = "jdbc:sqlserver://entityAdb.database.windows.net:1433;databaseName=corda;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30"
    dataSource.user = Corda
    dataSource.password = password
}
database {
    schema = dbo
}
security {
    authService {
        dataSource {
            type = INMEMORY
            users = [
                {
                    password = password
                    permissions = [
                        ALL
                    ]
                    username=user
                }
            ]
        }
    }
}
useTestClock = false
enterpriseConfiguration = {
    externalBridge = true // Ensure node doesn't run P2P AMQP bridge, instead delegate to the BridgeInner.
    messagingServerConnectionConfiguration = "CONTINUOUS_RETRY"
    messagingServerBackupAddresses = ["vmInfra2:11005"] // See "messagingServerAddress" above, Note: SWAP1
    mutualExclusionConfiguration = { // Enable the protective heartbeat logic so that only one node instance is ever running.
        on = true
        updateInterval = 20000
        waitInterval = 40000
    }
    messagingServerSslConfiguration = {
                sslKeystore = artemis/artemis.jks
                keyStorePassword = artemisStorePass
                trustStoreFile = artemis/artemis-truststore.jks
                trustStorePassword = artemisTrustpass
    }
}
networkServices {
    doormanURL = "http://r3-doorman:10001"
    networkMapURL = "http://r3-netman:10001"
}
devMode = false // Turn off things like key autogeneration and require proper doorman registration.
detectPublicIp = false // Do not perform any public IP lookup on the host.
sshd {
    port = 2223
}

Nodes keystores generation

Given two configuration files above, in order to produce node keystores the following command should be used:

java -jar corda-tools-ha-utilities-4.9.jar node-registration --config-files=./entityA/node.conf --config-files=./entityB/node.conf --network-root-truststore=network-root-truststore.jks --network-root-truststore-password=trustpass

This call will process node.conf files and for each legal name performs Doorman registration. Depending on Corda Network configuration this process may require manual approval and the program will poll for for Certification Signing Request(CSR) completion. For more information see Joining a compatibility zone.

After successful execution this will produce two directories entityA/certificates and entityB/certificates containing the following files:

• truststore.jks, the network/zone operator’s root certificate in keystore with a locally configurable password as protection against certain attacks;
• nodekeystore.jks, which stores the node’s identity key pairs and certificates;
• sslkeystore.jks, which stores the node’s TLS key pair and its certificate.

These are the keystores that will be used by each of the nodes.

Also, file called network-parameters will be produced which represents global parameters for this Corda Network.

-Dcapsule.jvm.args="-Dhttp.proxyHost=10.0.0.100 -Dhttp.proxyPort=8800 -Dhttps.proxyHost=10.0.0.100 -Dhttps.proxyPort=8800"

.
├── artemis
│   ├── artemis.jks
│   └── artemis-truststore.jks
├── certificates
│   ├── nodekeystore.jks
│   ├── sslkeystore.jks
│   └── truststore.jks
├── corda.jar
├── network-parameters
└── node.conf

CorDapps installation

In the node’s base directory create cordapps sub-directory and install all the required CorDapps you intend to work with. In this example we are going to use Finance CorDapp which is supplied as part of Corda Enterprise distribution.

DB drivers installation

As discussed above each of the nodes will be using database to store node’s data. Corda Enterprise supports a number of databases, however in order for a Corda Node to store its data in the DB, a JDBC driver needs to be installed into drivers sub-directory.

In this example we are using MSSql Server DB, therefore mssql-jdbc-6.4.0.jre8.jar will be installed.

Keystore aggregation for the Bridge

Since there is a single Bridge instance representing multiple nodes, it will need to have an aggregated SSL keystore representing all the nodes. In order to produce such aggregated keystore, the following command should be used:

java -jar corda-tools-ha-utilities-4.9.jar import-ssl-key --bridge-keystore-password=bridgeKeyStorePassword --bridge-keystore=./nodesCertificates/nodesUnitedSslKeystore.jks --node-keystores=./entityA/certificates/sslkeystore.jks --node-keystore-passwords=entityAStorePass --node-keystores=./entityB/certificates/sslkeystore.jks --node-keystore-passwords=entityBStorePass

As a result ./nodesCertificates/nodesUnitedSslKeystore.jks file will be produced containing 2 entries.

vmNodeSecondary setup

vmNodeSecondary is supposed to be an almost exact replica of vmNodesPrimary host. The only difference between those two machines that they are residing in two different data centres for BCP purposes.

Since all the keystores been already created on vmNodesPrimary and nodes X.500 names been registered with Doorman, all it takes to clone vmNodesPrimary setup onto vmNodesSecondary is to copy base directories with all the files recursively for Entity A and Entity B.

The only thing in node configuration files that ought to be changed is configuration of Artemis connectivity. This is needed to ensure DataCentre locality whenever possible.

See SWAP1 note in the node.conf files. For both nodes (Entity A and Entity B) configured on vmNodeSecondary:

• messagingServerAddress should be set to vmInfra2:11005 ;
• enterpriseConfiguration.messagingServerBackupAddresses should be set to ["vmInfra1:11005"].

Float VMs setup

It is advisable for each of the hosts vmFloat1 and vmFloat2 to be dual homed machines with two network interfaces. However, this is not mandated. Addresses vmFloat1-int and vmFloat2-int are exposed to the internal trusted zone only. The externally accessible addresses of the DMZ servers are vmFloat1-ext and vmFloat2-ext, which the Internet facing firewall/load balancer maps to banka.com.

Each of the vmFloat1 and vmFloat2 should have a base directory where Corda Firewall will be installed.

Configuration file: firewall.conf for vmFloat1 should look as follows:

firewallMode = FloatOuter
inboundConfig {
    listeningAddress = "vmFloat1-ext:10005" // NB: Replace with "vmFloat2-ext:10005" on vmFloat2 host
}
floatOuterConfig {
    floatAddress = "vmFloat1-int:12005" // NB: Replace with "vmFloat2-int:12005" on vmFloat2 host
    expectedCertificateSubject = "CN=bridge, O=Corda, L=London, C=GB" // This X.500 name must align with name that Bridge received at the time of internal certificates generation.
    tunnelSSLConfiguration {
        keyStorePassword = "tunnelStorePass"
        keyStorePrivateKeyPassword = "tunnelPrivateKeyPassword"
        trustStorePassword = "tunnelTrustpass"
        sslKeystore = "./tunnel/float.jks"
        trustStoreFile = "./tunnel/tunnel-truststore.jks"
    }
}
healthCheckPhrase = "HelloCorda"

Files tunnel/float.jks and tunnel/tunnel-truststore.jks should be copied from Tunnel keystore generation stage. network-parameters file should be copied from one of the node hosts, which has already been produced from Nodes keystores generation stage.

corda-firewall.jar is included into Corda Enterprise distribution and should be copied into base directory on vmFloat1 and vmFloat2 hosts.

For reference, base directory for vmFloat1 and vmFloat2 should look as follows:

.
├── corda-firewall.jar
├── firewall.conf
├── network-parameters
└── tunnel
    ├── float.jks
    └── tunnel-truststore.jks

Infra VMs setup

vmInfra1 and vmInfra2 are hosting infrastructural processes which enable nodes to perform in/out communication with the rest of Corda Network.

The following process will be hosted by each of the VMs:

• Apache ZooKeeper cluster participant;
• Bridge instance;
• Artemis cluster participant.

Apache ZooKeeper setup

Apache ZooKeeper(ZK) is needed to facilitate leader election among two hot-warm Bridge Instances. We require using version 3.5.4-beta and have 3 cluster participants which will be hosted on vmInfra1, vmInfra2 and vmZkWitness.

Assuming /opt/corda is the base directory for ZK instance on vmInfra1 the following files needs to be created:

• config/zoo1/zoo.cfg containing:

dataDir=/opt/corda/config/zoo1/
syncLimit=2
initLimit=5
tickTime=2000
dynamicConfigFile=/opt/corda/config/zoo1/zoo.cfg.dynamic

On vmInfra2 zoo1 should be replaced with zoo2. On vmZkWitness zoo1 should be replaced with zoo3

• config/zoo1/zoo.cfg.dynamic containing:

server.1=vmInfra1:4000:4001:participant;10.155.0.5:11105
server.2=vmInfra2:4000:4001:participant;10.155.0.6:11105
server.3=vmZkWitness:4000:4001:participant;10.155.0.7:11105

10.155.0.x are assumed to be trusted zone corresponding IP addresses of each of the VMs. The content of this file is identical across all 3 hosts: vmInfra1, vmInfra2 and vmZkWitness.

• config/zoo1/myid containing:

1

On vmInfra2 it should contain 2.

On vmZkWitness it should contain 3.

Bridge instances setup

Base directory for Bridge instance should be created on each of the vmInfra1 and vmInfra2 hosts.

File copy from previous stages:

• Files tunnel/bridge.jks and tunnel/tunnel-truststore.jks should be copied from Tunnel keystore generation stage.
• Files artemis/artemis.jks and artemis/artemis-truststore.jks should be copied from Artemis keystore generation stage.
• File nodesCertificates/nodesUnitedSslKeystore.jks should be copied from Keystore aggregation for the Bridge stage.
• File network-root-truststore.jks along with the password to read this keystore provided by the CN owner.
• File network-parameters should be copied from Nodes keystores generation stage.

corda-firewall.jar is included into Corda Enterprise distribution and should be copied into Bridge base directory on vmInfra1 and vmInfra2 hosts.

Configuration file: firewall.conf for vmInfra1 should look as follows:

firewallMode = BridgeInner

// Public SSL settings
keyStorePassword = "bridgeKeyStorePassword"
sslKeystore = "nodesCertificates/nodesUnitedSslKeystore.jks"
trustStorePassword = "trustpass"
trustStoreFile = "nodesCertificates/network-root-truststore.jks"

outboundConfig {
    artemisBrokerAddress = "vmInfra1:11005" // NB: for vmInfra2 swap artemisBrokerAddress and alternateArtemisBrokerAddresses. Note: SWAP2
    alternateArtemisBrokerAddresses = ["vmInfra2:11005"] // Note: SWAP2
    proxyConfig {
       version = SOCKS5
       proxyAddress = "vmSocks:1080"
       userName = "proxyuser"
       password = "password"
    }

    artemisSSLConfiguration {
        keyStorePassword = "artemisStorePass"
        trustStorePassword = "artemisTrustpass"
        sslKeystore = "artemis/artemis.jks"
        trustStoreFile = "artemis/artemis-truststore.jks"
    }
}
bridgeInnerConfig {
    floatAddresses = ["vmFloat1:12005", "vmFloat2:12005"] // NB: for vmInfra2 change the ordering. Note: SWAP3
    expectedCertificateSubject = "CN=float, O=Corda, L=London, C=GB" // This X.500 name should match to the name of the Float component which was used during Tunnel keystore generation above.
    tunnelSSLConfiguration {
        keyStorePassword = "tunnelStorePass"
        keyStorePrivateKeyPassword = "tunnelPrivateKeyPassword"
        trustStorePassword = "tunnelTrustpass"
        sslKeystore = "./tunnel/bridge.jks"
        trustStoreFile = "./tunnel/tunnel-truststore.jks"
    }
}
haConfig {
    haConnectionString = "zk://vmInfra1:11105,zk://vmInfra2:11105,zk://vmZkWitness:11105" // NB: for vmInfra2 change the ordering. Note: SWAP4
}
networkParametersPath = network-parameters // The network-parameters file is expected to be copied from the node registration phase and here is expected in the workspace folder.

For reference, base directory for the Bridge instance should look as follows:

.
├── artemis
│   ├── artemis.jks
│   └── artemis-truststore.jks
├── corda-firewall.jar
├── firewall.conf
├── network-parameters
├── nodesCertificates
│   ├── nodesUnitedSslKeystore.jks
│   └── network-root-truststore.jks
└── tunnel
    ├── bridge.jks
    └── tunnel-truststore.jks

Artemis cluster participant

Artemis will be deployed as a standalone process cluster and will be used as a communication bus for multiple applications(nodes and bridges). The required configuration files can be easily generated using the ha-utilities command line tool. The tool can also install a configured Artemis instance provided that a distribution already exists. For the purpose of this example, commands are provided to use the ha-utilities to install and configure 2 Artemis instances in HA mode.

ha-utilities with configure-artemis option will create two configurations for two processes known as master and slave. For more information please see: Artemis HA Documentation

Apache Artemis distribution can be downloaded from here.

File copy from previous stages:

• Files artemis/artemis.jks and artemis/artemis-truststore.jks should be copied from Artemis keystore generation stage.

vmInfra1 box will host Artemis master instance. To generate application distribution with the config files, please run:

java -jar corda-tools-ha-utilities-4.9.jar configure-artemis --install --distribution ${ARTEMIS_DISTRIBUTION_DIR} --path ${WORKING_DIR}/artemis-master --user "CN=artemis, O=Corda, L=London, C=GB" --ha MASTER --acceptor-address vmInfra1:11005 --keystore ./artemis/artemis.jks --keystore-password artemisStorePass --truststore ./artemis/artemis-truststore.jks --truststore-password artemisTrustpass --connectors vmInfra1:11005,vmInfra2:11005

Where ARTEMIS_DISTRIBUTION_DIR - is the path to the directory where Artemis was downloaded and extracted. Example: /home/apache-artemis-2.6.3

Files artemis/artemis.jks and artemis/artemis-truststore.jks from Artemis keystore generation stage need to be copied into ${WORKING_DIR}/artemis-master/etc/artemis.

vmInfra2 setup

vmInfra2 Artemis cluster participant setup

Repeat steps from Artemis cluster participant section for WORKING_DIR creation as well as keystores copy and Apache Artemis distribution download.

vmInfra2 box will host Artemis slave instance. To generate application distribution with the config files, please run:

java -jar corda-tools-ha-utilities-4.9.jar configure-artemis --install --distribution ${ARTEMIS_DISTRIBUTION_DIR} --path ${WORKING_DIR}/artemis-slave --user "CN=artemis, O=Corda, L=London, C=GB" --ha SLAVE --acceptor-address vmInfra2:11005 --keystore ./artemis/artemis.jks --keystore-password artemisStorePass --truststore ./artemis/artemis-truststore.jks --truststore-password artemisTrustpass --connectors vmInfra2:11005,vmInfra1:11005

Where ARTEMIS_DISTRIBUTION_DIR - is the path to the directory where Artemis was downloaded and extracted. Example: /home/apache-artemis-2.6.3

Files artemis/artemis.jks and artemis/artemis-truststore.jks from Artemis keystore generation stage need to be copied into ${WORKING_DIR}/artemis-slave/etc/artemis.

vmInfra2 Bridge instance setup

vmInfra1 setup been done in the section above Bridge instances setup.

For vmInfra2 the whole of base directory can be copied across to vmInfra2 and then firewall.conf ought to be modified.

Please see SWAP2, SWAP3 and SWAP4 comments in the configuration file. The key principle here is to ensure DataCentre locality whenever possible, making same DataCentre connection a priority. This applies to Artemis connection, Float connection and Zookeeper connection.

Starting all up

Please see the notes above on connectivity through the proxy and the details of running Capsule fat jars.

Starting Float processes

In order to run each of the Float processes on vmFloat1 and vmFloat2, in the base directory chosen during Float VMs setup, the following command should be executed:

nohup java –jar corda-firewall.jar &

[main] internal.FirewallStartup.startFirewall - Firewall started up and registered in 2.86 sec

In addition, traffic stats are logged every minute, like so:

Load average: 5%
Memory:
        Free: 75 MB
        Total: 200 MB
        Max: 200 MB
Traffic totals:
        Successful connection count: 1(inbound), 0(outgoing)
        Failed connection count: 1(inbound), 0(outgoing)
        Packets accepted count: 0(inbound), 0(outgoing)
        Bytes transmitted: 0(inbound), 0(outgoing)
        Packets dropped count: 0(inbound), 0(outgoing)
Traffic breakdown:
        Successful connections in:
                /13.80.124.64:57196 -> 1
        Failed connections in:
                /81.148.212.130:6546 -> 1

Starting Apache ZooKeeper processes

With configuration completed during Apache ZooKeeper setup stage, start ZK instance using the following command in the base directory:

On vmInfra1 host:

./zookeeper/bin/zkServer.sh --config /opt/corda/config/zoo1 start

On vmInfra2 host:

./zookeeper/bin/zkServer.sh --config /opt/corda/config/zoo2 start

On vmZkWitness host:

./zookeeper/bin/zkServer.sh --config /opt/corda/config/zoo3 start

Every zkServer.sh start should report back to console with:

Starting zookeeper ... STARTED

After all ZooKeeper clusters have been successfully started on every host, execute:

./zookeeper/bin/zkServer.sh --config /opt/corda/config/zooX status

Where zooX is zoo1 on vmInfra1, zoo2 on vmInfra2 and zoo3 on vmZkWitness

Messages similar to the ones below should appear on two of the hosts indicating the follower status:

Using config: ./config/zoo3/zoo.cfg
Client port not found in static config file. Looking in dynamic config file.
Client port found: 11105. Client address: 10.155.0.8.
Mode: follower

Whereas on the remaining host, the leader status is indicated by:

Using config: ./config/zoo2/zoo.cfg
Client port not found in static config file. Looking in dynamic config file.
Client port found: 11105. Client address: 10.155.0.4.
Mode: leader

Starting Artemis cluster

In order to start Artemis, the following command should be issued for master on vmInfra1:

nohup ${WORKING_DIR}/artemis-master/bin/artemis run &

To confirm master has been successfully started, nohup.out file should contain:

AMQ221007: Server is now live
AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3 [0.0.0.0, nodeID=5df84d6f-0ea4-11e9-bdbf-000d3aba482b]

In order to start Artemis, the following command should be issued for slave on vmInfra2:

nohup ${WORKING_DIR}/artemis-slave/bin/artemis run &

To confirm slave has been successfully started, nohup.out file should contain:

AMQ221024: Backup server ActiveMQServerImpl::serverUUID=5df84d6f-0ea4-11e9-bdbf-000d3aba482b is synchronized with live-server.
AMQ221031: backup announced

In this example, 5df84d6f-0ea4-11e9-bdbf-000d3aba482b is the cluster ID of the Artemis master instance.

Starting Bridge processes

There will be two Bridge instances running on vmInfra1 and vmInfra2. They will be running in Hot-Warm mode whereby one on the processes is active(leader) and the other one is running, but it a passive mode(follower).

Should primary process goes down or otherwise becomes unavailable, the stand-by process will take over. Apache Zookeper cluster will be used to ensure this leader transition process happens smoothly.

In order to run each of the Bridge processes the following command should be executed on vmInfra1 and vmInfra2:

nohup java –jar corda-firewall.jar &

Checking any file in the logs folder should ideally reveal no ERROR nor WARN messages. If bridge start-up sequence has been successful the following INFO level log messages should be observed for both Bridge instances:

artemis.BridgeArtemisConnectionServiceImpl.artemisReconnectionLoop - Session created

One of the Bridges will become a leader and should have the following in its log:

setting leadership to true; old value was false
...
Waiting for activation by at least one bridge control inbox registration

Domino effect

There is a concept of chained activation of the services which is often internally called the Domino effect. When services are not activated they run as Java operating system processes, however they are dormant from data processing point of view. E.g. Float service when not activated does not even have the port for inbound communication open. This makes perfect sense as underlying backend infrastructure may not be running at all and if a message was received from the outside, it will not be able to route it correctly for processing.

Given ZK, Artemis, Bridge and Float running, but without any nodes started, the environment is largely in the dormant state.

When node starts the following happens:

• Node creates queues on the Artemis side;
• Using Artemis communication mechanism, the node sends a special activation message to the Bridge. In response to this message, the Bridge activates;
• Bridge then sends a special activation message to a Float via the tunnel communication channel;
• Float starts to listen for inbound communication (port 10005 in the example above) and this will make it available for processing traffic from the Internet facing loadbalancer.

Starting node processes

Each of the boxes vmNodesPrimary and vmNodesSecondary is capable of hosting both nodes for Entity A and Entity B at the same time.

vmNodesPrimary and vmNodesSecondary are meant to be located in different datacentres and in case when one of the datacentres is unavailable, the whole application plant will be running on the other datacentre’s hardware.

In this setup Corda Nodes for each of the entities work in Hot-Cold mode. Which means that if the node is running on vmNodesPrimary, the node for the same identity on vmNodesSecondary cannot even be started. For more information, please see Hot-cold high availability deployment.

This implies that when starting nodes they should be running in re-start loop.

In order to start Corda Node normally on any of the hosts (vmNodesPrimary or vmNodesSecondary) for either of the entities (Entity A or Entity B) the following command should be used from the base directory:

nohup bash -c 'while true; do java -jar corda.jar; done' &

Upon successful startup of primary nodes there should be no ERROR level lines in node logs. Once node’s startup sequence completes, the following line will be printed in the logs:

Node for "Entity A" started up and registered in 35.58 sec

If the primary node is already running, secondary nodes will gracefully shutdown with the following lines in the logs:

[ERROR] PID: 7256 failed to become the master node. Check if /opt/corda/entityA/vmNodesPrimary, PID: 7593 is still running. Try again in PT40S
[INFO ] Will sleep for MutualExclusionConfiguration(on=true, machineName=vmNodesSecondary, updateInterval=20000, waitInterval=40000).waitInterval seconds till lease expires then shutting down this process.

Performing basic health checks

Checking Float port is open

If the Domino effect happened successfully and all the services activated, one of the the Floats should be listening on port 10005. To check this is indeed the case, logon to vmFloat1 or vmFloat2 host and check that the port is bound:

lsof | grep 10005

This should produce a non-empty list of processes that are listening to this port.

Checking Float is reachable from the outside

Using a computer that is capable to perform external communication to the environment, run:

telnet banka.com 10005

Type healthCheckPhrase which is in our example - HelloCorda. Initially, no characters will be echoed back, however once you finish the phrase, it will be echoed back in full to the terminal as well as any subsequent symbols that you type.

This will ensure that the Float can be contacted from the outside and is performing normally.

Running some flows

The ultimate test is of course running some flows. it would make sense to check that EntityA can successfully talk to Entity B, as well as have some external node sending flows to EntityA and Entity B.

Desired effect is dependent on the CorDapps installed, however the Bridge and the Float will log some stats every minute detailing the number of messages relayed in every direction.