Publisher Service General Knowledge
The publishing service, or publisher, provides two methods with which a customer can integrate Canary views data with third party systems. The first method uses SparkplugB over MQTT 3.1.1 and the second method uses JSON over Websocket. Access it from the Canary Admin user interface. The Publisher is organized into three pages: 'Configuration', 'Status', and 'Settings'. The 'Status' page has no worthwhile information until a connection gets configured. Continue reading to learn about 'Configuration' and 'Settings'.
- Click 'New' in the top left-hand corner in 'Configuration'
The ‘Configuration’ section is broken up into two subsections. The fields the user must interact with change depending on the type of connection used.
The WebSocket Connection
WebSocket - Allows the user to initiate a connection while bypassing firewalls on other socket ports on a server, for the purposes of sending and receiving data. Http:// is an example of a socket connection. WebSocket allows one to transfer html data through an http:// socket port while bypassing the firewalls on other socket ports on an MQTT server.
WebSocket Field Mapping -The field mapping portion to the right allows the admin to markup objects in JSON. Customers can create their own JSON structure that is mapped out to the Views service after Assets are selected, you can choose the little tag icon as shown above, to identify specific tags. The field mapping portion is disabled for the MQTT connection.
Name - The name of the session, usually related to the assets that you wish to Publish to ESRI or an MQTT server.
Type - This is where the type of publishing connection can be chosen, WebSocket connection or MQTT connections are currently the only two types supported.
URI - Enter in the uniform resource identifier for the connection endpoint. Generally, it starts with 'ws' followed by a series of forward slashes and numbers. If a client certificate is used, then the prefix will be 'wss' which stands for WebSocket secure. The URI information for the desired connection can be obtained from the server documentation for the server the user is attempting to connect to.
Serialization - The process of translating data structures or an object state into a format that can be stored in a file or memory buffer and reconstructed later (possibly in a different computer environment). Currently the only data structure formats that can be serialized are 'JSON' and 'SparkplugB'. This setting cannot be altered in the publisher service.
Asset Host - The admin can scroll down to select the asset host, which will be the machine or historian that houses the asset models and assets already built in ‘Views’. Keep in mind that asset models can be pulled from multiple machines using remote view. However, most operations have all their virtual views and assets built out onto a centralized corporate historian.
Asset Path - The folder structure that the relevant assets are nested in.
Asset - A grouping of renamed tags that adhere to a specific folder structure. Assets are created in the Views service and can help unify the tag namespace to avoid confusion and unnecessary complexity.
Flush Interval - Determines how often records are published. When a record flushes out, it means that all the cached data from a set time interval gets published. This is essential to curb data loss when the WebSocket/MQTT connection is lost temporarily. If the 'Flush Interval' is set for a longer time interval and the connection resumes sooner than expected, duplicate data publishes. That is the cost of ensuring no data loss. The 'Flush Interval' buffers changes for a certain amount of time and then publishes. To ensure that new assets get published as soon as possible, adjust the flush interval to run less often than the 'Live Data Check' and the 'Views Change Check'.
Keep Alive Interval – Specific to the MQTT connection. Sometimes phantom connections occur and it appears that your connection is good on the 'Status' page, but when data is received, an error is received. Adjust the 'Keep Alive Interval' to know sooner when a connection is lost.
Reconnect Interval – Determines how long the service waits before attempting to re-connect after a lost connection
Use Client Certificate - Used to secure the connection between the 'Publisher' and the WebSocket connection endpoint or MQTT server. Applies to all connections.
The MQTT Connection
The 'MQTT Connection' type is different from the WebSocket connection because it supports durable streams while treating the historian as a store and forward buffer. It uses 'SparkplugB' as the payload format as opposed to JSON which makes it more efficient when you are dealing with higher amounts of tags and faster data rates. There is no field mapping section for the MQTT connection.
Client ID - Defaults to Canary publishing, must be unique within the MQTT server, the user must ensure that multiple clients are not writing to the same Client ID.
Sparkplug B Group ID - The defined topic/folder structure that you must adhere to. It is five levels deep. The first level is group ID. These levels make up the topic path folder structure that the publishing service is going to communicate over.
Sparkplug B Node ID - Second level of the defined topic/folder
Sparkplug B Device ID - Third level of the defined topic/folder
Sparkplug B Primary ID -Last level of the defined topic/folder. The 'Primary ID' allows two clients that are communicating within MQTT to identify each other, which enables store and forward.
Alias Tags - With Sparkplug B the user can alias the tag name with an integer index and it saves space on each publish from having to republish that name out, it is a space saving mechanism.
Compression Options - Choose between GZIP or DEFLATE, which are standard compression options for Sparkplug B that drastically reduce the payload size. In some cases, up to 90% reduction in size is possible.
Durable Stream - When disabled, the target server just gets the latest values as they come into the publisher from the views service. When this box is checked it will basically save the last timestamp that has been published for each tag. If coming back online from a shutdown it re-reads the values from all the tags from that time forward and flushes them out.
Publish Properties - Allows the user to publish every time values change so that the client always gets the meta data every time something changes. This feature makes the payload bigger but allows the client to have all the contextual data values they might need when a property changes. Typically, this is set to birth and change.
Publish Qos -There are three different quality of service settings. Qos O means it is going to send the payload one time without checking to see if it got published or not. Qos 1 means it is going to keep sending the data out until it gets published at least once. Qos2 means that it is going to send the data out and make sure that the data gets published exactly once. The higher your Qos settings the more chattiness occurs between servers. With the higher Qos settings there is more CPU usage.
Views Host - This is where you pick an Asset to publish out. The System gets all the tags associated with that Asset and ships them out. It is monitoring those values for changes and shipping out those changes as they come in.
Servers - The server list at the bottom of the screen is used for a high availability MQTT structure with multiple servers. The subscribing clients subscribe to all the servers. The publishing clients only publish to one of them at a time. If one broker goes out of service, the next server in line will automatically start publishing its data. A server can be taken out of the pool and still have data flowing from the publisher to the subscriber.
The Service Settings
- Click 'Settings' at the bottom of the page, to the right of 'Configuration'.
Delayed Start Seconds – A general delay can be entered so the service starts when needed. Sometimes other services need to start running before a machine starts up . For example, the 'Publisher' requires the 'Views Service' on another machine to be fully functioning before it can run. This setting gives other services time to get up and running after a machine reboot.
Views User - The 'Client Server Name' that will be used to connect to the 'Asset Host' . Used to prevent unwanted clients from connecting to the views. Used to control who has access to a views service before they attempt to publish data.
Live Data Check Interval - The 'Live Data Check Interval' can be changed from its default setting which is 00:00:01 seconds. This means that the 'Publishing Service' checks the 'Views Service' every second for changes in the data. Whether it is set to MQTT or WebSocket does not matter. For example, if you have ten subscriptions for tag A that do not change, you are only getting Tag A published once even though the publisher service is checking it every second for changes, which is why it may be necessary to change the 'Live Data Check Interval'. Doing this will lower CPU Usage.
Views Change Check Interval - Furthermore, the 'Views Change Check Interval' is currently set to 1 minute, which means that the 'Publishing Service' is checking for new assets that have either been created or added every minute and is publishing that new asset to ESRI.
Durable Stream Tag Save Interval – 'Durable' means a machine restart does not affect it. As data flushes, the service records each of the tags sent and the latest time the tags were sent. If connection is lost or a machine goes down, that information is important to let us know where the 'Publisher' starts when connection resumes.
- The 'Durable Stream Tag Save Interval' is how often data is saved to the disk. With an operation publishing 100k tags or more, this can be an important decision to make in terms of efficient CPU usage. The user must ask the question: How frequently do I want to save the status of those tags so when I lose connection, I can resume without data loss? The 'Publisher' user must find the balance between duplicate data being published and data loss.
The Client Certificate
The Client Certificate allows the user to alter security related settings.
Store Name - The file location of the client certificate.
Find Type - Tells the Canary Admin what to search for on the client certificate.
Subject Name - Typically starts with the name of your machine and is created by the client's IT department.
The Status Screen
There is not much to see on this screen other than data flowing from one system to the other.
Name - A unique name of the connection, named after a location, purpose, or asset usually
Type – Type of connection, either WebSocket or MQTT
Endpoint – Should have the port number of the endpoint in the connection
Tags - Total tags included in the session
Assets - Total assets being shipped out from the publishing service
Serialization - Either set to JSON or SparkplugB depending on whether your connection is WebSocket or MQTT
Enabled – Shows whether the connection is enabled or not
State – Either active or inactive
Connected Since – Shows last successful connection attempt
Data Sent – Shows what has been successfully published to the ESRI server