Sync

StatMap Sync – the synchronisation proxy

Overview

Sync is used in environments where data synchronisation between data stores is required. The most common case is an isolated StatMap Aurora environment with its dedicated database which requires data to be transferred from and to the main database store on regular basis. Sync is also incorporated into StatMap Comet software and allows this application to pull/push data between its local database and the master environment’s database. Sync acts as a proxy application for the two environments which cannot communicate directly.

The requirements for this communication to be possible are as follows:

TCP port 80 must be open on source/target server.
Synchronisation service (synchronizationsevice.svc) must be accessible from outside the server hosting the source/target environment.
Due to a possible necessity of transferring large files (up to 1GB) any limitations on size of single file upload/download should be disabled on both servers.

Configuration

sync.xml

Let’s start with the sync.xml file which is the configuration file for the synchronisation process.

1
2
3
4
5
6
7
8
9
10
11
12
13

<?xml version="1.0" encoding="utf-8"?>
<Synchronizer>
  <SourceSynchronizationService>http://local/map/SynchronizationService.svc</SourceSynchronizationService>
  <SourceDeviceId>Test</SourceDeviceId>
  <SourceRepository>Main Repository</SourceRepository>
  <TargetSynchronizationService>http://remote/map/SynchronizationService.svc</TargetSynchronizationService>
  <TargetDeviceId>Aurora</TargetDeviceId>
  <TargetRepository>Main Repository</TargetRepository>
  <Profile>default</Profile>
  <WorkingDirectory></WorkingDirectory>
  <Debug>false</Debug>
  <ResultPollIntervalInSeconds>10</ResultPollIntervalInSeconds>
</Synchronizer>

Description of each element can be found below:

SourceSynchronizationService	This entry contains the URL address of synchronisation service of the local (source) environment.
SourceDeviceId	The designation of the source environment. It can be found in the environment’s web.config. It is the value of the <ThisServerId> entry e.g.:
SourceRepository	The repository (source database) from which the tables requested to be synchronised are read. Please leave as is.
TargetSynchronizationService	This entry contains the URL address of synchronisation service of the remote (target) environment.
TargetDeviceId	The designation of the target environment. It can be found in the target environment’s web.config. It is the value of the <ThisServerId> entry similar to the SourceDeviceId. IMPORTANT: Each target device must be registered in the web.config of the source environment. It is accomplished by adding <ServerX> entry with the value set as the target device’s ID, e.g.: In our example the Server2 is the target device for synchronisation.
TargetRepository	The repository (destination database) to which the tables requested to be synchronised are written. Please leave as is.
Profile	This is the synchronisation profile which will be used during the synchronisation process to determine what data should be transferred. Synchronisation profiles are configured on the source environment in Earthlight. The tool Devices on Administration tab is used for this.
WorkingDirectory	Points sync software to a directory where operational data is written during synchronisation process. If not set the temporary data folders are created within the Sync main folder.
Debug	This is an optional parameter which increases the granularity of synchronisation progress logging.
ResultPollIntervalInSeconds	This is an optional parameter which controls the time interval (in seconds) between requests for individual task completion. If not set the requests are send every 2 seconds.

sync.bat

Sync.bat batch file, located next to the sync.exe program, is responsible for executing the application, pointing it to the configured file as well as redirecting the console output (synchronisation progress log) to a file.

1	sync.exe sync.xml > sync_results.txt

Explanation:

sync.exe – synchronisation application
sync.xml – configuration file
sync_results.txt – synchronisation progress report file

sync_results.txt

Sync_result.txt is a text file which contains the progress report logged during the last synchronisation run. Successfully run synchronisation process should log the following entries at the end of its report:

If instead you see and error logged, please check the error message. In general errors require intervention of StatMap Support but errors described in the next section can be dealt with by you.

Known errors

1. Non-existent schema issue

This error is associated with multiple schemas. If you have multiple schemas in your source database it is required to re-create the schemas in the destination database. If a schema is absent or the user doesn’t have permission to access a schema a message is logged by Sync software in its report. If this occurs please create the schema in the destination database and re-run synchronisation process. Depending on the number of different schemas required in the destination database you may see the error again however it should be listing another schema. The message will look similar to the one below.

2. Communication issue on the target server

From time to time you may get the following communication error when running synchronisation process. The cause of this is being investigated but the best thing to do in this situation is to give it couple of minutes and re-run synchronisation.It seems that sometimes the target server denies data upload.

In the synchronisation report you will notice that the proceeding steps are:

Downloading file from source …
Download file size …
Uploading file to destination …
Error

The error report will look like this: