Activity Archiving

The system provides for activity data archiving. This allows reducing the load on the main built-in file database, decreasing the database size, and reducing the number of background processes related to activity.

Creating a Database Instance

A database for a specific year is created in two cases:

• When migrating data from the current structure based on metadata about existing monthly columns in the system
• If activity arrives in the system where the year differs from the existing annual databases in the system, and also if there is no information about its detachment from the system

The Operavix server, upon receiving an activity packet from the monitoring agent, writes the received data to an intermediate built-in file database. The intermediate DB is located at the path {dataDir}\databases\monitoring_raw_data. After this, the received information is collected and processed, and then written to another built-in DB corresponding to the year for which the information arrived.

A new database is created by default in the {dataDir} directory, the platform's "working directory" — {dataDir}\databases\monitoring_activity_{yyyy}, where:

• {dataDir} — working directory
• monitoring_activity_ — database name prefix
• {yyyy} — column year

Activity data is indexed in the order of arrival at the server. The first row of activity arriving at the Operavix server will receive id = 1, and the tenth arriving row will receive id = 10.

If there is no record for the {yyyy} year in activity_database, the process of creating a database for the specified year is initiated. A folder monitoring_activity_{yyyy} is created in the directory {dataDir}\databases\. If a folder already exists in the directory, the application writes the corresponding exception to the log. A DBProvider is initialized — an instance of connection to the built-in file database. The following columns are created in the monitoring_activity_{yyyy} database:

• activity_idx_{yyyy}
• activity_{yyyy}
• default

The following data is entered into the "default" ColumnFamily:

• CREATE_DATE — database instance creation date
• YEAR — the year the database refers to
• GUID — unique identifier

If an error occurs during DB initialization, the application writes the corresponding exception to the log. The data of the created database is entered into the activity_database domain entity in the monitoring database:

• guid
• year

A new instance of connection to the built-in file database is registered in the activity recording service.

Periodic synchronization of activity data occurs between the Operavix server's built-in file database and the ClickHouse DBMS, which is used as the system Data Storage.

On the ClickHouse DBMS side, data is written to the main.monitoring_activity table. During synchronization, the Operavix server determines the last activity index recorded in the specified ClickHouse table, and then writes the missing data to it relative to the built-in file DBMS.

Activity data from monitoring agents is stored simultaneously on the Operavix server side and on the system ClickHouse DBMS server side. This is done for reliability, as the data from monitoring agents is generated by the Operavix system itself and cannot be restored from other sources.

Deleting and Restoring Activity Data on the ClickHouse DBMS Server

The Operavix server synchronizes data based on the last activity index present in the table on the ClickHouse DBMS side. If rows are deleted before this index, they will not be re-synchronized.

Activity data in the main.monitoring_activity table can be manipulated up to the row with the very last index by any method available in the ClickHouse DBMS.

By truncating the main.monitoring_activity table, you can initiate full synchronization of all activity data from the Operavix built-in DBMS to the ClickHouse DBMS.

Deleting and Restoring Activity Data on the Operavix Server

Activity data can be deleted from the Operavix server by years, detaching the corresponding internal databases. To do this, perform the following actions:

1. Stop the Operavix server service (Operavix or Operavix-app depending on the operating system).
2. Cut the directory with data for the required year from the {dataDir}\databases\monitoring_activity_{yyyy} directory.
3. Start the Operavix server service.

After this, the Operavix server will not be able to access data for that year.

If monitoring agents send data for a deleted year, it will be placed in the corrupted category. Corrupted data can be correctly processed if the database for the required year is reconnected.

To connect a previously deleted database with annual data to the Operavix server, perform the actions in reverse order:

1. Disconnect the Operavix server service.
2. Place the previously cut database folder into the {dataDir}\databases\ directory.
3. Start the service.

Moving Activity Data on the Operavix Server to Another Directory

If necessary, file databases with activity data on the Operavix server can be moved to another directory. To do this, perform the following actions:

1. Stop the Operavix server service (Operavix or Operavix-app depending on the operating system).
2. Move the directory with data for the required year from the {dataDir}\databases\monitoring_activity_{yyyy} directory to the desired directory.
3. Make changes to the configuration file com.operavix.subsystem.monitoring.json. Add a new block to the file according to the example below:
   external_activity_databases:["C:\\activity_data_database\2024"]
   

   If there are several such databases, list them separated by commas.
4. Start the Operavix server service.