Data Collection by Monitoring Agent

The monitoring agent serves as a data source for analyzing user activity on workstations. It plays a key role in building a Task Mining system, enabling precise tracking of employee work actions, switching between programs, interaction with interfaces, and other OS elements. Collected data is used to identify repetitive tasks, build business process models, and analyze employee workload.

Key capabilities of the monitoring agent:

• Records employee work activity in applications, browsers, and other programs
• Helps determine user engagement level, periods of activity, and idle time
• Collects data required for building process models and identifying recurring tasks

Depending on the required level of detail, the agent can operate in basic or extended monitoring mode and supports data collection from Java applications and HID devices.

The monitoring agent collects user activity consisting of events. An event is a user action (button click, tab switch, field input, etc.) or a program action (e.g., dialog window appearing, window title change, active window change, etc.). All events are associated with the currently active application window. If an action leads to no consequence (e.g., clicking an inactive control element), such events are not recorded in the activity log.

Each event contains information about the time of the action, application name, its version, path to the executable file, and, depending on the monitoring type, additional details. For more information, see Description of Data Storage Structure in ClickHouse.

Basic Monitoring

In basic monitoring mode, the agent records information about open application windows and switches between them:

• For browser usage — the active tab title and page URL
• For desktop applications — the window title

• For MS Excel — the active sheet title

Basic monitoring includes window hierarchy events and HID activity.

When storing activity locally (without sending to the server), the disk space occupied per user over one month is approximately 25 MB.

Extended Monitoring

Extended monitoring collects form data (parameters) when available, along with events. Parameters refer to values from editable fields (of type Edit and ComboBox). These are detected upon gaining and losing focus in editable fields. Once parameters are identified, they are added to all subsequent events within that window.

The monitoring agent does not collect or store user passwords. Password input fields are not recorded — only the fact of editing a Password field is stored.

Monitoring Java Applications

Java applications that support the Java Access Bridge technology are available for extended monitoring. This allows the monitoring agent to expand the list of applications whose activity needs to be tracked. Examples include Oracle, SweetHome 3D, and others.

To capture user activity in Java applications, the Java Monitoring module must be used. Java monitoring provides more detailed information about Java application usage, aiding in analysis, optimization, and efficient management. With this module, full extended data is captured in applications using Java Access Bridge; without it, only window title information is recorded.

Collection and Transmission of User Activity Data

To accurately track activity per user, the agent additionally collects:

• User account name, login, domain, or workgroup
• Computer name, its domain, or workgroup
• Time zone of the user session

The Operavix server receives employee activity data from the monitoring agent and stores it in an intermediate database. The data is then synchronized with the monitoring_activity table in the ClickHouse DBMS. The structure of the monitoring_activity table, including column descriptions, data types, and content, is detailed in Description of Data Storage Structure in ClickHouse.

Types of Events

Examples of window hierarchy events:

• WindowOpen — triggered whenever something new opens (window, tab, file, URL, even if the window ID remains unchanged)
• DocumentOpen — when a new document is opened
• DocumentSave — when a document is saved
• SheetOpen — when a sheet is opened or switched in Excel
• WindowSwitch — whenever switching between existing tabs or windows
• WindowUpdate — whenever the tab or window title changes

Events Not Recorded

• Actions that do not cause changes: window clicks, scrolling, moving mouse over dropdown elements
• Focus changes via arrow keys
• Clicking an editable field
• Switching to the desktop and performing actions there
• Switching to the taskbar and interacting with it
• Opening the Start menu and performing actions within it
• Using the search panel and related actions

Activity Recording Specifics

UI Automation objects associated with inactive browser tabs remain constantly in the agent’s memory.

HID Activity

The monitoring agent additionally records activity from HID devices such as mouse and keyboard. HID activity helps identify periods of user activity and inactivity at the workstation.

HID activity from user actions is classified as hardware event — hardware. Robotized or programmatically generated events are classified as injected.

HID activity is recorded during the following actions:

• Mouse click: injected (InjectedMouseClick) and hardware (HardwareMouseClick)
• Mouse scroll: injected (InjectedMouseScroll) and hardware (HardwareMouseScroll)
• Keyboard press: injected (InjectedKeyboardEvent) and hardware (HardwareKeyboardEvent)

Each event has its own type represented as a numeric code. For detailed information about event types, refer to Monitoring Logs.

Data for an HID event is inherited from the previous event but updated with the corresponding HID activity type (HardwareMouseClick, HardwareKeyboardEvent, etc.). HID activity is recorded only for fields where prolonged interaction is expected, such as editable fields.

An HID event is recorded relative to the last captured event when the interval between them is 1 minute or longer. However, the time from an HID event to a user action event may be less than 1 minute.

If a user works such that each action occurs at intervals shorter than 1 minute (e.g., window switching), HID events may not be recorded at all.

Operation Algorithm

The timestamp of the last HID event and the last HID event recorded in the activity log are stored. When a new event occurs:

• If the event time is greater than or equal to the last event time + 1 minute, both the last and the new HID events are added to the activity log
• If the event time is greater than or equal to the last recorded HID event time + 1 minute, the new event is added to the activity log

In all other cases, only the timestamp is remembered.

Example 1:

A user types in a text document, and the last HID event was recorded at 10:00:00. They continue typing until 10:00:35, then leave their workstation for 5 minutes and click the mouse again at 10:05:20, resuming work. In this case, the agent records HID events at 10:00:00, 10:00:35, and 10:05:20.

By analyzing the timestamps, we can conclude that the user was inactive from 10:00:35 to 10:05:20, as the gap between 10:00:00 and 10:00:35 is less than one minute.

Example 2:

A user works in a browser, and the last HID event was recorded at 15:00:30. They click in the browser at 15:00:55, pause, and click in another application at 15:01:20. The agent records only one HID event at 15:01:20, as the interval between 15:00:30 and 15:01:20 is less than one minute, making it unnecessary to record 15:00:55 as a separate HID event.