XProtect Database Errors after System Crash

I’m running Milestone XProtect Essential+ 2019 R2 in a Windows 10 virtual machine. The Hyper-V host is having some issues and crashing periodically, which of course causes the XProtect guest to crash.

There are two disks attached to the VM, a 1TB live database and a 4TB archive. My disk monitoring tools warned me that free space on the live disk was falling below 12%. Milestone is set to use a maximum of 800GB on the live disk, but that wasn’t being enforced; it was up to about 860GB.

I started looking at some of the logs in C:\ProgramData\Milestone\XProtect Recording Server\Logs.

The RecorderEngine.log repeats this sequence of messages (often without the first WARNING):

2020-01-17 10:59:56.989-08:00 [    29] WARNING    - MediaDb event: ARCHIVE_EVENT__ARCHIVING_TABLE_FAILED; Description: Archiving failed. Device 'Hik DS-2CD2342WD-I 4mm - Alley'. Archive name: 'Archive 1'
2020-01-17 10:59:56.989-08:00 [    29] INFO       - Unhandled MediaDb event: REPAIR_EVENT__COMMIT_SEQUENCE_COLUMN_OUT_OF_SYNC_WITH_MASTER_TABLE_COLUMN
2020-01-17 10:59:57.005-08:00 [    29] INFO       - Unhandled MediaDb event: REPAIR_EVENT__EXPANDING_COMMIT_SEQUENCE_COLUMN_TO_BE_IN_SYNC_WITH_MASTER_TABLE_COLUMN
2020-01-17 10:59:57.005-08:00 [    29] INFO       - Unhandled MediaDb event: REPAIR_EVENT__EXPANDING_COMMIT_SEQUENCE_COLUMN_TO_BE_IN_SYNC_WITH_MASTER_TABLE_COLUMN_FAILED

Today’s Database_2020_01_17-16.log file (yes, already the 16th log file today), repeatedly shows this sequence of messages:

NmDatabase::CmTable::Open (line Core/Table.cpp, 419): Error opening table!
NmDatabase::CmTableColumn::ExpandTimeSlice (line Core/TableColumn.cpp, 1915): Expansion timeslice is not valid!
2020-01-17 10:28:15.761-08:00 [  7292] WARNING     - Commit sequence column ([ 01/15/20 05:08:31 ; 01/15/20 05:08:31 ]) is out of sync with master table column ([ invalid time slice ]) in table 22c63e5d-b218-4e62-bf00-2faf95c15b4a
2020-01-17 10:28:15.761-08:00 [  7292] WARNING     - Expanding commit sequence column to match time slice of master table column in table 22c63e5d-b218-4e62-bf00-2faf95c15b4a
2020-01-17 10:28:15.762-08:00 [  7292] FATAL ERROR - Expanding commit sequence column to match time slice of master table column failed!

The DeviceHandling.log (119,000 lines) repeats this error:

2020-01-17 10:52:34.178-08:00 [    64] ERROR      - 22c63e5d-b218-4e62-bf00-2faf95c15b4a  Hik DS-2CD2342WD-I 4mm - Alley  EnsureTableAndCursorExists  Exception: 
VideoOS.Recorder.CreateTableException: Failed to create table ---> VideoOS.Recorder.CreateTableException: Failed to create table ---> VideoOS.Database.TableException: Error opening table '22c63e5d-b218-4e62-bf00-2faf95c15b4a'!, Error opening table '22c63e5d-b218-4e62-bf00-2faf95c15b4a'!, Error opening table!, Expansion timeslice is not valid! ---> VideoOS.Database.TableException: NmDatabase::CmClientTable::Open (line Client/ClientTable.cpp, 264): Error opening table '22c63e5d-b218-4e62-bf00-2faf95c15b4a'! ---> VideoOS.Database.TableException: NmDatabase::CmServerClient::OpenTable (line Server/ServerClient.cpp, 944): Error opening table '22c63e5d-b218-4e62-bf00-2faf95c15b4a'! ---> VideoOS.Database.DatabaseException: NmDatabase::CmTable::Open (line Core/Table.cpp, 419): Error opening table! ---> VideoOS.Database.DatabaseException: NmDatabase::CmTableColumn::ExpandTimeSlice (line Core/TableColumn.cpp, 1915): Expansion timeslice is not valid!
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at VideoOS.Database.Connection.CreateTable(String tableName, String bankDirectory, String description, String group)
   at VideoOS.Recorder.MediaDb.CreateAndConfigureTable(Guid deviceId, String tableName, TableOption tableOption, String tableDescription, String tableDataGroupName, String mediaDataType, Boolean manuallyControlCommitSequence, RawDbChannel channel)
   --- End of inner exception stack trace ---
   at VideoOS.Recorder.MediaDb.CreateAndConfigureTable(Guid deviceId, String tableName, TableOption tableOption, String tableDescription, String tableDataGroupName, String mediaDataType, Boolean manuallyControlCommitSequence, RawDbChannel channel)
   at VideoOS.Recorder.MediaDb.OpenTable(Guid deviceId, String tableName, String tableDescription, String tableDataGroupName, String mediaDataType, RawDbChannel channel)
   --- End of inner exception stack trace ---
   at VideoOS.Recorder.MediaDb.OpenTable(Guid deviceId, String tableName, String tableDescription, String tableDataGroupName, String mediaDataType, RawDbChannel channel)
   at VideoOS.Recorder.WriteTableCursorBase.EnsureTableAndCursorExists()

One of the messages mentions the alley camera. Sure enough, recording has stopped on that camera:

XProtect DB 1

The Recording tab for that camera confirms the error:

XProtect DB 2

Stopping and starting recording did not re-activate the camera.

Solution

Finally I followed the advice (for a somewhat different problem) in this post:  with Recording Server stopped, in the root of the live disk’s E:\MediaDatabase folder, I deleted archive_cache.xml and cache.xml. After restarting the Recording Server, the cache was rebuilt and Alley camera recording resumed:

XProtect DB 3

XProtect DB 4

Unfortunately, recent historical footage for the Alley camera is not available.

The good news is that the archiving function started working again; MediaDatabase is back to just over 800GB of space used. The end of the latest Database log shows archiving progressing with the note that “Maximum size limit (819200 MB) reached for bank!” So apparently these errors were blocking archiving.

1 thought on “XProtect Database Errors after System Crash

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.

This site uses Akismet to reduce spam. Learn how your comment data is processed.