Senzing v3 G2Engine Redo Processing

Redo Processing

Redo records are automatically created when certain conditions occur during entity resolution where additional processing is required. The most common reasons are:

  • A feature value becomes generic and previous decisions need to be revisited
  • Clean up after some record deletes
  • Detected related entities were being changed at the same time
  • A table inconsistency exists, potentially after a non-graceful shutdown

countRedoRecords

countRedoRecords() returns the integer count of the remaining internally queued redo records in the Senzing repository.

g2_engine.countRedoRecords()
Click to expand `countRedoRecords()` example
Example
#! /usr/bin/env python3

from senzing import G2Engine, G2Exception

# REPLACE /home/user/your_project with the path to your Senzing project
senzing_engine_configuration_json = '{ "PIPELINE": { "CONFIGPATH": "/home/user/your_project/etc", "SUPPORTPATH": "/home/user/your_project/data", "RESOURCEPATH": "/home/user/your_project/resources" }, "SQL": { "CONNECTION": "sqlite3://na:na@/home/user/your_project/var/sqlite/G2C.db" } }'

g2_engine = G2Engine()

try:
    g2_engine.init("G2Engine", senzing_engine_configuration_json)

    result_count = g2_engine.countRedoRecords()

    g2_engine.destroy()

    print(result_count)

except G2Exception as err:
    print(g2_engine.getLastException())

Response message:

5

getRedoRecord

getRedoRecord() retrieves the next internally queued redo record into the Senzing repository.

Unlike processRedoRecord(), getRedoRecord() does not actually process the record. To process the record, use process() 

g2_engine.getRedoRecord(response_bytearray)
Parameters
  • response_bytearray: (bytearray) Object to store the output of the method.
Click to expand `getRedoRecord()` example
Example
#! /usr/bin/env python3

from senzing import G2Engine, G2Exception

# REPLACE /home/user/your_project with the path to your Senzing project
senzing_engine_configuration_json = '{ "PIPELINE": { "CONFIGPATH": "/home/user/your_project/etc", "SUPPORTPATH": "/home/user/your_project/data", "RESOURCEPATH": "/home/user/your_project/resources" }, "SQL": { "CONNECTION": "sqlite3://na:na@/home/user/your_project/var/sqlite/G2C.db" } }'

g2_engine = G2Engine()

response_bytearray = bytearray()

try:
    g2_engine.init("G2Engine", senzing_engine_configuration_json)

    g2_engine.getRedoRecord(response_bytearray)

    print(response_bytearray.decode())

except G2Exception as err:
    print(err)
Output
{
    "REASON": "LIB_FEAT_ID[13] of FTYPE_ID[7] went generic for CANDIDATES ECLASS_ID[1] in LENS_ID[1]",
    "DATA_SOURCE": "TEST",
    "RECORD_ID": "39F84F26E1F85ABCCE20D5FDBCA7D08729DD829A",
    "ENTITY_TYPE": "GENERIC",
    "DSRC_ACTION": "X"
}

process

process() processes the redo record retrieved using getRedoRecord().

g2_engine.process(response_bytearray)
Parameters
  • response_bytearray: (bytearray) the redo record obtained by getRedoRecord() to process.
Click to expand `process()` example 
#! /usr/bin/env python3

from senzing import G2Engine, G2Exception

# REPLACE /home/user/your_project with the path to your Senzing project
senzing_engine_configuration_json = '{ "PIPELINE": { "CONFIGPATH": "/home/user/your_project/etc", "SUPPORTPATH": "/home/user/your_project/data", "RESOURCEPATH": "/home/user/your_project/resources" }, "SQL": { "CONNECTION": "sqlite3://na:na@/home/user/your_project/var/sqlite/G2C.db" } }'

g2_engine = G2Engine()

response_bytearray = bytearray()

try:
    g2_engine.init("G2Engine", senzing_engine_configuration_json)

    g2_engine.getRedoRecord(response_bytearray)

    g2_engine.process(response_bytearray)

    print(response_bytearray.decode())

except G2Exception as err:
    print(err)

processWithInfo

processWithInfo() processes the redo record retrieved using getRedoRecord(), and returns a JSON document containing the ENTITY_ID values of the affected entities.

g2_engine.processWithInfo(response_bytearray, process_response_bytearray)
Parameters
  • response_bytearray: (bytearray) the redo record obtained by getRedoRecord() to process.
  • process_response_bytearray: (bytearray) Object to store the output of the method.
Click to expand `processWithInfo()` example
Example
#! /usr/bin/env python3

from senzing import G2Engine, G2Exception

# REPLACE /home/user/your_project with the path to your Senzing project
senzing_engine_configuration_json = '{ "PIPELINE": { "CONFIGPATH": "/home/user/your_project/etc", "SUPPORTPATH": "/home/user/your_project/data", "RESOURCEPATH": "/home/user/your_project/resources" }, "SQL": { "CONNECTION": "sqlite3://na:na@/home/user/your_project/var/sqlite/G2C.db" } }'

g2_engine = G2Engine()

response_bytearray = bytearray()

with_info_bytearray = bytearray()

try:
    g2_engine.init("G2Engine", senzing_engine_configuration_json)

    g2_engine.getRedoRecord(response_bytearray)

    g2_engine.processWithInfo(
        response_bytearray, 
        with_info_bytearray)

    print(with_info_bytearray.decode())

except G2Exception as err:
    print(err)
Output
{
    "REASON": "LIB_FEAT_ID[1307] of FTYPE_ID[14] went generic for CANDIDATES AND SCORING ECLASS_ID[1] in LENS_ID[1]",
    "DATA_SOURCE": "CUSTOMERS",
    "RECORD_ID": "2181",
    "ENTITY_TYPE": "GENERIC",
    "DSRC_ACTION": "X"
}

processRedoRecord

processRedoRecordWithInfo() processes the next redo record in the redo queue. If processRedoRecord() returns a blank response_bytearray, then the redo queue is empty at the point of execution. Additional redo records can be created after reevaluation.

g2_engine.processRedoRecord(response_bytearray)
Parameters
  • response_bytearray: (bytearray) Object to store the output of the method.
Click to expand `processRedoRecord()` example
Example
#! /usr/bin/env python3

from senzing import G2Engine, G2Exception

# REPLACE /home/user/your_project with the path to your Senzing project
senzing_engine_configuration_json = '{ "PIPELINE": { "CONFIGPATH": "/home/user/your_project/etc", "SUPPORTPATH": "/home/user/your_project/data", "RESOURCEPATH": "/home/user/your_project/resources" }, "SQL": { "CONNECTION": "sqlite3://na:na@/home/user/your_project/var/sqlite/G2C.db" } }'

g2_engine = G2Engine()

try:
    g2_engine.init("G2Engine", senzing_engine_configuration_json)

    g2_engine.processRedoRecord()

except G2Exception as err:
    print(err)
Output
{
  "DATA_SOURCE": "CUSTOMERS",
  "RECORD_ID": "2192",
  "AFFECTED_ENTITIES": [
    {
      "ENTITY_ID": 105
    }
  ],
  "INTERESTING_ENTITIES": {
    "ENTITIES": []
  }
}

processRedoRecordWithInfo

processRedoRecordWithInfo() processes the next redo record in the redo queue, and returns a JSON document containing the ENTITY_ID values of the affected entities.

g2_engine.processRedoRecordWithInfo(response_bytearray, with_info_bytearray)
Parameters
  • response_bytearray: (bytearray) Object to store the output of the method.
  • info_bytearray: (bytearray) Returns the “withInfo” section
Click to expand `processRedoRecordWithInfo()` example
Example
#! /usr/bin/env python3

from senzing import G2Engine, G2Exception

# REPLACE /home/user/your_project with the path to your Senzing project
senzing_engine_configuration_json = '{ "PIPELINE": { "CONFIGPATH": "/home/user/your_project/etc", "SUPPORTPATH": "/home/user/your_project/data", "RESOURCEPATH": "/home/user/your_project/resources" }, "SQL": { "CONNECTION": "sqlite3://na:na@/home/user/your_project/var/sqlite/G2C.db" } }'

g2_engine = G2Engine()

response_bytearray = bytearray()

with_info_bytearray = bytearray()

try:
    g2_engine.init("G2Engine", senzing_engine_configuration_json)

    g2_engine.processRedoRecordWithInfo(
        response_bytearray,
        with_info_bytearray)

    print(with_info_bytearray.decode())

except G2Exception as err:
    print(err)
Output
{
  "DATA_SOURCE": "CUSTOMERS",
  "RECORD_ID": "2192",
  "AFFECTED_ENTITIES": [
    {
      "ENTITY_ID": 105
    }
  ],
  "INTERESTING_ENTITIES": {
    "ENTITIES": []
  }
}