Duplicate columns

A good practice in reconciliation processes is the identification of duplicates within the data.

Types of duplicate columns At Simetrik we have identified two types of duplicates:

Operational duplicates.

These duplicates correspond to two transactions that actually happened and that at a transactional level are independent, usually created by system and Operational errors.

Data duplicates

Data duplicates correspond to transactions that did not occur two or more times but were generated because data was duplicated. These duplicates are usually attributed to data being uploaded twice or more to a source, or the very nature of the sources sending the same transaction more than once.

Duplicate Identification

To identify what kind of duplicates your information contains, create duplicate columns, choosing the criteria under which you want to consider a transaction a duplicate. In this way, the system will generate an enumeration of each of the times that the set of criteria is repeated in the source. With this functionality, you will also be able to sort the enumeration, ensuring that you can consider the original transactions, the newest, the oldest or whatever criteria you choose.

Finally, thanks to this identification you will be able, if you wish, to adjust and/or filter these items so that they are not considered in the reconciliation and thus avoid false positives or manage them. In order to create the column that allows us to identify if there are duplicates, whether operational or data, follow the steps below:

Step 1 - Create Duplicates Column Go to the source table on which you want to create the duplicates column and click Add Column and then Duplicates Column.

Step 2 - Define criteria Once the duplicate column creation section is enabled, you will be asked which columns identify your unique records, here you should select the columns in the file that should be repeated to consider a transaction as a duplicate, for example, two or three transactions that have the same date, name and value could be a duplicate.

which columns identify your unique records?

In the Sort section, you must select the data that will function as the computer of the duplicates found by the system. It is recommended to use fields such as dates or numeric fields. Once you select the field, you can tell the system whether you want it to sort the data in ascending or descending order. Press Create column and you will have your duplicate identification column ready.

Step 3 - Duplicate Column Generation The system will automatically regenerate a new column on the table, which will have the number of times a transaction is repeated in the data. The original or unique transactions will be classified with the number 1 and the duplicate transactions with higher numbers, depending on how many times the transactions are repeated.

To omit duplicate transactions, apply a filter on the SKT_UNIQUENESS column and filter only those containing a 1. You can save this filter as a Reconcilable Group!

Step 4 – Duplicate Validation To verify if the transactions that the system identified as duplicates are correct, filter the table with one of the duplicate transactions and verify that the fields do indeed repeat.

Step 4 – Duplicate Validation

To verify if the transactions that the system identified as duplicates are correct, filter the table with one of the duplicate transactions and verify that the fields do indeed repeat.

Hide columns Sometimes it is necessary to hide one or more columns to have our information more organized and clearer for our reconciliation processes.

Step 1. Select the hide columns icon located in the filters section.

Step 2. Select the columns you want to see and apply the changes.

Step 3. Wait for the columns display to update. This option will also be available in your reconciliation tables.