"If a worker wants to do his job well, he must first sharpen his tools." - Confucius, "The Analects of Confucius. Lu Linggong"
Front page > Programming > Compare large csv files #eg38

Compare large csv files #eg38

Published on 2024-11-08
Browse:545

Here are two csv files (A and B) of same structure. Both use KEY_A, KEY_B and KEY_C as the primary key, and the two have different records.

Compare large csv files #eg38
Use Java to compare the two files in three ways and write the results respectively to new csv files. 1. Find records where primary keys are equal and values of the other fields are not equal, output their primary keys and then the other fields of A and the other fields in B. Below is the expected result:

Compare large csv files #eg38

  1. Find difference of A and B, that is, records existing in A but not existing in B, according to the primary key. Below is the expected result:

Compare large csv files #eg38

  1. Find difference of B and A according to the primary key. Below is the expected result:

Compare large csv files #eg38
Write SPL code to do the three comparisons. Below is for the 1st comparison:

Compare large csv files #eg38

Compare large csv files #eg38

Compare large csv files #eg38
T()function parses a csv file or write data to a csv file; @c option enables using the cursor to retrieve data from a file that cannot fit into the memory. sortx() function sorts data in a cursor. joinx()function performs a merge-join. merge() function merges records; @d option enables finding the diffefrence.

Read How to Call a SPL Script in Java to find how to integrate SPL into a Java application.

This is one of the problems on StackOverflow. You can click on it to see that the conventional solution is quite complicated, but the SPL approach is really simple and efficient.

SPL open source address

Release Statement This article is reproduced at: https://dev.to/esproc_spl/compare-large-csv-files-eg38-29m8?1 If there is any infringement, please contact [email protected] to delete it
Latest tutorial More>

Disclaimer: All resources provided are partly from the Internet. If there is any infringement of your copyright or other rights and interests, please explain the detailed reasons and provide proof of copyright or rights and interests and then send it to the email: [email protected] We will handle it for you as soon as possible.

Copyright© 2022 湘ICP备2022001581号-3