Problem description & analysis

Below is data in text file txt.txt:

5

5

5

5

5

8

8

9

9

9

9

10

10

11

Each row in the text file contains a number. There are duplicates and rows are already sorted. We are trying to trim the file to remove the duplicates and generate a new ordered file, as shown below:

5

8

9

10

11

Solution

Write the following script p1.dfx in esProc:

Explanation:

A1   Import the txt data; @i option enables returning a sequence when the result set has only one column.

A2  Perform distinct on A1’s ordered sequence and export result to result.txt.

Q & A Collection

https://stackoverflow.com/questions/58087265/deduplication-from-text-file-using-scanner-without-arrays-java