Topic Capture changed records

Beverly started the topic:
2017-05-01 16:00

Capture changed records

I would like to learn about this feature. We hope to run a job each day to capture data cleaner reports of data changed from last run. How to set up? I have looked at the screen. I don't see what I need to enter/set up. I have a timestamp column on all our data that is a date of when data was stored to our Oracle database table.
Dennis replied:
2017-05-02 08:58
Hi Beverly,

You need to choose that timestamp column in "Last modified column", and then a file, e.g. a "last_change" file inside your datastores folder, that the component can write its state to. If the file doesn't exist, it will be created by DataCleaner.

After that, you simply link the filter to a component, and then that component will only run on rows that has been changed since last time you ran the job.

BR,
Dennis
Beverly replied:
2017-05-02 12:50
OK - seems to work.
Beverly replied:
2017-05-03 13:14
No it doesn't. I get output file - no records captured but all the rows get processed for all the edits coded like good DOB, phone numbers, etc. LOTS of processing. If there are no rows, there should be no processing.
Dennis replied:
2017-05-03 13:21
Hmmmm... Filters only affect the component they are attached to, as well as components downstream from there.

How is your job structured?
Beverly replied:
2017-05-03 13:29
I have a source and a link for each "edit". You can't delete a link either. Not pretty at all.
COBOL if would be If PHONE = 'HOME" or 'WORK' or THEN GOOD else. (Probably use a different edit but this is to describe what I want). How is my job structured??? One box indicating source and a link for everything I want to do. How to tell you?? Like an explosion?
Dennis replied:
2017-05-03 15:02
If you put the DataCleaner job file somewhere I can download it, I can take a look if you want.

If you have multiple filters, they interact with each other, and it's unfortunately not always easy to predict how.

The links are simply showing how the job is configured, which you can see by double clicking the components. If you then uncheck the column in the configuration, the link should disappear as well. They "Link to..." in the context menu is mostly there as a quicker way, the main configuration is in the config dialogs.
Beverly replied:
2017-05-03 15:12
The xml? Where?
Beverly replied:
2017-05-04 12:49
Really hoping to send you what I have done. Can't be right. Where?
Dennis replied:
2017-05-04 13:04
I've sent you an email.
You are not signed in.
You need to be signed in to participate in the discussion.