Python Forum

Hi everyone,
I'm somewhat new to the language so I'm hoping there's a better way to do this. I have some loops in my program that execute, but I want them to only act on a specific object once. If the object has been processed in the loop before, I don't want it to be processed again.

What I have been doing to work around this is to keep a text file, and write some distinguishing feature of the object to it... then write that into the loop, to check the file, and if object is not included in the file, then process the object... however this results in a super fragile program that is completely broken if the text file is modified.

I wish there were something with persistent memory that I could write to for this purpose.. but there has to be a better way?


EDIT - is shelf what I'm looking for? I could shelve the id then do an "if not in"?

(Feb-25-2020, 06:31 PM)t4keheart Wrote: [ -> ]I wish there were something with persistent memory that I could write to for this purpose.. but there has to be a better way?

There most certainly is, but without your code there is little help to provide.

Well, it's a lot of code and various modules, but I can find a relevant function... I didn't really think the actual code is relevant because it isn't a code specific/syntax issue, but if it helps:

The whole program runs on a loop, to "listen" to a db table. New entries (rows) get processed. I handled this situation here by simply manipulating the row once finished.

However, now I have attachments to deal with. Each time the loop restarts, any attachments accompanying that row are returned. If they have been returned before, I don't want to act on them... only new.

This function pulls the attachment, and stores to a folder. Meaning, if I remove that image from that folder, it will continue to get re-written to that folder... because it's the newest attachment.

def attachments(attach, conv_id):
    convId=str(conv_id)
    for item in attach:
        attachId = str(item['id'])
        attachParams={'id': attachId}
        response=requests.get(attachUrl + attachId, headers=h, params=attachParams)
        response=response.content
        responseParsed=json.loads(response)
        content=b64decode(responseParsed['contents'])
        fileName=responseParsed['friendlyName']
        with open(r'attachments/' + convId + '_' + fileName + '.jpg', 'wb') as k:
            k.write(content)

---

is shelf what I'm looking for?

A set() could solve the problem.
A set is a collection of unique items, where the order is not kept.
Adding items to a set -> slow
Looking up items in a set -> blazing fast
Containment checking of a set -> also fast
You can't use index access on a set.

Here the modified code:

def attachments(attach, conv_id):
    convId = str(conv_id)
    processed = set()
    for item in attach:
        if item in processed:
            continue
        attachId = str(item['id'])
        attachParams = {'id': attachId}
        response = requests.get(attachUrl + attachId, headers=h, params=attachParams)
        response = response.content
        responseParsed = json.loads(response)
        content = b64decode(responseParsed['contents'])
        fileName = responseParsed['friendlyName']
        with open(r'attachments/' + convId + '_' + fileName + '.jpg', 'wb') as k:
            k.write(content)
        processed.add(item)

If it's not the item from attach which repeats, then use the other object.
If the conv_id is repeating, you should check this, before you call attachments.

Here is an updated version which may work with your code.
The thing is, you use the camel case names for objects and this is not very Pythonic (PEP8).
With pathlib the handling with paths is easier and compatible on all platforms.

from pathlib import Path


def attachments(attach, conv_id):
    processed = set()
    for item in attach:
        if item in processed:
            continue
        attach_id = item['id']
        attach_params = {'id': attach_id}
        response = requests.get(attachUrl + attach_id, headers=h, params=attach_params)
        response = response.content
        response_parsed = json.loads(response)
        content = b64decode(response_parsed['contents'])
        friendly_name = responseParsed['friendlyName']
        file_name = f"convId_{friendly_name}.jpg"
        attachment = "attachments" / Path(file_name)
        with attachment.open("wb") as fd:
            fd.write(content)
        processed.add(item)

If you use an IDE like PyCharm, the names which have no definition, will be red underlined.
He also complains about naming.