Python Forum

This is the list I am working with:

2015-01-25 14:05:41.274647 70
2015-01-25 14:08:05.036915 2070
2015-01-25 14:10:05.536604 70
2015-01-25 14:12:05.142511 2070
2015-01-25 14:14:05.045367 70
2015-01-25 14:16:05.005320 70
2015-01-25 14:18:06.639303 70
2015-01-25 14:20:04.753845 70

the third line of the third column is supposed to be an error, and my code should make it equal to the average
between the value before and after. In short I should be replacing that 70 with 2070.
My code keeps printing 70.
Not sure why.

from astral import Astral 
from scipy import *
from pylab import*
import numpy as np
from numpy import array
import matplotlib.pyplot as plt
import datetime
from datetime import timezone
from datetime import timedelta
import matplotlib.dates as dates
import pandas as pd  
import pytz
import sys

def last_digits(num, last_digits_count=2):
    return abs(num) % (10**last_digits_count)

orig_date=[]
orig_time=[]
movements=[]

with open('bird_jan25jan16.txt', 'r') as f:
    for line in f:
        data = line.split()    # Splits on whitespace        
        orig_date.append(data[0][:])
        orig_time.append(data[1][:])
        movements.append(int(data[2][:]))
    for i in range(len(orig_date)):
        if (len(str(movements[i])) >= 3 ):
            if movements[i]==0 or (  movements[i-1] == movements[i+1] !=movements[i] and  (last_digits(movements[i]) == last_digits(movements[i+1]))):
                movements[i]=(movements[i-1]+movements[i+1])/2
print(movements[2])

---

I have tried to change the code:

from astral import Astral 
from scipy import *
from pylab import*
import numpy as np
from numpy import array
import matplotlib.pyplot as plt
import datetime
from datetime import timezone
from datetime import timedelta
import matplotlib.dates as dates
import pandas as pd  
import pytz
import sys

def last_digits(num, last_digits_count=2):
    return abs(num) % (10**last_digits_count)

orig_date=[]
orig_time=[]
movements=[]

with open('bird_jan25jan16.txt', 'r') as f:
    for line in f:
        data = line.split()    # Splits on whitespace        
        orig_date.append(data[0][:])
        orig_time.append(data[1][:])
        movements.append(int(data[2][:]))
    for i in range(len(orig_date)):
        if movements[i]==0 or (  (movements[i-1] == movements[i+1] !=movements[i]) and  
                    ((last_digits(movements[i]) == last_digits(movements[i+1])))  and 
                    ((len(str(movements[i-1])) - len(str(movements[1]))) >=2)):
            movements[i]=(movements[i-1]+movements[i+1])/2
print(movements[2],orig_date[0],orig_time[0])

and now I get this error

Error:
  File "<ipython-input-80-389e619abf53>", line 1, in <module>
    runfile('C:/Users/Desktop/python ode/birds_2nd_draft.py', wdir='C:/Users/Desktop/python ode')

  File "C:\Users\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 668, in runfile
    execfile(filename, namespace)

  File "C:\Users\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/Desktop/python ode/birds_2nd_draft.py", line 36, in <module>
    if movements[i]==0 or (  (movements[i-1] == movements[i+1] !=movements[i]) and

IndexError: list index out of range

I have also added this line for datetime object conversion

t=[datetime.datetime.strptime(time, "%H:%M:%S.%f").time() for time in orig_time]
print((t[0]))

but the outcome is not a datetime object

Output:
14:05:41.274647

how so?

---

I would really appreciate some help if anyone knows

Hello, I'd like to help you but could you explain what is the exact meaning of "70" and "2070"? (I understand that these values have to be compared with their 2 surrounding neighbors, but what happens with the first and last value? These have only 1 neighbor, should they still be compared somehow?)

There's a bug that I can see but I don't know how to fix it without knowing the answer to question above. The bug is here:

for i in range(len(orig_date)):
        if (len(str(movements[i])) >= 3 ):            
            if movements[i]==0 or (  movements[i-1] == movements[i+1] !=movements[i] and  (last_digits(movements[i]) == last_digits(movements[i+1]))):
                movements[i]=(movements[i-1]+movements[i+1])/2
                print(i)

In situation where i is equal to 0, movement[i-1] returns the last element of list instead of previous one. Similar thing can happen when i is an offset to the last element of the list, in such case movement[i+1] will result in "index out of range" error.

the 70 in between the two 2070 is a mistake that my code should correct by turning that 70 into 2070 as well. The two 2070 should stay the same. Moreover, this code should be examining every single line of the list in case similar mistakes are present in other places too.
Does it answer your question?
If not, let me know.

Also, do you perhaps know what mistake did I make in the datetime object conversion?

(May-23-2019, 09:06 PM)mcgrim Wrote: [ -> ]I have also added this line for datetime object conversion

t=[datetime.datetime.strptime(time, "%H:%M:%S.%f").time() for time in orig_time]
print((t[0]))

but the outcome is not a datetime object

Output:
14:05:41.274647

how so?

---

I would really appreciate some help if anyone knows

You use .time() method, so it is not datetime, it's time.

>>> import datetime
>>> datetime.datetime.strptime('14:04:04', "%H:%M:%S")                     
datetime.datetime(1900, 1, 1, 14, 4, 4)
>>> datetime.datetime.strptime('14:04:04', "%H:%M:%S").time()              
datetime.time(14, 4, 4)

How to Read data from text file which contain list of queries and split into multiple CSV..files by python

But what happens with the first and last values? These have only 1 neighbor, should they still be compared somehow?

Quote:this code should be examining every single line of the list in case similar mistakes are present in other places too

What is considered an error though? I asked about the exact meaning/nature of these values because I have questions like:
What are the possible values, is it only 70 and 2070, or other numbers can also be there?
If other values are permitted, then let's assume this sequence:
x = [70, 50, 70, 50, 70]
Should the middle "70" (x[2]) be changed into "50"?
And why the first '50' (x[1]) shouldn't be changed instead, knowing that it is surrounded by two "70"? Is "70" a value that is not taken into consideration as surrounding neighbor? Or is it because any 2-digits values are not taken into consideration? Is it just the values that are between "2070" that are considered erroneous? Or is it about values between any 4-digit numbers? What about 3-digits numbers?

If I understood what is the meaning/origin of the 3rd column of that table, or if I could see the whole table (knowing that it is not variable), then all these and similar questions would be automatically answered so that's why I asked.

((len(str(movements[i-1])) - len(str(movements[1]))) >=2))

Quote:is it only 70 and 2070, or other numbers can also be there?

other numbers that match those characteristics are involved

Quote:If other values are permitted, then let's assume this sequence:
x = [70, 50, 70, 50, 70]
Should the middle "70" (x[2]) be changed into "50"?

no, the following line

 ((len(str(movements[i-1])) - len(str(movements[1]))) >=2)):

in my code above tells you that the digits in the previous number must be at least two more than the middle one.

Quote:Or is it because any 2-digits values are not taken into consideration? Is it just the values that are between "2070" that are considered erroneous? Or is it about values between any 4-digit numbers? What about 3-digits numbers?

two digit values are to be left as they are, unless like above, there is a 2 digit value between two largest and equal numbers whose last two digits match the middle number. With 3 digits, is the same as 4, if they are equal and surround a two digits number equal to the last two digits, it should be changed the same way.

I hope this answer all your questions, or else, let me know.

#from astral import Astral 
from scipy import *
#from pylab import*
import numpy as np
from numpy import array
#import matplotlib.pyplot as plt
import datetime
from datetime import timezone
from datetime import timedelta
#import matplotlib.dates as dates
import pandas as pd  
import pytz
import sys
 
def last_digits(num, last_digits_count=2):
    return abs(num) % (10**last_digits_count)
 
orig_date=[]
orig_time=[]
movements=[]
 
with open('bird_jan25jan16.txt', 'r') as f:
    for line in f:
        data = line.split()    # Splits on whitespace
        orig_date.append(data[0])
        orig_time.append(data[1])
        movements.append(int(data[2]))
        
    for i in range(1, len(orig_date)-1):
        if (len(str(movements[i])) < 3 ):
            
            if movements[i]==0 or (  movements[i-1] == movements[i+1] !=movements[i] and  (last_digits(movements[i]) == last_digits(movements[i+1]))):
                new_val = (movements[i-1]+movements[i+1])/2
                print('Index of changed element =',i, '\nPrevious value =',movements[i], '\nNew value =',new_val, '\n')
                movements[i] = new_val

print(movements)

Output:
Index of changed element = 2 
Previous value = 70 
New value = 2070.0 

[70, 2070, 2070.0, 2070, 70, 70, 70, 70]

Main 2 lines that were changed:

    for i in range(1, len(orig_date)-1):
        if (len(str(movements[i])) < 3 ):

It only attempts to change elements that are under 3 digits long, it also doesn't attempt to change the first and last elements (because they have only 1 neighbor each)