Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Best way of taking a date prefix from a line and forming a file path from it?
#1
i'm getting a string that begins with "YYYY-mm-dd HH:MM:SS". i need to get "YYYY-mm/dd/HH" from the string (to be the file name to write the string into). speed is most import for this as there can be bursts of hundreds per second (maybe even thousands, in rare cases). the code i am first thinking of is:
    name = s[:7]+'/'+s[8:10]+'/'+s[11:13]
f-strings is possible as this script is just for my use and i have 3.6.8, but i haven't learned their format syntax, yet. and since it is not for public use, pythonic is not a requirement.
Yoriz wrote Jul-22-2019, 08:20 PM:
I made the title more descriptive of the problem at hand, Please try to use more descriptive titles in future posts.
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Quote
#2
Now is it more normal to use a date library when working with dates.
>>> from datetime import datetime
>>> 
>>> s = '2019-04-01 13:15:00'
>>> d = datetime.fromisoformat(s)
>>> d
datetime.datetime(2019, 4, 1, 13, 15)
>>> d.strftime('%Y-%m/%d/%H')
'2019-04/01/13'
>>> 
>>> # With f-string
>>> f'{d:%Y-%m/%d/%H}'
'2019-04/01/13'
Now is this format with date and hour together a little strange.
More like hour alone.
>>> f'{d:%Y-%m/%d %H}'
'2019-04/01 13' 
Or replace with 00 if want that after hour.
>>> d
datetime.datetime(2019, 4, 1, 13, 15)
>>> h = d.replace(minute=00)
>>> f'{h:%Y-%m/%d %H:%M:%S}'
'2019-04/01 13:00:00'
Quote
#3
the string i get is already formatted. i just need to change a couple of characters. another option might be to spread the characters out into a list, update the two places that need to be changed, and join the characters back together. that, or maybe a bytearray.
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Quote
#4
Your thread should have a more descriptive title of the problem at hand, actual examples of input and required output would also improve the chances of getting the answer you would like.
Quote
#5
In [13]: import datetime 
    ...:  
    ...:  
    ...: def manual_conv(s): 
    ...:     return s[:7]+'/'+s[8:10]+'/'+s[11:13] 
    ...:  
    ...:  
    ...: def manual_conv_fstring(s): 
    ...:     return f'{s[:7]}/{s[8:10]}{s[11:13]}' 
    ...:  
    ...: def dt_conv_fstring(s): 
    ...:     return f'{datetime.datetime.fromisoformat(s):%Y-%m/%d/%H}' 
    ...:  
    ...: def dt_conv(s): 
    ...:     return datetime.datetime.fromisoformat(s).strftime('%Y-%m/%d/%H') 
    ...:  
    ...:  
    ...: s = '2019-04-01 13:15:00' 
    ...: a, b, c, d = manual_conv(s), manual_conv(s), dt_conv(s), dt_conv_fstring(s) 
    ...: print(f'a: {a}, b: {b}, c: {c}') 
    ...: if a == b == c == d: 
    ...:     print('All functions return the same result') 
    ...:  
    ...: %timeit manual_conv(s) 
    ...: %timeit manual_conv_fstring(s) 
    ...: %timeit dt_conv(s) 
    ...: %timeit dt_conv_fstring(s)                                                                                        
a: 2019-04/01/13, b: 2019-04/01/13, c: 2019-04/01/13
All functions return the same result
438 ns ± 2.89 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
336 ns ± 7.46 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
2.24 µs ± 34.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
2.43 µs ± 14 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Quote
#6
here is what i have coded so far:

exe = argv.pop()
cmd = ['tcpdump','-lnttttU'] + argv

proc = Popen(cmd,stdout=PIPE)
pipe = proc.stdout

while True:
    buff = pipe.read()
    if not buff:
        pipe.close()
        proc.wait()
        proc = Popen(cmd,stdout=PIPE)
        pipe = proc.stdout
        continue
    prev_ymdh = None
    for line in buff.splitlines():
        ymdh = line[:7]+'/'+line[8:10]+'/'+line[11:13]
        if ymdh != prev_ymdh:
            curr_file.close()
            ymd = ymdh[:10]
            if ymd != prev_ymd:
                ym = ymd[:7]
                if ym != prev_ym:
                    os.mkdir(ym)
                    prev_ymd = None
                    prev_ym = ym
                os.mkdir(ymd)
                prev_ymdh = None
                prev_ymd = ymd
            curr_file = open(ymdh,'ab')
            prev_ymdh = ymdh
        curr_file.write(line)
i'm trying to improve line 17, right now. there could be other things wrong with this code that are not what i am asking for help on. and the imports are not coded, yet.

for example, i need to skip the close() the first time since no file is open, yet. if you see bugs, ignore them and let me find them. i may well end up rewriting most of it.
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  How to get file name without the full path details and without extension aruncom2006 1 96 Jan-13-2020, 07:37 AM
Last Post: Larz60+
  How to include Variable in File Path penahuse 3 183 Jan-05-2020, 03:08 AM
Last Post: ichabod801
  Detect end of line in text file including line breaks DanielM 4 188 Dec-18-2019, 11:57 AM
Last Post: Malt
  update txt file but keep a specific line 3Pinter 2 162 Dec-16-2019, 07:54 AM
Last Post: 3Pinter
  How to search full path of specified file SriRajesh 3 180 Dec-14-2019, 04:52 PM
Last Post: Gribouillis
  python one line file processing har 4 286 Dec-09-2019, 06:10 AM
Last Post: har
  prefix to infix rontoto 4 234 Dec-08-2019, 12:32 PM
Last Post: rontoto
  line number of first and second occurance of string in a file mdalireza 1 120 Nov-18-2019, 09:55 AM
Last Post: perfringo
  How to read text file line by line SriRajesh 1 145 Nov-05-2019, 01:51 PM
Last Post: snippsat
  insert value to specific line in CSV file asheru93 1 202 Oct-21-2019, 03:37 PM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)