Python Forum
Multiple conditions, one is null
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multiple conditions, one is null
#1
I am not really good at python but I am trying to make a pretty basic program to concatenate tons of files, be able to filter according to some criteria, and then export a file with the result. This is what I have done so far (I still need to add a lot of input validation though):

from glob import glob
import numpy as np
import pandas as pd
pd.options.display.max_columns = 100

files = glob('C:/Personal/data*.log')
datos = pd.concat([pd.read_csv(f, header=None, names=range(100), low_memory=False) for f in files])

ip = '10.1.1.5'
user = 'john'
ini_date = '07/14/2020'
fin_date = '07/18/2020'
ini_hour = '01:42:00'
fin_hour = '16:15:20'

result = datos[(datos[0] == ip) & (datos[1] == user) & (datos[2] >= ini_date) & (datos[2] <= fin_date) & (datos[3] >= ini_hour) & (datos[3] <= fin_hour)] 

result.to_csv (C:/Personal/result.csv', index = False)
I would like to know the best way to do that my program ignore these variables/conditions which are not set. For example, if I run this the resut is anything:
ip = ''
user = 'john'
ini_date = '07/14/2020'
fin_date = '07/18/2020'
ini_hour = '01:42:00'
fin_hour = '16:15:20'
I need that these null variables would be ignored or taken into account like "any". If this possible? Thanks!
Reply
#2
I would do something like the following:

from operator import gt, lt, eq, and_
from functools import reduce

ip = '10.1.1.5'
user = 'john'
ini_date = '07/14/2020'
fin_date = '07/18/2020'
ini_hour = '01:42:00'
fin_hour = '16:15:20'



conditions = {
            'ip': {'value': '10.1.1.5', 'op': eq, 'index': 0},
            'user': {'value': 'john', 'op': eq, 'index': 1},
            'ini_date': {'value': '07/14/2020', 'op', gt, 'index': 2},
            'fin_date': {'value': '07/18/2020', 'op', lt, 'index': 2},
            'ini_hour': {'value': '01:42:00', 'op', gt, 'index': 3},
            'fin_hour': {'value': '16:15:20', 'op', lt, 'index': 3}
}


 
result = datos[reduce(and_, (v.get('op')(datos.iloc[:, v.get('index')], v.get('value')) for k, v in contidions.items() if v.get('value'))] 
However, it still a bit ugly; you can convert date and time columns into one datetime; use corresponding .between method to find items fallen between two dates. Hope that helps.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to express null value klatlap 3 815 Mar-25-2023, 10:40 AM
Last Post: klatlap
  value null when update in json file 3lnyn0 6 3,082 Dec-30-2021, 05:52 PM
Last Post: ndc85430
  How do you format Update statement with multiple conditions hammer 4 2,040 Dec-16-2021, 10:49 PM
Last Post: hammer
  Multiple conditions when indexing an array with or without np.where noob2305 1 2,608 Oct-25-2020, 02:06 PM
Last Post: jefsummers
  I didnt get the NULL values salwa17 0 1,555 Jul-10-2020, 02:54 PM
Last Post: salwa17
  Find only the rows containing null values Bhavika 2 2,403 Jun-10-2020, 01:25 PM
Last Post: Bhavika
  multiple conditions Chandan 7 3,939 Jan-31-2020, 12:53 AM
Last Post: Chandan
  g Null Byte using DictReader eshwinsukhdeve 13 7,374 May-15-2019, 08:50 AM
Last Post: buran
  Array/Jarray problem contains null value? Maverick0013 1 3,933 Aug-10-2018, 12:56 PM
Last Post: Windspar
  Help with multiple conditions on ATM program jakegold98 6 9,128 Dec-06-2017, 05:18 PM
Last Post: jakegold98

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020