|
Search Results
|
| Post |
Author |
Forum |
Replies |
Views |
Posted
[asc]
|
| |
|
Thread: Get anything before arguments
Post: RE: Get anything before arguments
| Thanks for your reply. This prints /path/to/Python37-32/python.exe or /path/to/Python38-32/python.exe when I use python and py in command line, respectively. But that still doesn't tell whether the us... |
|
Robotguy |
General Coding Help |
3 |
2,114 |
May-18-2021, 11:17 PM |
| |
|
Thread: Get anything before arguments
Post: Get anything before arguments
| I understand how to retrieve arguments from any user input from powershell. However, I want to get what exactly was typed before arguments. For example, for the command entered by user: py myscript.py... |
|
Robotguy |
General Coding Help |
3 |
2,114 |
May-17-2021, 11:16 PM |
| |
|
Thread: How to collate Multiprocessing-Process results?
Post: How to collate Multiprocessing-Process results?
| Let us consider the following code where I calculate the factorial of 4 really large numbers, saving each output to a separate .txt file (out_mp_{idx}.txt). I use multiprocessing (4 processes) to redu... |
|
Robotguy |
Data Science |
1 |
2,245 |
May-01-2021, 06:42 AM |
| |
|
Thread: Extract continuous numeric characters from a string in Python
Post: Extract continuous numeric characters from a strin...
| I am interested in extracting a number that appears after a set of characters ('AA='). However, the issue is I: (i) am not aware how long the number is, (ii) don't know what appears right after the nu... |
|
Robotguy |
General Coding Help |
2 |
2,650 |
Jan-15-2021, 10:41 PM |
| |
|
Thread: Pytables : Printing without special characters
Post: Pytables : Printing without special characters
| I create a dataset containing some file information (characters). Aim is to print the content. Using info[:] also prints special characters like \n, b. Is there a way to avoid print of these special c... |
|
Robotguy |
Data Science |
0 |
1,687 |
Nov-06-2020, 10:55 PM |
| |
|
Thread: Automating to generate multiple arrays
Post: Automating to generate multiple arrays
| I have a for loop and have to create a new histogram every time it executes (hist1, hist2, ...). I tried the below but it isn't working. Any suggestions?
for i in range(5):
f'hist{i}' = np.zeros(... |
|
Robotguy |
Data Science |
1 |
1,814 |
Nov-04-2020, 11:33 PM |
| |
|
Thread: Delete all variables except few
Post: Delete all variables except few
| Hello,
I know how to use del var1, var2,... to delete multiple variables. However, this is not handy when I have to delete several variables. Is there an alternative to delete all variables except th... |
|
Robotguy |
General Coding Help |
1 |
6,454 |
Oct-28-2020, 10:47 PM |
| |
|
Thread: Automating PyTables Dataset Creation and Append
Post: Automating PyTables Dataset Creation and Append
| In my script, I create several datasets manually:
import tables
dset1 = f.create_earray(f.root, "dataset1", atom=tables.Float64Atom(), shape=(0, 2))
dset2 = f.create_earray(f.... |
|
Robotguy |
Data Science |
1 |
1,814 |
Oct-16-2020, 11:21 PM |
| |
|
Thread: Allocating maximum memory to chunks
Post: Allocating maximum memory to chunks
| I read large files (>10 GBs) in chunks; later performing some arithmetic operations on these chunks.
Reading in chunks is needed to avoid memory-issues. For my machine, each chunk can be as large ... |
|
Robotguy |
Data Science |
1 |
1,415 |
Oct-12-2020, 11:10 PM |
| |
|
Thread: Reading a table generated using PyTables
Post: Reading a table generated using PyTables
| Hello,
I have a table with three columns and multiple rows. I know how to extract specific rows using table[0:5]. How can I extract just the elements of first two columns for these 5 rows?
Here is w... |
|
Robotguy |
Data Science |
1 |
1,542 |
Sep-18-2020, 01:36 AM |
| |
|
Thread: Accessing details of chunks in HDF5 file
Post: Accessing details of chunks in HDF5 file
| Hi Everyone,
I found following to work with Jupyter notebook to access details of a chunked file I create. Does anyone know how to implement this in my python script?
!h5ls -v filename |
|
Robotguy |
Data Science |
0 |
1,577 |
Aug-29-2020, 06:51 AM |
| |
|
Thread: Can the data types be different for different columns?
Post: RE: Can the data types be different for different ...
| I am using PyTables-which is built on top of the HDF5 library. |
|
Robotguy |
Data Science |
2 |
2,126 |
Aug-19-2020, 09:24 PM |
| |
|
Thread: Can the data types be different for different columns?
Post: Can the data types be different for different colu...
| Hi Everyone,
I create an expandable earray of Nx4 columns. Some columns require float64 datatypes, the others can be managed with int32. Is it possible to vary the data types among the columns? Right... |
|
Robotguy |
Data Science |
2 |
2,126 |
Aug-19-2020, 06:06 PM |
| |
|
Thread: Pytables: Reducing size of an appended Earray
Post: Pytables: Reducing size of an appended Earray
| I am using PyTables-append to output the processed data. It is time efficient for large files (1-10 Gbs), at least better than resizing the HDF5 using h5py module!
However, in my case the output fil... |
|
Robotguy |
Data Science |
0 |
1,654 |
Aug-19-2020, 05:35 PM |
| |
|
Thread: Deleting multiple variables/arrays
Post: Deleting multiple variables/arrays
| Hey Everyone:
I often play with large arrays (>10 GBs of files) and have immense number of independent for loops in my script. Between the loops I delete the variables I no longer need to save mem... |
|
Robotguy |
General Coding Help |
0 |
1,502 |
Aug-18-2020, 09:56 PM |
| |
|
Thread: Most Compatible Text Editor to Handle Large Files?
Post: Most Compatible Text Editor to Handle Large Files?
| Hey Everyone,
I often deal with large text files; at minimum a few GBs, with >60 million rows (cols = 4).
I was wondering what is the most suitable (and freely available) text editor to view the ... |
|
Robotguy |
Data Science |
2 |
2,393 |
Aug-13-2020, 05:49 PM |
| |
|
Thread: Merging sorted dataframes using Pandas
Post: Merging sorted dataframes using Pandas
| I have a large (Nx4, >10GB) array that I need to sort based on col.2.
I am reading my data in chunks and sorting using Pandas. But I am unable to combine the sorted chunks to give me a final large... |
|
Robotguy |
Data Science |
1 |
2,211 |
Aug-12-2020, 06:01 PM |
| |
|
Thread: Using Pytables to Sort Large Arrays?
Post: Using Pytables to Sort Large Arrays?
| Hi
I have two large input files (>10 GBs, Nx4). Task is to sort these files based on column-2 as fast as possible. Right now I am chunking and saving the sorted lines in text files (code below). T... |
|
Robotguy |
Data Science |
0 |
2,012 |
Aug-12-2020, 03:35 PM |
| |
|
Thread: Automating to save generated data
Post: RE: Automating to save generated data
| Using mapper[:] instead of mapper on line-7 and line-12 (in the shared code) solved the problem.
Thanks again! |
|
Robotguy |
Data Science |
3 |
2,270 |
Aug-12-2020, 03:32 PM |
| |
|
Thread: Automating to save generated data
Post: RE: Automating to save generated data
| Thanks, I tried that. Although it runs fine, I wonder why the information is not stores in the map.dat files created.
That is, when I read via print(f[:10]), I get all zeros. Do you know what's happe... |
|
Robotguy |
Data Science |
3 |
2,270 |
Aug-12-2020, 02:07 AM |
