String to List question help

James_Thomas · Sep-05-2023, 03:52 PM

Hello, I have been working with PyVISA doing S_param measurements. It is extremely convinent on my set up to just pull SNP (Touchstone files) because it can do it all in one sweep. I am having trouble splitting the tring to individual lists. I was hoping I could get some help with this problem and see how smarter people would go about solving this issue. I've attached a picture of how the string is set up, i was wondering how to go about splitting the string into seperate lists i.e Freq, S11M, S11A, etc. The data is delimited with spaces but I can not get it to go into seperate lists. I have gotten it to go with every signal digit being its own element and not being seperated by the space.

**deanhystad** · (This post was last modified: Sep-05-2023, 06:45 PM by deanhystad.)

Please post text, not screenshots. When you post text, others can easily copy/paste to create data for their own testing.

It looks like this would be easy using pandas.

I mocked up some data that looks something like yours. Tab delimited values with some lines at the top that need to be skipped, and an annoying "!" before the column headers.

Output:# 34589043! FILE NAME
! DATE NOW
! RAW DATA
# More information
!   FREQ	S11M	S21M	S21A
0.11444405920970335	0.22851792934978143	0.16724464908750036	0.054813079444560264
0.11565032195951042	0.23010933442970138	0.3048773751796404	0.11465260170716174
0.20102541854079847	0.27830101316888567	0.3995954975408248	0.12034618280172404
0.41276142607996513	0.31147127777115224	0.46636510024384636	0.2806783250819618
0.495356229079312	0.3495931528852747	0.6389401268977468	0.38079631999232333
0.556074773658631	0.3894689120388013	0.6779475794713076	0.43728550829612134
0.6332585933784733	0.4924761223011904	0.7503465368809027	0.7386632480838207
0.6341059239183447	0.5130383410762516	0.8554371206142789	0.7695854491274398
0.8902822004610119	0.7219019187548961	0.8712110079483135	0.8443884202719537
0.9170577193238573	0.7970927214052331	0.9344766589022325	0.8597198838156557

This code reads the file to extract the column names, then uses pandas to read the file into a dataframe.

import pandas as pd

# Read column names from 4th line in file.  Remove leading "!"
with open("data.csv", "r") as file:
    columns = file.readlines()[4][1:].split()

# Read file into dataframe.  Skip the first 5 rows and use the column
# names from above.
df = pd.read_csv("data.csv", sep="\t", skiprows=5, names=columns)
print(df)

Output:       FREQ      S11M      S21M      S21A
0  0.114444  0.228518  0.167245  0.054813
1  0.115650  0.230109  0.304877  0.114653
2  0.201025  0.278301  0.399595  0.120346
3  0.412761  0.311471  0.466365  0.280678
4  0.495356  0.349593  0.638940  0.380796
5  0.556075  0.389469  0.677948  0.437286
6  0.633259  0.492476  0.750347  0.738663
7  0.634106  0.513038  0.855437  0.769585
8  0.890282  0.721902  0.871211  0.844388
9  0.917058  0.797093  0.934477  0.859720

Now it is easy to get the columns.

print(df["FREQ"])

Output:0    0.114444
1    0.115650
2    0.201025
3    0.412761
4    0.495356
5    0.556075
6    0.633259
7    0.634106
8    0.890282
9    0.917058
Name: FREQ, dtype: float64

James_Thomas · Sep-06-2023, 09:14 AM

This is a neat way of doing it. the difficulty is that I am importing the data directly from my machine through GPIB and its importing as a str "type" and I am not actually calling a CSV file. So Im stuck at being able to seperate the string into seperate columns. The screen shot of the data I show is how the Python imports the data directly from the machine.

(Sep-05-2023, 06:45 PM)deanhystad Wrote: Please post text, not screenshots. When you post text, others can easily copy/paste to create data for their own testing.

It looks like this would be easy using pandas.

I mocked up some data that looks something like yours. Tab delimited values with some lines at the top that need to be skipped, and an annoying "!" before the column headers.

Output:# 34589043! FILE NAME
! DATE NOW
! RAW DATA
# More information
!   FREQ	S11M	S21M	S21A
0.11444405920970335	0.22851792934978143	0.16724464908750036	0.054813079444560264
0.11565032195951042	0.23010933442970138	0.3048773751796404	0.11465260170716174
0.20102541854079847	0.27830101316888567	0.3995954975408248	0.12034618280172404
0.41276142607996513	0.31147127777115224	0.46636510024384636	0.2806783250819618
0.495356229079312	0.3495931528852747	0.6389401268977468	0.38079631999232333
0.556074773658631	0.3894689120388013	0.6779475794713076	0.43728550829612134
0.6332585933784733	0.4924761223011904	0.7503465368809027	0.7386632480838207
0.6341059239183447	0.5130383410762516	0.8554371206142789	0.7695854491274398
0.8902822004610119	0.7219019187548961	0.8712110079483135	0.8443884202719537
0.9170577193238573	0.7970927214052331	0.9344766589022325	0.8597198838156557

This code reads the file to extract the column names, then uses pandas to read the file into a dataframe.

import pandas as pd

# Read column names from 4th line in file.  Remove leading "!"
with open("data.csv", "r") as file:
    columns = file.readlines()[4][1:].split()

# Read file into dataframe.  Skip the first 5 rows and use the column
# names from above.
df = pd.read_csv("data.csv", sep="\t", skiprows=5, names=columns)
print(df)

Output:       FREQ      S11M      S21M      S21A
0  0.114444  0.228518  0.167245  0.054813
1  0.115650  0.230109  0.304877  0.114653
2  0.201025  0.278301  0.399595  0.120346
3  0.412761  0.311471  0.466365  0.280678
4  0.495356  0.349593  0.638940  0.380796
5  0.556075  0.389469  0.677948  0.437286
6  0.633259  0.492476  0.750347  0.738663
7  0.634106  0.513038  0.855437  0.769585
8  0.890282  0.721902  0.871211  0.844388
9  0.917058  0.797093  0.934477  0.859720

Now it is easy to get the columns.

print(df["FREQ"])

Output:0    0.114444
1    0.115650
2    0.201025
3    0.412761
4    0.495356
5    0.556075
6    0.633259
7    0.634106
8    0.890282
9    0.917058
Name: FREQ, dtype: float64

Pedroski55 · Sep-06-2023, 10:59 AM

Can't you just save the string as text, then follow deanhystad's elegant answer?

I just copied the output he posted into a .txt file and then followed his answer, no problems!

I don't know much about pandas, or what GPIB is!

**deanhystad** · (This post was last modified: Sep-06-2023, 12:22 PM by deanhystad.)

No problem. Same solution with a string. Why did you mention touchstone files in the OP?

import pandas as pd
from io import StringIO  # Might need to use TextIO for input from GPIB.

gpib_string = """# 34589043! FILE NAME
! DATE NOW
! RAW DATA
# More information
!   FREQ	S11M	S21M	S21A
0.11444405920970335	0.22851792934978143	0.16724464908750036	0.054813079444560264
0.11565032195951042	0.23010933442970138	0.3048773751796404	0.11465260170716174
0.20102541854079847	0.27830101316888567	0.3995954975408248	0.12034618280172404
0.41276142607996513	0.31147127777115224	0.46636510024384636	0.2806783250819618
0.495356229079312	0.3495931528852747	0.6389401268977468	0.38079631999232333
0.556074773658631	0.3894689120388013	0.6779475794713076	0.43728550829612134
0.6332585933784733	0.4924761223011904	0.7503465368809027	0.7386632480838207
0.6341059239183447	0.5130383410762516	0.8554371206142789	0.7695854491274398
0.8902822004610119	0.7219019187548961	0.8712110079483135	0.8443884202719537
0.9170577193238573	0.7970927214052331	0.9344766589022325	0.8597198838156557"""

# Read column names from 4th line in string.  Remove leading "!"
columns = gpib_string.split("\n")[4][1:].split()
 
# Read file into dataframe.  Skip the first 5 rows and use the column
# names from above.

df = pd.read_csv(StringIO(gpib_string), sep="\t", skiprows=5, names=columns)
print(df)

James_Thomas · (This post was last modified: Sep-06-2023, 02:34 PM by deanhystad.)

Hey Sorry about that, I must've not made it clear. Let me just say how much I appreciate the help. Everything works and I am getting a list correctly except its placing all of the data in the first column.
I have attached a photo of how it outputs the data. Its so close. I really appreciate the help. Can I buy you a cup of coffee or beer?

here is my code;

import pyvisa 
import numpy as np
import os
import matplotlib.pyplot as plt 
import csv
import pandas as pd
from io import StringIO
rm = pyvisa.ResourceManager()
VNA = rm.open_resource('GPIB0::6::INSTR')
print(VNA.query("*IDN?"))

print("Please wait, retrieving data...")
	## Get the VNA data
VNA = rm.open_resource('GPIB0::6::INSTR')
VNA.write('CH1')
f = VNA.query("OS2P")
flist = f.split("\n")[4][1:].split()

df = pd.read_csv(StringIO(f), sep ="\t", skiprows=5, names=flist)

print(df)
VNA.close()

(Sep-06-2023, 12:22 PM)deanhystad Wrote: No problem. Same solution with a string. Why did you mention touchstone files in the OP?

import pandas as pd
from io import StringIO  # Might need to use TextIO for input from GPIB.

gpib_string = """# 34589043! FILE NAME
! DATE NOW
! RAW DATA
# More information
!   FREQ	S11M	S21M	S21A
0.11444405920970335	0.22851792934978143	0.16724464908750036	0.054813079444560264
0.11565032195951042	0.23010933442970138	0.3048773751796404	0.11465260170716174
0.20102541854079847	0.27830101316888567	0.3995954975408248	0.12034618280172404
0.41276142607996513	0.31147127777115224	0.46636510024384636	0.2806783250819618
0.495356229079312	0.3495931528852747	0.6389401268977468	0.38079631999232333
0.556074773658631	0.3894689120388013	0.6779475794713076	0.43728550829612134
0.6332585933784733	0.4924761223011904	0.7503465368809027	0.7386632480838207
0.6341059239183447	0.5130383410762516	0.8554371206142789	0.7695854491274398
0.8902822004610119	0.7219019187548961	0.8712110079483135	0.8443884202719537
0.9170577193238573	0.7970927214052331	0.9344766589022325	0.8597198838156557"""

# Read column names from 4th line in string.  Remove leading "!"
columns = gpib_string.split("\n")[4][1:].split()
 
# Read file into dataframe.  Skip the first 5 rows and use the column
# names from above.

df = pd.read_csv(StringIO(gpib_string), sep="\t", skiprows=5, names=columns)
print(df)

deanhystad write Sep-06-2023, 02:34 PM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

**deanhystad** · (This post was last modified: Sep-06-2023, 02:32 PM by deanhystad.)

The screenshot looked like tabs. That is another advantage of posting text over screenshots. I would have known the columns were separated by spaces, sometimes multiple spaces. Again, not a problem. Pandas if accommodating. Change the separator to a space, and skipinitialspace to take care of when there are multiple spaces between columns.

df = pd.read_csv(StringIO(gpib_string), sep=" ", skiprows=5, names=columns, skipinitialspace=True)

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	syntax error question - string	mgallotti	5	1,334	Feb-03-2023, 05:10 PM Last Post: mgallotti
	General programming question (input string)[	jamie_01	2	1,613	Jan-08-2022, 12:59 AM Last Post: BashBedlam
	Question about change hex string to integer sting in the list (python 2.7)	lzfneu	1	2,545	May-24-2021, 08:48 AM Last Post: bowlofred
	Question on HTML formatting with set string in message	Cknutson575	3	3,514	Mar-09-2021, 08:11 AM Last Post: Cknutson575
	question: finding multiple strings within string	djf123	4	3,004	May-16-2020, 01:00 PM Last Post: snippsat
	Simple newb string question	Involute	2	2,231	Sep-08-2019, 12:50 AM Last Post: Involute
	string parsing question ( I think)	diggee17	4	3,026	Jul-24-2019, 02:37 PM Last Post: diggee17
	I converted string to 'list', but it doesn't look like a list!	mrapple2020	3	3,263	Apr-07-2019, 02:34 PM Last Post: mrapple2020
	please help with this question about using operators to multiply a string?	GilesTwigg	3	4,407	Feb-27-2019, 04:13 PM Last Post: ichabod801
	about List question	longmail	2	2,366	Nov-30-2018, 02:08 AM Last Post: longmail

String to List question help

User Panel Messages

Announcements