Python Forum
Yahoo_fin, Pandas: how to convert data table structure in csv file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Yahoo_fin, Pandas: how to convert data table structure in csv file
#11
buran, sorry about that. I will refrain from removing posted content in the future. Thank you for the advice about df.pivot. That was good advice. Also thank you for keeping my posts containing private matters deleted.

BTW, I copied the final code with the 2 lines about csv import and conversion of column to list to the entry above the entry containing the VBA code. I placed the final solution there so that interested people find the final solution quickly.

Regarding the advice to read the csv file into python rather than using Excel VBA to create a string that then needs to be pasted into the python script. That I do not consider an advice. It should be obvious to anybody with just a little programming skills that that is possible and more efficient. As I explained in my now removed post, due to private matters posted, I was just not willing to spend the time figuring out how to do it. I had other priorities. Writing the VBA took me a couple minutes. But eventually my curiosity took over and I did put in the work to learn how to do it in python. Figuring out how to read the csv and that I need to convert the ticker column to a list before it can be further processed took me a couple hours. If I would have been given an example on how this is done conceptually I could have learned it in a few minutes. That would have been advice. Yes, advice should include encouraging to think, but it should much more importantly help in speeding up the learning process. But to your defense I have to add that I discouraged you to give me advice in the form of code. But bottom line, as is, your input should, in my opinion, not counted as advice.

You can delete this reply. You have my consent.
Reply
#12
(Feb-15-2021, 06:05 AM)detlefschmitt Wrote: Regarding the advice to read the csv file into python rather than using Excel VBA to create a string that then needs to be pasted into the python script. That I do not consider an advice. It should be obvious to anybody with just a little programming skills that that is possible and more efficient. As I explained in my now removed post, due to private matters posted, I was just not willing to spend the time figuring out how to do it. I had other priorities. Writing the VBA took me a couple minutes. But eventually my curiosity took over and I did put in the work to learn how to do it in python. Figuring out how to read the csv and that I need to convert the ticker column to a list before it can be further processed took me a couple hours. If I would have been given an example on how this is done conceptually I could have learned it in a few minutes. That would have been advice. Yes, advice should include encouraging to think, but it should much more importantly help in speeding up the learning process. But to your defense I have to add that I discouraged you to give me advice in the form of code. But bottom line, as is, your input should, in my opinion, not counted as advice.

I did show how to read csv file. In my very first post in this thread. So, only you if did pay attention...
(Feb-12-2021, 09:47 AM)buran Wrote:
import pandas as pd
df = pd.read_csv('test.csv')

Then, converting dataframe column to list is google search away. Not that you need to convert to list, you can just iterate over dataframe column values.
By the way, same apply to reading from csv or excel, if you ignored my first post.
So, couple of hours spent on this is your problem, not my fault.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#13
I don't blame you for having to spend a couple hours figuring out a 2-line code.

The problem was not reading the csv. That info is provided as the first line of code on any website explaining how to import a csv. You must be greedy for advice credit if you want to claim that as advice.

The problem was realizing that I need to convert the column to a list. That's what took time. Once I saw that on a website I realized immediately that this is the solution. With some trial and errors I finally came up with the very simple code. If you would have told me that after csv import the ticker column needs to be converted to a list, I would have needed only a fraction of the time. And that would have been a good advice.

Quote:why not learn how to read from a file, e.g. csv/text file with all tickers. instead of constructing a big literal.
And, please, make an effort first. You can read it in number of ways, incl. pandas obviously
That is not an advice.

Quote:I've been working with VBA/Office automation for years. Almost never looked back after I started with python. And you already know how to read data from a file into pandas dataframe and loop over it - what's the problem to read a list of tickers from cav or excel? Basically there is no learning here.
On a broader scale - isn't it better to store data in a DB and then connect from Excel, either with Power Query or VBA/ADODB if you prefer to manipulate your data in Excel?
That is also not an advice.

I am not a bioinformatics person nor an IT guy. At work I never had to deal with DBs that would require the use of Power Query or VBA/ADODB to transfer data to Excel. The only time I used Power Query was to obtain financial statistics data from Yahoo Finance for personal reasons. So, I really don't have much experience with the input and output of data between DBs and Excel. Hence it took time to realize that I need to convert the column to a list. One can have a lot of experience with Excel VBA in one area and not know much in another. On top my knowledge in python is essentially zero.
Reply
#14
So, this is my last post here. This is not about credit for advice (or non-advice). This is about you acting like help vampire from the start and ever so more with each consecutive post.
As I already said, converting to list is not necessary at all in the first place, and second it's a google search away - right from the first result. In any case, it takes less time to do the search, than to write these lengthy posts about getting advice or non-advice. And no where close to spending couple of hours on it.
I don't know how much experience you have with VBA (now it's obvious that not much), so I just asked about why not using alternatives with Power Query or ADODB. You could have ignored it or simply do your research about it, if you are curious.

Anyway, I am done with your threads.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#15
Help vampire. hahahahaha. I think it is exaggerated to call me that after having asked two questions on this forum. I didn't asked for help about importing csv and incorpotaring csv column data into the python script. You volunteered that on your own after I provided the Excel VBA code. I merely said in that post that alternatively instead of using Excel VBA it could be done with python code but I didn't know how to do it. I really didn't want to spend the time to figure it out on my own. So, I left it to somebody else to post it. I did not need the python solution. I was fully happy using the VBA instead.

About my VBA skill. 35 years ago I took a couple semester of Turbo Pascal as an undergraduate. After that I did not program for 25 years. Then 10 years ago we tried in the office to solve the problem of finding key opinion leaders in various disease areas. My opinion was that key opinion leaders can be found in the publication literature and that they could be extracted from the literature database using Excel. I had a very clear idea about how to do that but I had no VBA skills. So, I joined an Excel VBA forum and I broke up the large computation work flow into very small steps. At each step where I didn't know how to code it I asked on the forum. I always asked using an example unrelated to my project that would teach me the concept of a particular programing step. By literally asking a hundred of such questions I finally finished my program. Despite all these questions I absolutely do not think that I acted like a help vampire. The help in the Excel VBA forum was absolutely outstanding and none of the experts complained. By the end of the project I had learned enough to continue developing other programs on my own. I can only recommend learning a programming language this way. It was fun and efficient.

After this first program I developed Excel VBA programs to do text mining, clinical trial data analysis, databasing publication records, various programs to do analysis of chemical and pharmacological data. And most recently I have done a large conceptual work to show that data-to-text (NLG) can be done in Excel using VBA. Data-to-text is a big issue in pharma especially in paperwork preparation for clinical trial data filing. All this was possible because of what I learned in developing the initial key opinion leader program. Yet, I would consider my skill not higher than intermediate. I am aware of various areas in VBA programming where I know very little about. I learn only what I need. After all, programing for me has always remained only a side project at work. I am scientist, not a programmer.

In all my interactions with VBA experts on the Excel VBA forum I never had an awful experience like this one in the python forum. And that basically in my first interaction on this forum. It also never happened on the VBA forum that somebody claimed an advice where there was none. I am the first to give credit where credit is due. But I am also the first to call BS.

That's it for me for this post. No further responses. I suggest you delete all posts below the post with the VBA code.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python Pandas: How do I average ONLY the data >1000 from several columns? JaneTan 0 334 Jul-17-2021, 01:34 PM
Last Post: JaneTan
  Python Pandas: How do I extract all the >1000 data from a certain column? JaneTan 0 305 Jul-17-2021, 09:09 AM
Last Post: JaneTan
  How to convert binary data into text? ZYSIA 3 566 Jul-16-2021, 04:18 PM
Last Post: deanhystad
  SaltStack: MySQL returner save less data into Database table columns xtc14 2 387 Jul-02-2021, 02:19 PM
Last Post: xtc14
  Need help on extract dynamic table data Dr_Strange 0 728 Apr-30-2021, 07:03 AM
Last Post: Dr_Strange
  Load the data from multiple source files to one table amy83 2 671 Apr-27-2021, 12:33 AM
Last Post: Pedroski55
  code for CSV file to html file without pandas jony057 1 620 Apr-24-2021, 09:41 PM
Last Post: snippsat
  Appropriate data-structure / design for business-day relations (week/month-wise) sx999 2 815 Apr-23-2021, 08:09 AM
Last Post: sx999
  Convert MultiLayer XML to DataFrame using Pandas vsingh17 0 647 Apr-14-2021, 03:50 PM
Last Post: vsingh17
  what data structure to use? Winfried 4 720 Mar-16-2021, 12:11 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020