Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
General Curiosity
#1
Hi All,

I have been working in Python from a data scientist perspective for a couple of months now. I must admit that I am in love. I come from a very strong TSQL background in my everyday job, but studied software engineering. Thus the more I work with Python the more questions I get.

A few weeks ago I built my first little app to assist me with my workload, a few co-workers saw it and asked if I could share. This go me thinking, I am running the script in an environment I setup for myself. So I will need to package it for their use, however I tried to do research on how to package this and the best solution I could find was pyToExe and was wondering what is the best way to package my app?

I now wanted to version my little app as I still need to make a lot of changes and will need to track which version everyone is using if they have issues. However I am not sure what the correct way is to achive this in my scripts.

I do most of my analysis in pandas, and afterwards I normally require to move my now transformed data back to SQL. I normally achieve this using the to_sql function included in pandas with a sqlalchemy engine defined, but with big data this is terribly slow due to the fact that this happens row by row. Is there a better way to move batch data from a pandas dataframe to sql server?

Please point me into the right direction.
Reply
#2
Some general thoughts:

1. Use a version control system (e.g. Git) to track changes you make to the code. The commit hashes can be used for versioning.

2. Write automated tests (unit tests, for example) for your application so that you're sure it works as it should and continues to do so as you make changes.
Reply
#3
(Oct-13-2019, 03:11 PM)JVoogt Wrote: but with big data this is terribly slow due to the fact that this happens row by row. Is there a better way to move batch data from a pandas dataframe to sql server?

Have you tried solutions proposed in to_sql is too slow
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020