Python Forum
Automating a Data Extraction Process
Thread Rating:
  • 3 Vote(s) - 3.33 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Automating a Data Extraction Process
#1
Hi everyone,

First of all, this is my first proper post on Python-Forum, so please excuse any poor forum etiquette on my part... Blush

I want to automate a repetitive computational procedure - nothing new, right! Unfortunately, I have no experience in programming, so in fact it is new for me...

Rather than explain everything here, I have put together a PDF which clearly details the steps which I would like to automate using Python:
 
https://www.dropbox.com/s/ng4573puyyq0o2...n.pdf?dl=0

Judging from other threads, you guys seem very helpful. This would genuinely be a useful tool for me, which would allow me to easily extract data from the 3D simulation package that I use.

If this program will be too difficult to create in a short time, please let me know why!

Let me know what you think, guys!

Thanks,
Harrison
Reply
#2
As described, this is just making Python replace your fingers. What are the chances that these applications have a programming interface that could be used directly? Since these apps are professional engineering apps, they likely stick to some standard and there may be python libraries to use the data directly.
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Reply
#3
As a general rule, we don't write things for people. We help them to figure out how they can do it themselves.
If that's something you're interested in, we can offer tips on how to get started.
If that's NOT what you're after, and was just hoping to have a working solution, there are a couple other directions you can go (such as Fiverr.com).
Reply
#4
Hi guys,

Thanks for your replies!

@Ofnuts, I thought the fact that I am constantly jumping between applications would mean that I wouldn't be able to use a specific application's programming interface? Tell me if I'm wrong - I am fully aware that I have no knowledge of such things!

@nilamo, I am happy to learn how to program. I just thought that it was slightly inefficient considering there are probably others who could create an outline of some code for me with relative ease. I appreciate that I'm a newbie and would welcome any help or advice!

I'll pay $80 to someone who can produce a template for me :)
Reply
#5
Take  a look at Automate the Boring Stuff, Pywinauto.
Reply
#6
(Mar-30-2017, 09:06 PM)snippsat Wrote: Take  a look at Automate the Boring Stuff, Pywinauto.

Thanks snippsat, will check that out
Reply
#7
(Mar-30-2017, 08:16 PM)Harrison Wrote: @Ofnuts, I thought the fact that I am constantly jumping between applications would mean that I wouldn't be able to use a specific application's programming interface? Tell me if I'm wrong - I am fully aware that I have no knowledge of such things!
The fact that you are jumping across applications only means that you have never considered that there could be ways to do it without a mouse. How it can be done programmatically is however still subject to study and for this one would need to know what each application really does (where are the docs...) and what you are putting in/getting from them. Ideally the whole process can be re-engineered to run without involving the graphical interfaces. Using the window-scraping technique you describe should only be a fall-back plan.

(Mar-30-2017, 08:16 PM)Harrison Wrote: I'll pay $80 to someone who can produce a template for me :)

Reminds me of the proverbial sign in mechanics shops:

Car repairs: $40/hour
Car repairs while you watch: $60/hour
Car repairs while you help: $80/hour

Without seeing your software, if I were a freelancer I would contract for two weeks (and hoping to pull it off in three days of course, but there can be unforeseen problems). Add to this that you have to learn programming, which means 1) learning a language such as Python and 2) acquiring a vast computer culture so that you don't drown under the rain of little problems inherent to this kind of project (file access, text encoding, data representations...).

PS: why photos and not proper screenshots ([prtsc] key)
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Reply
#8
In the 1970's George Morrow owned a company he named Thinker-Toys.
He sold Single Board Computers.
His adds all read similar to this:
Kit $399 Fully Assembled $199
Reply
#9
(Mar-31-2017, 12:04 AM)Ofnuts Wrote: The fact that you are jumping across applications only means that you have never considered that there could be ways to do it without a mouse. How it can be done programmatically is however still subject to study and for this one would need to know what each application really does (where are the docs...) and what you are putting in/getting from them. Ideally the whole process can be re-engineered to run without involving the graphical interfaces. Using the window-scraping technique you describe should only be a fall-back plan.

Thanks for the reply, Ofnuts.

Would it be easier to do it programmatically? I assumed that getting a program to click an 'OK' button would be easier than going into the file locations. Each simulation which I am running is saved in a different folder, but in the same drive. The applications automatically detect the specific locations. I appreciate that it would be nice to have the program go to the correct locations but, as I say, I assumed this route may be easier. Again, tell me if I'm wrong!
Reply
#10
Click an "OK" button by program can be hard. When you are a program you don't really know where some other program's buttons are. If the program uses the standard widgets that can work, but some programs (especially those that have implementations for different OSes) don't.

Several years ago, I read a blog in which the author related that during a mission in some south-american administration he found a guy whose only job was to:


  1. open a directory in the file explorer
  2. take a screenshot
  3. save the screen shot as an image file
  4. open that file in an OCR program
  5. save the result (or append it to the existing list if he was scrolling down the directory)
Being helpful, he showed the guy that he could do the same with dir *.* >files.txt but the guy made it clear that if he didn't want to lose his job...

So, when asking a computer person, come with your raw problem, because the solution you would come up with may not be the most efficient/easy one.
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs: gimp-forum.net
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How should I process this data? SuchUmami 2 348 Jan-13-2024, 05:30 PM
Last Post: RockBlok
  How would I process this data? SuchUmami 2 515 Nov-12-2023, 10:05 AM
Last Post: SuchUmami
  Reading data from excel file –> process it >>then write to another excel output file Jennifer_Jone 0 1,047 Mar-14-2023, 07:59 PM
Last Post: Jennifer_Jone
Question Is there a python program for automating this invoicing workflow? PG_Archipelago 3 1,029 Feb-02-2023, 11:01 PM
Last Post: Larz60+
  Data extraction from a table based on column and row names tgottsc1 1 2,368 Jan-09-2021, 10:04 PM
Last Post: buran
  Help with Creating a Script for Automating Reports SunWers 1 1,882 Dec-29-2020, 10:21 PM
Last Post: jjc385
  Automating to run python script 100 times by changing parameters pmt 1 2,562 Dec-29-2020, 10:31 AM
Last Post: andydoc
  eml file data extraction ajetrumpet 2 2,571 Jul-04-2020, 04:34 AM
Last Post: ajetrumpet
  Process Data from one csv file and write to another CSV file specific column ajin9581 1 1,943 Jun-17-2020, 06:00 PM
Last Post: buran
  Automating Windows GUI applications metro17 4 12,176 Feb-10-2020, 09:46 AM
Last Post: metro17

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020