Python Forum
Importing Postgres Heroku from AWS S3
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Importing Postgres Heroku from AWS S3
#1
I’m learning how to handle Postgres instances by backing them up and restoring them on Heroku for a Django project (a small rudimentary CMS). The amount of data is a few hundred kilobytes because it's just text that I am storing in my my db. I'm practicing backups and restores just to learn for fun.

I realize this is loosely related to Python/Django, but it does fall into the general category of development / programming. I hope my post is welcome here.

I downloaded the binary data to my local machine using this particular section of the Heroku doc.

The next step was to create an AWS account, including setting up Access Keys which I located in the dashboard and entered them into my local dev environment. I named my bucket. I uploaded the binary to S3.

I’ve made it all the way to the end of Heroku’s import Postgres guide.

I install the awscli package with pip which enabled me to presign my s3 bucket (which succeeded).

I am right at the final step of importing my backup to Heroku Postgres. I am so close!

My traceback at this point indicates that Heroku is expecting an HTTP 200 (request has succeeded) but instead it receives an HTTP 400 (can’t process) ‘due to the source URL being inaccessible’. This points towards the restrictive permissions in place on my AWS S3 bucket.

You can find my traceback in full at the bottom of this post.

With regards to my AWS S3 bucket, in the dashboard, the main Permissions switch relevant here is the “Block all public access” option. Whether this checkbox is enabled or disabled (I carefully tried both), I encountered the same HTTP 200/400 in my traceback. This is where I believe the issue is.

I’m not sure what else to try. I’m also a little concerned that with the vast number of variables available for Amazon’s S3 service, I don’t know how I might share or export my configuration nicely for you people to take a closer look. What other information could I provide to better help you people help me?

Here is the restore command I am using:

Quote:$ heroku pg:backups:restore 'https://postgres-restore-<project-name>.s3.amazonaws.com/2021May<hash redacted>aa9376\?X-Amz-Algorithm\=AWS4-HMAC-SHA256\&X-Amz-Credential\=AKIAY<hash redacted>0527%2FOhio%2Fs3%2Faws4_request\&X-Amz-Date\=2021<hash redacted>33Z\&X-Amz-Expires\=3600\&X-Amz-SignedHeaders\=host\&X-Amz-Signature\=19<hash redacted>17'

Here is the output:

Quote: HEROKU_POSTGRESQL_PUCE -a <project-name>
› Warning: heroku update available from 7.50.0 to 7.51.0.
▸ WARNING: Destructive Action
▸ This command will affect the app <project-name>
▸ To proceed, type <project-name> or re-run this command with
▸ --confirm <project-name>

> <project-name>
Starting restore of https://postgres-restore-<project-name>....\=199<hash redacted>d86a0317 to postgresql-animate-93816... done

Use Ctrl-C at any time to stop monitoring progress; the backup will continue restoring.
Use heroku pg:backups to check progress.
Stop a running restore with heroku pg:backups:cancel.

Restoring... !
▸ An error occurred and the backup did not finish.

▸ waiting for restore to complete
▸ pg_restore finished with errors
▸ waiting for download to complete
▸ download finished with errors
▸ please check the source URL and ensure it is publicly accessible

▸ Run heroku pg:backups:info r019 for more details.

$ heroku pg:backups:info r019 -a <project-name>
› Warning: heroku update available from 7.50.0 to 7.51.0.
=== Backup r019
Database: BACKUP
Started at: 2021-05-27 10:29:02 +0000
Finished at: 2021-05-27 10:29:02 +0000
Status: Failed
Type: Manual
Backup Size: 0.00B (0% compression)

=== Backup Logs
2021-05-27 10:29:02 +0000 2021/05/27 10:29:02 aborting: could not write to output stream: Expected HTTP Status 200, received: "400 Bad Request"
2021-05-27 10:29:02 +0000 pg_restore: error: could not read from input file: end of file
2021-05-27 10:29:02 +0000 waiting for restore to complete
2021-05-27 10:29:02 +0000 pg_restore finished with errors
2021-05-27 10:29:02 +0000 waiting for download to complete
2021-05-27 10:29:02 +0000 download finished with errors
2021-05-27 10:29:02 +0000 please check the source URL and ensure it is publicly accessible
$
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Star Leapcell: The Python-Friendly Alternative to Heroku + Airtable Hybrid IssacChan 1 396 Feb-01-2024, 06:00 AM
Last Post: Athi
  Deployed Spider on Heroku: How do I email downloaded files? JaneTan 2 1,523 Mar-24-2022, 08:31 AM
Last Post: JaneTan
  Heroku Error H10 jamesaarr 1 1,979 Oct-21-2021, 03:43 PM
Last Post: jamesaarr
  Django project deployed to Heroku: Postgres security Drone4four 0 1,912 Mar-26-2021, 10:17 AM
Last Post: Drone4four
  Flask Can't Save Screenshot to Postgres Db firebird 3 2,342 Sep-21-2020, 09:22 PM
Last Post: firebird
  Scrape script when deployed on heroku not giving expected outcomes alok001 1 3,227 Oct-19-2019, 12:39 PM
Last Post: alok001
  flask app to save images locally when deployed on heroku not working Prince_Bhatia 1 5,231 Feb-20-2019, 11:59 PM
Last Post: snippsat
  Deploy flask app on Heroku Prince_Bhatia 0 4,012 Feb-20-2019, 09:05 AM
Last Post: Prince_Bhatia
  Hosting statistic tool on heroku with flask secure? Zoja 1 2,841 Oct-29-2018, 10:07 AM
Last Post: Zoja

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020