Python Forum
Git clone all of a Github user's public repositories (download all repositories)
Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Git clone all of a Github user's public repositories (download all repositories)
#1
A simple script to download a user's Github repositories... only their own though... not forked ones



#! /usr/bin/python3.7
# rootVIII
# Download/clone all of a user's public repositories
# Pass the Github user's username with the -u option
# Usage: python git_clones.py -u <github username>
# Example: python git_clones.py -u rootVIII
#
from argparse import ArgumentParser
from sys import exit
from re import findall
from urllib.request import urlopen
from subprocess import call


class GitClones:
    def __init__(self, user):
        self.url = "https://github.com/%s?tab=repositories" % user
        self.git_clone = "git clone https://github.com/%s/" % user
        self.git_clone += "%s.git"
        self.user = user

    def get_repo_data(self):
        try:
            r = urlopen(self.url)
        except Exception:
            print("Unable to make request to %s's Github page" % self.user)
            exit(1)
        else:
            pattern = r"repository_nwo:%s/(.*)," % self.user
            for line in findall(pattern, r.read().decode('utf-8')):
                yield line.split(',')[0]

    def get_repositories(self):
        return set([repo for repo in self.get_repo_data()])

    def download(self, git_repos):
        for git in git_repos:
            cmd = self.git_clone % git
            try:
                call(cmd.split())
            except Exception as e:
                print(e)
                print('unable to download:%s\n ' % git)


if __name__ == "__main__":
    message = 'Usage: python git_clones.py -u <github username>'
    h = 'Github Username'
    parser = ArgumentParser(description=message)
    parser.add_argument('-u', '--user', required=True, help=h)
    d = parser.parse_args()
    clones = GitClones(d.user)
    repositories = clones.get_repositories()
    clones.download(repositories)
Reply
#2
Im just posting to stop the unread notifications every time you modify the OP code
Recommended Tutorials:
Reply
#3
Edited 11 times ?? someone posted buggy code to begin with?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
No not buggy... just kept realizing that certain lines could be combined/shortened (after the fact)
Reply
#5
The script does nothing on Linux Mint 19 (python 3.6.8)
Reply
#6
Which user's repositories are you trying to clone? It will only clone the repositories actually made by the user.. not forked ones. Linux Mint should have nothing to do with it... I run this on Ubuntu, CentOS, and Kali with no issues.

Actually nothing in the code is OS specific whatsoever... It should run even on Windows to... haven't tried it but system should not be a factor

Screenshots on my Kali Desktop
Reply
#7
the code looked OK to me. i was thinking of using on AWS' repositories there.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#8
(Jul-04-2019, 04:19 PM)rootVIII Wrote: Which user's repositories are you trying to clone?

I tried it with my own.(and others)
Reply
#9
Hmmmm yeah just tried to get all your repos and nothing... hold on looking into it now. I suspect the regex

Yup it's the regex... should have a fix in a little bit... sorry for that

Okay just got all your repos... had to fix that regex. Also this one should run with Python 2 or 3...

# rootVIII
# Download/clone all of a user's public repositories
# Pass the Github user's username with the -u option
# Usage: python git_clones.py -u <github username>
# Example: python git_clones.py -u rootVIII
#
from argparse import ArgumentParser
from sys import exit, version_info
from re import findall
from subprocess import call
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import Request, urlopen


class GitClones:
    def __init__(self, user):
        self.url = "https://github.com/%s?tab=repositories" % user
        self.git_clone = "git clone https://github.com/%s/" % user
        self.git_clone += "%s.git"
        self.user = user

    def http_get(self):
        if version_info[0] != 2:
            req = urlopen(self.url)
            return req.read().decode('utf-8')
        req = Request(self.url)
        request = urlopen(req)
        return request.read()

    def get_repo_data(self):
        try:
            response = self.http_get()
        except Exception:
            print("Unable to make request to %s's Github page" % self.user)
            exit(1)
        else:
            pattern = r"<a\s?href\W+%s/(.*)\"\s+" % self.user
            for line in findall(pattern, response):
                yield line.split('\"')[0]

    def get_repositories(self):
        return set([repo for repo in self.get_repo_data()])

    def download(self, git_repos):
        for git in git_repos:
            cmd = self.git_clone % git
            try:
                call(cmd.split())
            except Exception as e:
                print(e)
                print('unable to download:%s\n ' % git)


if __name__ == "__main__":
    message = 'Usage: python git_clones.py -u <github username>'
    h = 'Github Username'
    parser = ArgumentParser(description=message)
    parser.add_argument('-u', '--user', required=True, help=h)
    d = parser.parse_args()
    clones = GitClones(d.user)
    repositories = clones.get_repositories()
    clones.download(repositories)
Reply
#10
No problem.

On github I use Axel-Erfurt as username
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Pong clone with classes OhNoSegFaultAgain 1 3,850 May-11-2019, 07:44 PM
Last Post: keames

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020