Scraping from multiple URLS to print in a single line.

[jb89]

Hi there,

Please forgive me if I have trouble explaining myself, i'm quite new to Python.

Basically I've been tasked with scraping some information from an atlassian site that prints out the plans in a project and the repository's, variables and stages of that plan. I've managed to do this but it prints out each segment individually because i'm pulling data from 4 different URLS one at a time.

The print looks like this:
"Plans"
"Repos"
"Variables"
"Stages"

I've been tasked for it to print like this:
"Plans","Repos","Variables","Stages"

My code is below, thank you.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

import requests from bs4 import BeautifulSoup   params = {     'X-Atlassian-Token': 'no-check',     'Accept':'application/json',     'Content-Type':'application/x-www-form-urlencoded' }   # List of Plans in a project. r = requests.get(url='https://xxxx/project/viewProject.action?projectKey=xxxx', params=params, auth=('xxxx', 'xxxx'), verify=False) soup = BeautifulSoup(r.text, 'html.parser') for td in soup.findAll('td'):     if td.get_attribute_list(key='class')[0] == 'build':         print(td.text)   # List of Repos in a plan. r = requests.get(url='https://xxxx/chain/admin/config/editChainRepository.action?buildKey=xxxx', params=params, auth=('xxxx', 'xxxx'), verify=False) soup = BeautifulSoup(r.text, 'html.parser') for h3 in soup.findAll('h3'):     if h3.get_attribute_list(key='class')[0] == 'item-title':         print(h3.text)   # List of Variables in a plan. r = requests.get(url='https://xxxx/chain/admin/config/configureChainVariables.action?buildKey=xxxx', params=params, auth=('xxxx', 'xxxx'), verify=False) soup = BeautifulSoup(r.text, 'html.parser') for td in soup.findAll('td'):     if td.get_attribute_list(key='class')[0] == 'variable-key':         print(td.text)   # List of Stages in a plan. r = requests.get(url='https://xxxx/chain/admin/config/defaultStages.action?buildKey=xxxx', params=params, auth=('xxxx', 'xxxx'), verify=False) soup = BeautifulSoup(r.text, 'html.parser') for span in soup.findAll('span'):     if span.get_attribute_list(key='class')[0] == 'stage-name':         print('\t'+span.text)

[Larz60+]

one way to do this is create an empty string before scraping:
combined_text = ''
then add word after each site scraped:
combined_text = f"{combined_text} {new_word}"

[jb89]

Thank you Larz60+

Do you mean something like the below? I'm unsure what to enter for "new_word"

Thank you

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

combined_text = ''   # List of Plans in a project. r = requests.get(url='https://xxxx/project/viewProject.action?projectKey=xxxx', params=params, auth=('xxxx', 'xxxx'), verify=False) soup = BeautifulSoup(r.text, 'html.parser') for td in soup.findAll('td'):     if td.get_attribute_list(key='class')[0] == 'build':         print(td.text)         combined_text = f"{td.text}"   # List of Repos in a plan. r = requests.get(url='https://xxxx/chain/admin/config/editChainRepository.action?buildKey=xxxx', params=params, auth=('xxxx', 'xxxx'), verify=False) soup = BeautifulSoup(r.text, 'html.parser') for h3 in soup.findAll('h3'):     if h3.get_attribute_list(key='class')[0] == 'item-title':         print(h3.text)         combined_text = f"{td.text} {h3.text}"

[jb89]

I put each request into a function named plans, repos, variables and stages and tried the following, how ever it still prints out the same.

1 2	combine_text = f'{plans()}{repos()}{variables()}{stages()}' print(combine_text)

[perfringo]

Another option is to append values to list and print that list:

1 2 3

>>> lst = ["Plans","Repos","Variables","Stages"]                     >>> print(*lst, sep=', ')                                            Plans, Repos, Variables, Stages