how to make my product description fetching function generic?

[PrateekG]

Hi All,

I am fetching product description(with html tags) from a site using BeautifulSoup+Python3.6.

My code is as below-

def get_soup(url):
try:
    response = requests.get(url)
    if response.status_code == 200:
        html = response.content
        return BeautifulSoup(html, "html.parser")
except Exception as ex:
    print("error from " + url + ": " + str(ex))

def get_product_details(url):
try:
    desc_list = soup.select('p ~ ul')
    prod_details['description'] = ''.join([str(i) for i in desc_list])
    return prod_details
except Exception as ex:
    logger.warning('%s - %s', ex, url)

if __name__ == '__main__':
    print("product1 description:")
    get_product_details("http://www.aprisin.com.sg/p-748-littletikespoptunesguitar.html")
    print("product2 description:")
    get_product_details("http://www.aprisin.com.sg/p-1052-172083littletikesclassiccastle.html")

The problem with my above code is that it is not able to fetch description for some product urls.
Like in above code product2 description: is blank.Sample output-

product1 description:
<ul>
<li>Freestyle</li>
<li>Play along with 5 pre-set tunes: </li>
</ul><ul>
<li>Each string will play a note</li>
<li>Guitar has a whammy bar</li>
<li>2-in-1 volume control and power button </li>
<li>Simple and easy to use </li>
<li>Helps develop music appreciation </li>
<li>Requires 3 "AA" alkaline batteries (included)</li>
</ul>
product2 description:

So what changes I need to make here so that it may work for all types of product?