Ttykuu

I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and can extract the table of information I need.

The HTML for the table looks like this (yes, it has the commented out lines)

The I have below works, but it feels inefficient. I feel like I have a lot of loops and these feel unnecessary. I'm not sure how to be more efficient though.

The only change between this and my real code is that I've moved the HTML to pastebin to cut down on the length of this post. That pastebin is here and the HTML is not written or editable by me. It's generated on the cable modem. Since you aren't able to see the results of my cable modem by running the code I use to extract it from the modem, I hope this is good enough.

from bs4 import BeautifulSoup
import requests

results_url = "https://pastebin.com/raw/bLZLFzy6"
content = requests.get(results_url).text

measurement = "modem"
hostname = "home-modem"


def strip_uom(data):
 """Strip Unit of Measurement

 Some of our fields come with unit of measure. Rip that out and keep only 
 the value of the field.
 """
 uom = ["dB", "Hz", "dBmV", "Ksym/sec"]
 dataset = []
 for d in data:
 if d.split(" ")[-1] in uom:
 dataset.append(d.split(" ")[0])
 else:
 dataset.append(d)
 return dataset


soup = BeautifulSoup(content, 'html.parser')
dstable = soup.find('table', 'id': 'dsTable')

# Pull the headers from the downstream table 
# We want to make these tag friendly, so make them lowercase and remove spaces
dstable_tags = [td.get_text().lower().replace(" ", "_") for td in dstable.tr.find_all('td')]

# Pull out the rest of the data for each row in the table; associate it with the correct tag; Strip UoM 
downstream_data = []
for row in dstable.find_all('tr')[1:]:
 column_values = [col.get_text() for col in row.find_all('td')]
 downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))

# Print line protocol lines for telegraf's inputs.exec plugin to handle
for data in downstream_data:
 line_protocol_line = f"measurement,hostname=hostname"
 fields = ",".join(["=".join(_) for _ in data.items()])
 line_protocol_line = line_protocol_line + f" channel=data['channel'] fields"
 print(line_protocol_line)

Finally, the output this script generates is this:

modem,hostname=home-modem channel=1 channel=1,lock_status=Locked,modulation=QAM 256,channel_id=121,frequency=585000000,power=5.6,snr=37.0,correctables=19430,uncorrectables=11263
modem,hostname=home-modem channel=2 channel=2,lock_status=Locked,modulation=QAM 256,channel_id=6,frequency=591000000,power=5.7,snr=37.0,correctables=19520,uncorrectables=9512
modem,hostname=home-modem channel=3 channel=3,lock_status=Locked,modulation=QAM 256,channel_id=7,frequency=597000000,power=5.7,snr=36.8,correctables=17439,uncorrectables=9736
modem,hostname=home-modem channel=4 channel=4,lock_status=Locked,modulation=QAM 256,channel_id=8,frequency=603000000,power=5.9,snr=37.0,correctables=12743,uncorrectables=11156
modem,hostname=home-modem channel=5 channel=5,lock_status=Locked,modulation=QAM 256,channel_id=122,frequency=609000000,power=2.6,snr=36.6,correctables=1852963,uncorrectables=18388
modem,hostname=home-modem channel=6 channel=6,lock_status=Locked,modulation=QAM 256,channel_id=10,frequency=615000000,power=2.6,snr=37.0,correctables=844991,uncorrectables=14615
modem,hostname=home-modem channel=7 channel=7,lock_status=Locked,modulation=QAM 256,channel_id=11,frequency=621000000,power=2.5,snr=37.6,correctables=281024,uncorrectables=13998
modem,hostname=home-modem channel=8 channel=8,lock_status=Locked,modulation=QAM 256,channel_id=12,frequency=627000000,power=2.5,snr=35.9,correctables=77942,uncorrectables=13695

I have loops and list comprehensions to get the tags, associate data with tags for each row, and to generate the line protocol lines. I have two additional comprehensions in those and a loop to see if I need to strip out an included unit of measure. Can I eliminate some of these to be more efficient?

I am running all of this code on Python 3.6.7.