Ttykuu

I'd like to calculate some statistics of given data (frequencies of difference among elements on various distances in percent multiplied by 10) using moving window within that data. Is it possible to speed up the code below? I noticed that some calculations are repeating. But I was not able to exclude them without additional slowness.

def get_dist_stat(pdata, pwin_length):
 ''' pdata - given data array
 pwin_length - the lenght of window
 the function returns stat table where
 row represents the distance between elements
 col represents the difference for that distance in percent multiplied by 10 (assume that maximum difference can be 20 percent)

 '''

 l_data = len(pdata)
 l_win = pwin_length
 print("l_data=", l_data)
 print("l_win=", l_win)

 # stat table
 stat_table = np.zeros((l_win-1, 20*10), dtype = int)

 # loop over all data
 for k in range(l_data - l_win + 1):

 win = pdata[k : k + l_win]
 print('-' * 10)
 print("k=", k, " kend=", k + l_win )

 print("win=", win)

 # loop over window
 for i in range(1 , l_win):
 b=win[i:]
 a=win[:-i]
 diff=(abs((b-a)/a*100 ) * 10).astype(int)
 print("i=",i)
 print("b=", b)
 print("a=", a)
 print("diff=",diff)

 # storing found differences into stat table
 apercents, acount = np.unique(diff, return_counts = True)
 l_apercents = len(apercents)
 for j in range(l_apercents):
 stat_table[i-1, apercents[j]] += acount[j]
 return stat_table

adata=np.array([1.1,1.2,1.3,1.4,1.5])
print("adata=", adata)

astat_table=get_dist_stat(adata,3)
print(astat_table)

And that is its output

adata= [1.1 1.2 1.3 1.4 1.5]
l_data= 5
l_win= 3
----------
k= 0 kend= 3
win= [1.1 1.2 1.3]
i= 1
b= [1.2 1.3]
a= [1.1 1.2]
diff= [90 83]
i= 2
b= [1.3]
a= [1.1]
diff= [181]
----------
k= 1 kend= 4
win= [1.2 1.3 1.4]
i= 1
b= [1.3 1.4]
a= [1.2 1.3]
diff= [83 76]
i= 2
b= [1.4]
a= [1.2]
diff= [166]
----------
k= 2 kend= 5
win= [1.3 1.4 1.5]
i= 1
b= [1.4 1.5]
a= [1.3 1.4]
diff= [76 71]
i= 2
b= [1.5]
a= [1.3]
diff= [153]
[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
 0 0 0 0 2 0 0 0 0 0 0 2 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]

You've already made the key observation here, which is that most of the work is redone. Each time you pick a window, most of the calculations are the same as the previous window.

In fact it's much faster to do all the calculations ahead of time into one big ndarray, and then for each window, pick out the calculations that are relevant. So we don't need the temporary a and b lists.

How many dimensions do we need? Just starting point and length. It's going to be a triangular array, so we'll waste some space.

precomputed_results = np.zeros(l_win+1, l_data), dtype = int)
# First pass
for interval in range(1, l_win):
 for first_point_index in range(l_data-interval): 
 # compute diff relative to elements [first_point_index] and [first_point_index+interval]
 # line will be similar to precomputed_results[...] = ...

# Second pass
for interval in range(1, l_win):
 for first_point_index in range(l_data-interval):
 # use slicing on precomputed_results