Python Forum
visualizing huge correation matrix
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
visualizing huge correation matrix
#1
I have a huge correlation matrix with the dimension of (654345,2,2). It was generated by Pearson's coefficients, i.e. numpy.corrcoef. What is the best way
of visualizing it? Below is a small part of the entire code;

    def __init__(self):
        self.LayerIdx = ['CA3', 'DG', 'EC'] # propagation order

    def get_correlation(self, Layers, LayersPos):

        corr_xy = []
        for layers_count in range(self.LayerNo - 1):
            x1 = Layers[layers_count]
            x2 = Layers[layers_count + 1]
            y1 = LayersPos[layers_count]
            y2 = LayersPos[layers_count + 1]
            for ind1 in range(len(y1)):
                for ind2 in range(len(y2)):
                    corr_xy.append(np.corrcoef(x1[y1[ind1]], x2[y2[ind2]]))

            x_labels = [v for v in self.LayerIdx]
            y_labels = [v for v in self.LayerIdx]
            x_to_num = {p[1]: p[0] for p in enumerate(x_labels)}
            y_to_num = {p[1]: p[0] for p in enumerate(y_labels)}
            fig, ax = plt.subplots(figsize=(10, 6))
            fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
            for i in range(np.shape(corr_xy)[0]): #corr_xy is the collection of Pearsons coefficients
                sns.heatmap(corr_xy[i], annot=True, fmt='.2f')
                ax.grid(False)
                ax.set_xticks([x_to_num[v] for v in x_labels])  # [x_to_num[v] for v in x_labels]
                ax.set_yticks([y_to_num[v] for v in y_labels])
                cbar = ax.figure.colorbar(im, ax=ax, format='% .2f')
            plt.show()
Reply
#2
There are quite a few blogs on this, Google: plot numpy.corrcoef (Pearson's coefficients) correlation matrix python for list
Reply
#3
I have been googling for the last 9.5 hours. I think the problem is about how I generate the matrix. To be pragmatic, I do list by using ".append".

(Oct-12-2021, 04:22 PM)Larz60+ Wrote: There are quite a few blogs on this, Google: plot numpy.corrcoef (Pearson's coefficients) correlation matrix python for list
Reply
#4
Besides, many of those examples are for extremely cases.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [solved] how to speed-up huge data in an ascii file ? paul18fr 4 1,257 May-16-2023, 08:36 PM
Last Post: paul18fr
  Finding the median of a column in a huge CSV file markagregory 5 1,781 Jan-24-2023, 04:22 PM
Last Post: DeaD_EyE
  huge and weird values after applying some calculations karlito 2 2,156 Dec-13-2019, 08:32 AM
Last Post: karlito
  Loading HUGE data from Python into SQL SERVER Sandeep 2 21,085 Jan-13-2018, 07:52 AM
Last Post: Sandeep

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020