# Further Attention Utilization -- Efficience & Segmentation

[**Non-local**](https://sisyphus.gitbook.io/project/~/edit/drafts/-LNDRlf25_eHFI7-6Vkm/deep-learning-basics/classification/non-local-neural-networks) method exploits **Self-Attention (like covariance matrix)** to enhance and suppress targeted spatial information.

[**Squeeze and Excitation**](https://sisyphus.gitbook.io/project/~/edit/drafts/-LNDRlf25_eHFI7-6Vkm/deep-learning-basics/classification/squeeze-and-excitation-networks) method uses **fully connected layer** to estimate / calculate the channel-wise attention weights.

### CBAM: Convolutional Block Attention Module

![](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LNDRlrYI75RUPWtkZ0L%2F-LNDj0KgyhXsK-cj71hT%2FScreen%20Shot%202018-09-25%20at%2011.17.06%20AM.png?alt=media\&token=44ae07a5-1046-4205-be2e-b48cda93e611)

![](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LNDRlrYI75RUPWtkZ0L%2F-LNDivLGpOS2ZrOrps_X%2FScreen%20Shot%202018-09-25%20at%2011.17.17%20AM.png?alt=media\&token=ada91dc3-7972-4b68-b873-e7bd7641c4c9)

A simple yet effective attention module for convolutional neural networks. Given an intermediate feature map, CBAM **sequentially** infers attention maps along two separate dimensions, **channel** and **spatial**, then the attention maps are multiplied to the input feature map for adaptive feature refinement.

**Channel Attention** and **Spatial Attention** are processed **sequentially**, and **both are calculated via convolution layers** ( 1 x 1 for Channel Attention, 7 x 7 for Spatial Attention) not by covariance similarity or self-attention.

In the CBAM paper, author has done detailed ablation experiments to compare **average pooling** vs **max pooling** vs **average pooling + max pooling** in channel attention branch ( average + max > average > max ); and **channel pooling** vs **conv 1 x 1** **channel compression** in first stage of spatial attention branch (channel pooling > 1 x 1 conv); as well as **conv 3 x 3** vs **conv 7 x 7** in second stage of spatial attention branch ( conv 7 x 7 > conv 3 x 3). Also, the best way to integrate spatial and channel attention is also explored, either sequence of **channel + spatial**, **spatial + channel** or **channel & spatial in parallel** ( channel + spatial > spatial + channel > channel & spatial in parallel)

#### Channel Attention Computation Complexity

The FLOPs of Squeeze-Excitation channel attention structure are quite small, and  the only limitation is that the size of conv 1 x 1 is Cin x 1 x 1 x Cout, **which could not be neglected especially in the late stage of deep neural networks**. And for channel / spatial separated structures, like MobileNet, this could double the size of the whole model. So in **CBAM**, it **shares the FC layer for both max pooling branch and average pooling branch** so that the attention module parameters won't be doubled and it does work.&#x20;

#### Spatial Attention Computation Complexity

Self-Attention (feature map spatial covariance matrix), like **non-local neural network**, has to calculate \[NHW x C] dot product \[C x NHW] ( **FLOPS is NHW x Cin x NHW**, compared to **regular convolution NHW x K x K x Cin x Cout**, the non-local attention module could cost lots of  computations). Even non-local network has already done a channel bottleneck at the beginning for halfening the input channels ( e.g. 1024 -> 1 x 1 conv -> 512 ), the FLOP is still quite large. Different from non-local spatial attention to use self-attention (covariance) on all channels (e.g. 1024 channels), **CBAM here only utilize 2 channels** (average pooling and max spatial pooling) **and use 7 x 7 convolution** to calculate / estimate spatial attention, with NHW x 7 x 7 x 2 x 1 FLOPS. **When the feature map NHW x Cin > 7 x 7 x 2 x 1, the CBAM will always needs less computations than non-local spatial attention.**&#x20;

Considering the CBAM's efficiency in both channel attention and spatial attention, it could be migrated to light-weight networks like MobileNet and CBAM shows good results on .&#x20;

![](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LNDRlrYI75RUPWtkZ0L%2F-LNDj3IDFW9Dnct9JCtj%2FScreen%20Shot%202018-09-25%20at%2011.16.40%20AM.png?alt=media\&token=abf1dfe4-4392-4f6d-b6c9-38bc5b4341cf)

### Sparse Attention

{% embed url="https://arxiv.org/abs/1904.10509" %}

Stride Sparse Attention: A **locally dense, but long-range sparse sampling strategy** for Attention. To achieve this, these two sampling methods should be repeated alternatively.

![Dilated Sampling ](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LlZ5hL-m7KY4XaDnYQS%2F-LlZ6npgF-VX1wjkq5h_%2Fimage.png?alt=media\&token=c4e8b173-fa12-4858-b0f5-43525769d987)

![Local Sampling](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LlZ5hL-m7KY4XaDnYQS%2F-LlZ6vsWRVKDS6e018cw%2Fimage.png?alt=media\&token=25d72c4e-2c04-4f11-bb36-dab31f4bc225)

![Locally dense, long-range sparse Sampling](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LlZ5hL-m7KY4XaDnYQS%2F-LlZ72KzFTpxlRpfHSgK%2Fimage.png?alt=media\&token=7b742864-99be-4563-977d-a6ed41198d57)

Because we can only access the previous sequence, so the sampling would only be the lower triangular matrix:

![Left: original Attention Sampling; Right: Sparse Attention seeing  previous sequence](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LlZ5hL-m7KY4XaDnYQS%2F-LlZ7wLeoIPxu7CN6LIe%2Fimage.png?alt=media\&token=4877c9d5-6cea-479e-9a71-2ee7e8f10be6)

So that Sparse Attention can cover longer sampled sequence length, under the same computation limitation. OpenAI used this method to **extend Transformer's sequence sampling length 30 times** than the original way.&#x20;

### Dual Attention Network for Scene Segmentation

DANet is a segmentation network also based on two way attentions: position (spatial) attention and channel attention, but in parallel way.

![](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LNDRlrYI75RUPWtkZ0L%2F-LNDnFDcxDKgUEHI6UBb%2FScreen%20Shot%202018-09-25%20at%2011.36.29%20AM.png?alt=media\&token=d16d11fd-2a71-4e11-9a13-9df1e2462b84)

DANet doesn't use FC or Convolution to calculate the Attention Map, instead for either position or channel attention, it always uses the self-attention/covariance. The # of weights is definitely smaller but less FLOPs cannot be promised.  Compared to its baseline FCN model, DANet does help for boost the performance.&#x20;

The whole idea of DANet is straightforward, more interestingly,  the results of spatial attention experiment is quite illustrative.

![Spatial Attention Map for 2 points](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LNDRlrYI75RUPWtkZ0L%2F-LNDqK98QUY1Tbhu7WE7%2FScreen%20Shot%202018-09-25%20at%2011.49.00%20AM.png?alt=media\&token=b3ff068e-fe73-4557-82d7-142106045273)

![Channel Attention Map for two selected channels](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LNDRlrYI75RUPWtkZ0L%2F-LNDqHhLqtJsHhGpeIzW%2FScreen%20Shot%202018-09-25%20at%2011.49.42%20AM.png?alt=media\&token=c57c0490-bf2b-4b94-9b85-30bd2d425108)

