Colorectal cancer (CRC) is the third most common type of cancer worldwide. It can be prevented by screening the colon and detecting polyps which might become malign. Therefore, an accurate diagnosis of polyps in colonoscopy images is crucial for CRC prevention. The introduction of computational techniques, well known as Computed Aided Diagnosis, facilitates diffusion and improves early recognition of potentially cancerous tissues. In this work, we propose a novel hybrid deep learning architecture for polyp image segmentation named Polyp2Seg. The model adopts a transformer architecture as its encoder to extract multi-hierarchical features. Additionally, a novel Feature Aggregation Module (FAM) merges progressively the multilevel features from the encoder to better localise polyps by adding semantic information. Next, a Multi-Context Attention Module (MCAM) removes noise and other artifact...