Day 017 - 100DaysAWSIaCDevopsChallenge

Today in my series of 100 days of code challenge, I will show you how to decouple microservices developed with springboot using Amazon SQS.

What is Amazon SQS ?

Amazon SQS (Simple Queue Service) is a cloud service that helps applications communicate by sendyg, storing and receiving messages in a queue. It's like a waiting line where are stored until a consumer is ready to process them. This prevents systems from getting overhelmed and make sure no message is lost.

Consume a SQS Messages using Springboot application

To demonstrate how to consume SQS messages by creating a Spring Boot app that processes each message from an SQS queue. The infrastructure, built using CDK (Java), will include:

• a VPC with a public Subnet to host an EC2 instance where the Spring Boot app will run.
• An Internet Gateway for the EC2 instance to access the internet and download dependencies.
• A SQS Queue and Dead Letter Queue for message storage.
• An EC2 Instance for hosting SpringBoot App.
• An IAM Role to allow the EC2 instance to retrieve messages from the SQS queue (very important).

Create the Infrastructure

Set up the necessary infrastructure using CDK (Java)

VPC & Subne Internet Gateway

public class NetworkContruct extends Construct {
  private final IVpc vpc;
  public NetworkContruct(Construct scope, String id, StackProps props) {
    super(scope, id);
    this.vpc =
        new Vpc(
            this,
            "VpcResource",
            VpcProps.builder()
                .vpcName("my-vpc")
                .enableDnsHostnames(true)
                .enableDnsSupport(true)
                .createInternetGateway(true)
                .ipProtocol(IpProtocol.IPV4_ONLY)
                .ipAddresses(IpAddresses.cidr("10.0.0.1/16"))
                .maxAzs(1)
                .subnetConfiguration(
                    List.of(
                        SubnetConfiguration.builder()
                            .name("Public-Subnet")
                            .mapPublicIpOnLaunch(true)
                            .subnetType(SubnetType.PUBLIC)
                            .build()))
                .build());
  }
  public IVpc getVpc() {
    return vpc;
  }
}

This cdk construct will create:
?? A VPC named my-vpc and enable DNS hostname enabled.
?? A public subnet named Public-Subnet which allows resources to attach a public IP (if configured with one).
?? An Internet Gateway to enable internet traffic.

SQS Queue & Dead Letter Queue

public class QueueConstruct extends Construct {
  private final IQueue queue;
  public QueueConstruct(Construct scope, String id, IVpc vpc, StackProps props) {
    super(scope, id);
    IQueue dlq =
        new Queue(
            this,
            "DeadLetterQueue",
            QueueProps.builder()
                .deliveryDelay(Duration.millis(0))
                .retentionPeriod(Duration.days(10))
                .queueName("my-queue-dlq")
                .build());
    DeadLetterQueue deadLetterQueue = DeadLetterQueue.builder()
        .queue(dlq)
        .maxReceiveCount(32)
        .build();

    this.queue =
        new Queue(
            this,
            "SQSQueueResource",
            QueueProps.builder()
                .queueName("my-queue")
                .retentionPeriod(Duration.minutes(15))
                .visibilityTimeout(Duration.seconds(90))
                .deadLetterQueue(deadLetterQueue)
                .build());
  }

  public IQueue getQueue() {
    return queue;
  }
}

The above CDK construct will create the following resources:

• A Queue named my-queue, which will be used in the Spring Boot app.
• A DeadLetter Queue named my-queue-dlq which captures failed messages so they can be analysed and fixed later, without blocking the main queue.

EC2 Instance for Hosting the Spring Boot Application

// ComputerProps.java
public record ComputerProps(IVpc vpc, String sqsQueueArn) {}

// ComputerConstruct.java
public class ComputerConstruct extends Construct {
  private final IInstance computer;
  public ComputerConstruct(
      Construct scope, String id, ComputerProps computerProps, StackProps props) {
    super(scope, id);
    SecurityGroup securityGroup =
        new SecurityGroup(
            this,
            "WebserverSGResource",
            SecurityGroupProps.builder()
                .allowAllOutbound(true)
                .securityGroupName("Webserver-security-group")
                .disableInlineRules(true)
                .vpc(computerProps.vpc())
                .description("Allow trafic from/to webserver instance")
                .build());

    securityGroup.addIngressRule(Peer.anyIpv4(), Port.SSH, "Allow ssh traffic");
    securityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(8089), "Allow traffic from 8089 port");

    KeyPair keyPair =
        new KeyPair(
            this,
            "KeyPairResource",
            KeyPairProps.builder()
                .keyPairName("ws-keypair")
                .account(Objects.requireNonNull(props.getEnv()).getAccount())
                .type(KeyPairType.RSA)
                .format(KeyPairFormat.PEM)
                .build());

    new CfnOutput(
        this, "KeyPairId", CfnOutputProps.builder().value(keyPair.getKeyPairId()).build());

    Instance ec2Instance =
        new Instance(
            this,
            "WebServerInstanceResource",
            InstanceProps.builder()
                .securityGroup(securityGroup)
                .keyPair(keyPair)
                .instanceName("Webserver-Instance")
                .machineImage(
                    MachineImage.lookup(
                        LookupMachineImageProps.builder()
                            .name("*ubuntu*")
                            .filters(
                                Map.ofEntries(
                                    Map.entry("image-id", List.of("ami-0e86e20dae9224db8")),
                                    Map.entry("architecture", List.of("x86_64"))))
                            .windows(false)
                            .build()))
                .vpc(computerProps.vpc())
                .role(buildInstanceRole(computerProps))
                .instanceType(InstanceType.of(InstanceClass.T2, InstanceSize.MICRO))
                .associatePublicIpAddress(true)
                .blockDevices(
                    List.of(
                        BlockDevice.builder()
                            .mappingEnabled(true)
                            .deviceName("/dev/sda1")
                            .volume(
                                BlockDeviceVolume.ebs(
                                    10,
                                    EbsDeviceOptions.builder()
                                        .deleteOnTermination(true)
                                        .volumeType(EbsDeviceVolumeType.GP3)
                                        .build()))
                            .build()))
                .userDataCausesReplacement(true)
                .vpcSubnets(SubnetSelection.builder().subnetType(SubnetType.PUBLIC).build())
                .build());

    UserData userData = UserData.forLinux();
    userData.addCommands(readFile("./webserver-startup.sh"));

    ec2Instance.addUserData(userData.render());

    this.computer = ec2Instance;
  }

  public IInstance getComputer() {
    return computer;
  }

  private String readFile(String filename) {

    InputStream scriptFileStream = getClass().getClassLoader().getResourceAsStream(filename);

    try {
      assert scriptFileStream != null;
      try (InputStreamReader isr = new InputStreamReader(scriptFileStream, StandardCharsets.UTF_8);
          BufferedReader br = new BufferedReader(isr)) {
        StringBuilder content = new StringBuilder();
        String line;
        while ((line = br.readLine()) != null) {
          content.append(line).append("\n");
        }
        return content.toString();
      }
    } catch (IOException e) {
      throw new RuntimeException(e.getMessage());
    }
  }

  private IRole buildInstanceRole(ComputerProps props) {
    return new Role(
        this,
        "WebserverInstanceRoleResource",
        RoleProps.builder()
            .roleName("webserver-role")
            .assumedBy(new ServicePrincipal("ec2.amazonaws.com"))
            .path("/")
            .inlinePolicies(
                Map.ofEntries(
                    Map.entry(
                        "sqs",
                        new PolicyDocument(
                            PolicyDocumentProps.builder()
                                .assignSids(true)
                                .statements(
                                    List.of(
                                        new PolicyStatement(
                                            PolicyStatementProps.builder()
                                                .effect(Effect.ALLOW)
                                                .actions(
                                                    List.of(
                                                        "sqs:DeleteMessage",
                                                        "sqs:ReceiveMessage",
                                                        "sqs:SendMessage",
                                                        "sqs:GetQueueAttributes",
                                                        "sqs:GetQueueUrl"))
                                                .resources(List.of(props.sqsQueueArn()))
                                                .build())))
                                .build()))))
            .build());
  }
}

The above CDK construct will create the following resources:

• A Security Group named Webserver-security-group that allow inbound traffic on Port 22 for SSH connections and allow inbound traffic on Port 8089, which is the application connection port.
• A Key Pair named ws-keypair that will be used to connect to the app host via SSH. Since we are using CDK to build the infrastructure, if you need to download the private key (PEM file) after deployment, refer to my previous article on How the retrieve the private key file PEM after Cloudformation or CDK stack creation[↗].
• An Ec2 Instance named Webserver-Instance.
• An IAM Role for the Ec2 Instance named webserver-role, which allows the spring Boot application hosted on the Ec2 Instance to establish connections with the Amazon SQS Queue (already created) and perform actions: Delete Message, Receive Message, Send Message, Get Queue Attributes and Get Queue Url.

Create the stack

// MyStack.java
public class MyStack extends Stack {
  public MyStack(final Construct scope, final String id, final StackProps props) {
    super(scope, id, props);
    IVpc vpc = new NetworkContruct(this, "NetworkResource", props).getVpc();
    IQueue queue = new QueueConstruct(this, "QueueResource", vpc, props).getQueue();
    IInstance webserver =
        new ComputerConstruct(
                this, "ComputerResource", new ComputerProps(vpc, queue.getQueueArn()), props)
            .getComputer();
  }
}

// Day17App.java
public class Day017App {
  public static void main(final String[] args) {
    App app = new App();
    new MyStack(app,"Day017Stack",
        StackProps.builder()
                .env(
                    Environment.builder()
                        .account(System.getenv("CDK_DEFAULT_ACCOUNT"))
                        .region(System.getenv("CDK_DEFAULT_REGION"))
                        .build())
                .build());
    app.synth();
  }
}

Create SpringBoot Consumer Application

To keep things simple and avoid complicating my life, I will use Spring Cloud AWS Docs[↗]

Spring Cloud AWS simplifies using AWS managed services in a Spring Framework and Spring Boot applications. It offers a convenient way to interact with AWS provided services using well-known Spring idioms and APIs.

To configure the SQS service in my project, I will add the following beans in the configuration class:

@Configuration
public class ApplicationConfiguration {
    @Bean
    public AwsRegionProvider customRegionProvider() {
        return new InstanceProfileRegionProvider();
    }
    @Bean
    public AwsCredentialsProvider customInstanceCredProvider() {
        return  InstanceProfileCredentialsProvider.builder()
                .build();
    }
}

And the listener that captures all new message and prints their content.

@Slf4j
@Component
public class ExampleSQSConsumer {
    @SqsListener(queueNames = { "my-queue" }) // ??
    public void listen(String payload) {
        log.info("*******************  SQS Payload ***************");
        log.info("Message Content: {}", payload);
        log.info("Received At: {}", Date.from(Instant.now()));
        log.info("************************************************");
    }
}

You can find the full project in my GitHub repo[↗]

Deployment

⚠️⚠️ Before run the deployment commands ensure that you have java installed on your host machine. I used Java 21 under MacOs to build this insfrastructure.

Open the terminal anywhere and run the following commands:

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git

cd 100DaysTerraformAWSDevops/day_017

cdk bootstrap --profile cdk-user

cdk deploy --profile cdk-user Day017Stack

Resut

Your can find the full source code on GitHub Repo↗