WSO2 API Manager Sync Synapse Artifacts using S3 bucket

If you have used WSO2 API Manager you all might aware that when the API is developed by the publisher, In the gateway runtime it was defined by the synapse artifacts. Synapse configuration files contain artifacts such as API definitions, endpoints, and sequences. It is mandatory to share these configuration files through the gateways to maintain consistency among the nodes. WSO2 API-M provides three mechanisms for artifact synchronization with APIM 3.2.0 onward.

1. Shared file system (e.g., NFS)
2. Rsync
3. Inbuilt artifact synchronizer

If you already used WSO2 products, You might families with the 1st & 2nd option. But with the 3.2.0 release, Introduced a new option to sync artifacts between servers. The reason behind that to avoid the complications that occur with the Rsync as well as the maintenance overhead when using Shared file systems. Specially with the containerized deployments. Simply it is working as following which is based on events.

How the artifact synchronize mechanism works

https://apim.docs.wso2.com/en/latest/assets/img/learn/artifact-synchronizer-architecture.png

1. When API is being published or edited, Meta information is being stored in the persistence medium(Eg. DB, S3, Git …etc).
2. An event is published to the traffic manager saying a new event related to API operation.
3. Gateway will receive the notification over the Traffic Manager.
4. Upon a event relevant to a artifacts, It will check the persistence solution and update the artifacts accordingly by looking the metadata.

With the default pack, it shipping with a solution based on a database. Please refer to the official documentation for more information regarding that.

But thanks to the extensibility of the WSO2 products, We can come up with any sort of a custom artifacts synchronization implementation. In this, let us focus on S3 as the persistence solution. And following sample code and instructions will guide if anyone intended to use this mechanism for artifacts synchronization.

Pre-Requisites

• You should have a valid AWS account and a valid AWS IAM credentials to manage S3 services.
• You should have valid accessKey & secretKey created by aforementioned IAM user.
• You should download the AWS S3 SDK jar file (Please note for this particulate sample we have used SDK version 1.11.253).
• Network access from gateway and publisher nodes to AWS services.

Build and use the sample implementation

Please refer to the sample implementation, Which is a sample implementation achieving this goal. You can clone this repo and do the necessary modifications and build using following command.

mvn clean install

Upon a successful build, Created JAR file will be available in target directory.

Or else, Please go ahead and download the compiled jar file from here.

Setup API Manager to use the S3 Implementation

• Copy the downloaded AWS S3 SDK jar file to the <AM_HOME>/repository/components/lib directory.
• Copy the built S3 synchronize jar file which was created in previous step to the <AM_HOME>/repository/components/dropins directory.
• In the publisher, Please add the following section in the deployment.toml file.

• Add the following configurations in the deployment.toml files of the gateways.

Please note that S3Saver is the name of the artifacts server and S3Retriever is the name of the artifacts server and retriver. If you refer to the S3 Implementation this name is returned with public String getName() method.

Please note that you have to configure the traffic manager correctly is a MUST to use this feature. And also please note that the gateway_labels should be correctly configured in deployment.toml file of the publisher.

• Then Restart the APIM servers. In this case, For this sample we have taken the accessKey, secretKey & awsRegion as JVM parameters (We can add them to the wso2server.sh file as well).

./wso2server.sh start -DaccessKey=<AWS_Access_key> -DsecretKey=<AWS_Secret_key> -DawsRegion=<AWS_Region>

Once you successfully configured all, It will create artifacts in AWS S3 and those will sync accordingly.

Bucket is created with the name of API GW Environment

Metadata of the artifacts

Essential information of a metadata file

PS. When testing the solution, Encountering a OSGI wiring issues with a S3-SDK. Therefore to overcome that, I have tweaked the wso2 orbit repo and built that.

• Clone the orbit repo.
• Go to the awslambda component and open the pom.xml file.
• Add the S3 dependency to that.

<dependency>                                   <groupId>com.amazonaws</groupId>                                   <artifactId>aws-java-sdk-s3</artifactId>                                   <version>${com.amazonaws.version}</version>                               </dependency>

• Build the repo. And it will create a awslambda-1.11.253.wso2v1.jar file in the awslambda/1.11.253.wso2v1/target directory.
• Create a folder call patch1234 inside <AM_HOME>/repository/components directory and copy awslambda-1.11.253.wso2v1.jar to that patch1234 folder.

Please note that this is just a sample implementation and you can modify the code accordingly.

#! /bin/bash
echo 'Good Luck, Happy Coding'