Install a Variable Store on S3 with AWS Lambda
S3 Storage Lambda
OverOps can use AWS Lambda for a serverless Variable Store on S3. A CloudFormation stack is used to facilitate the deployment of these services.
Prerequisites
git - in order to clone the repository with the CloudFormation template
docker - we use docker to encapsulate the dependencies for deploying to AWS
S3 bucket - an S3 bucket for the overops storage
IAM Credentials - AWS credentials, which we'll copy into the deploy folder. If you already have credentials in ~/.aws/credentials file you may use this file.
The credentials must have appropriate access to AWS services to read/write S3, API gateway, lambda and create/manage roles to apply to those as well as CloudFormation itself.
The required roles are as follows:
AWSLambdaFullAccess
IAMFullAccess
AmazonS3FullAccess
AmazonAPIGatewayAdministrator
AWSCloudFormationFullAccess
These may be applied in AWS IAM (Identity and Access Management):
https://console.aws.amazon.com/iam/home

Setup / Configuration
The first thing you will need to do is clone the takipi-storage repository from github and checkout the branch with the lambda implementation.
git clone https://github.com/takipi/takipi-storage
git checkout s3-storage-lambda
After completing the above, the AWS credentials need to be copied into takipi-storage/deploy/docker/credentials.
Enter the name of the S3 bucket to be used for storage in takipi-storage/deploy/docker/Dockerfile:
ENV S3_BUCKET=your-overops-storage
If you are not deploying to us-east-1, you will need to edit takipi-storage/deploy/docker/config:
region=us-east-1
Deploy
The deploy script (deploy.sh) will package and deploy the CloudFormation stack for the overops-storage S3 Lambda services.
To execute the script make sure you are in the deploy directory before running it from a terminal.
For example:
cd takipi-storage/deploy
./deploy.sh
When the script completes successfully, you should see output in the console with the storage URL, similar to what is seen below.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Key StorageApi
Description URL for application
Value https://we7lby9gxj.execute-api.us-east-1.amazonaws.com/Prod
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
To view and manage what is deployed directly on AWS, navigate to https://console.aws.amazon.com/cloudformation/home.

Before finishing the setup, verify the deployment is healthy by using the StorageApi URL from the output above and appending "/storage/v1/diag/ping" to it.
For example: https://we7lby9gxj.execute-api.us-east-1.amazonaws.com/Prod/storage/v1/diag/ping
The response should give an "ok" indicating that the services are deployed and operational.
Using the Storage Server
Use the StorageApi URL (eg https://we7lby9gxj.execute-api.us-east-1.amazonaws.com/Prod) for the "Host name" in the OverOps Management Console.

Once entered, test the connection and then save it. Your OverOps installation is now configured to use AWS Lambda for handling all storage requests with S3.
Updated over 4 years ago