At Consegna, we like AWS and their services which are covered by a solid bench of documentation, blog posts and best practices. Because it is easy to find open source production ready code on GitHub, it is straightforward to deploy new applications quickly and at scale. However, sometimes, moving too fast may lead to some painful problems over time!
Deploying the AWS Serverless Developer Portal from Github straight to production works perfectly fine. Nevertheless, hardcoded values within the templates make complicated to deploy multiple similar environments within the same AWS account. Introducing some parameterization is usually the way to go to solve that problem. But that leads to deal with a production stack to not be aligned with the staging environments which is, of course, not a best practice…
This blog post describes the solution we have implemented to solve the challenge of migrating Cognito users from one pool to another at scale. The extra step of migrating API keys associated to those users is covered in this blog.
The Technology Stack
The deployed stack involves AWS serverless technologies such as Amazon API Gateway, AWS Lambda, and Amazon Cognito. It is assumed in this blog post that you are familiar with those AWS services but we encourage you to check out the AWS documentation or to contact Consegna for more details.
The Challenge
The main challenge is to migrate Cognito users and their API keys at scale without any downtime or requiring any password resets from the end users.
The official AWS documentation describes two ways of migrating users from one user pool to another:
1. Migrate users when they sign-in using Amazon Cognito for the first time with a user migration Lambda trigger. With this approach, users can continue using their existing passwords and will not have to reset them after the migration to your user pool.
2. Migrate users in bulk by uploading a CSV file containing the user profile attributes for all users. With this approach, users will require to reset their passwords.
We discarded the second option as we did not want our users to “pay” for this backend migration. So we used the following AWS blog article as a starting point while keeping in mind that it does the cover the entire migration we need to implement. Indeed, by default, an API key is created for every user registering on the portal. The key is stored in API Gateway and is named based on the user’s CognitoIdenityId attribute which is specific to each user within a particular Cognito user pool.
The Solution
The Migration Flow
The following picture represents our migration flow with the extra API key migration step.
Migration Flow
Notes
- The version of our application currently deployed in production does not support the Forget my password flow so we did not implement it in our migration flow (but we should and will).
- When a user registers, they must submit a verification code to have access to his API key. In the very unlikely situation where a user has registered against the current production environment without confirming their email address, the user will be migrated automatically with automatic confirmation of their email address by the migration microservice. Based on the number of users and the low probability of this particular scenario, we considered it as an acceptable risk. However it might be different for your application.
The Prerequisites
In order to successfully implement the migration microservice, you first need to grant some IAM permissions and to modify the Cognito user pool configuration.
- You must grant your migration Lambda function the following permissions (feel free to restrict those permissions to specific Cognito pools using
arn:${Partition}:cognito-idp:${Region}:${Account}:userpool/${UserPoolId}):
- Action:
- apigateway:GetApiKeys
- apigateway:UpdateApiKey
- cognito-identity:GetId
- cognito-idp:AdminInitiateAuth
- cognito-idp:AdminCreateUser
- cognito-idp:AdminGetUser
- cognito-idp:AdminRespondToAuthChallenge
- cognito-idp:ListUsers
Effect: Allow
Resource: "*"
- On both Cognito pools (the one you are migrating from and the one you are migrating to), enable Admin Authentication Flow (ADMIN_NO_SRP_AUTH) for allowing server-based authentication by the Lambda function executing the migration. You can do it via the Management Console or the AWS CLI with the following command:
aws cognito-idp update-user-pool-client \
--user-pool-id <value> \
--client-id <value> \
--explicit-auth-flows ADMIN_NO_SRP_AUTH
More details about the Admin Authentication Flow is available here.
You are all set. Let’s get our hands dirty!
The Implementation (in JS)
At the Application Layer
To allow a smooth migration for our users, the OnFailure of the login method should call our migration microservice instead of returning the original error back to the user. An unauthenticated API Gateway client is initialized to call the migrate_user method on our API Gateway. The result returned by the backend is straightforward: RETRY indicates a successful migration so the application must re login the user automatically else it must handle the authentication error (user does not exist, username or password incorrect and so on).
onFailure: (err) => {
// Save the original error to make sure to return appropriate error if required...
var original_err = err;
// Attempt migration only if old Cognito pool exists and if the original error is 'User does not exist.'
if (err.message === 'User does not exist.' && oldCognitoUserPoolId !== '') {
initApiGatewayClient() // Initialize an unauthenticated API Gateway client
var body = {
// Prepare the body for the request for all required information such as
// username, password, old and new Cognito pool information
}
// Let's migrate your user!
apiGatewayClient.post("/migrate_user", {}, body, {}).then((result) => {
resolve(result);
if (result.data.status === "RETRY") { // Successful migration!
// user can now login!
} else {
// Oh no, status is not RETRY...
// Check the error code and display appropriate error message to the user
}
}).catch((err) => {
// Handle err returned by migrate_user or return original error
});
} else {
// Reject original error
}
}
The Migration microservice
API Gateway is used in conjunction with Cognito to authenticate the caller but few methods such as our migrate_user must remain unauthenticated. So here the configuration of migrate_user POST method on our API Gateway:
/migrate_user:
post:
produces:
- application/json
responses: {}
x-amazon-apigateway-integration:
uri: arn:aws:apigateway:<AWS_REGION>:lambda:path/2015-03-31/functions/arn:aws:lambda:<AWS_REGION>:<ACCOUNT_ID>:function:${stageVariables.FunctionName}/invocations
httpMethod: POST
type: aws_proxy
options:
consumes:
- application/json
produces:
- application/json
responses:
200:
description: 200 response
schema:
$ref: "#/definitions/Empty"
headers:
Access-Control-Allow-Origin:
type: string
Access-Control-Allow-Methods:
type: string
Access-Control-Allow-Headers:
type: string
x-amazon-apigateway-integration:
responses:
default:
statusCode: 200
responseParameters:
method.response.header.Access-Control-Allow-Methods: "'DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT'"
method.response.header.Access-Control-Allow-Headers: "'Content-Type,Authorization,X-Amz-Date,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Origin: "'*'"
passthroughBehavior: when_no_match
requestTemplates:
application/json: "{\"statusCode\": 200}"
type: mock
The implementation of migrate_user is simply added to our express-server.js so no Lambda to manage so to speak. The function is available below and we are going to deep dive in details into each step:
app.post('/migrate_user', (req, res) => {
// 1 -- Extract paramters from the body
var username = req.body.username;
var password = req.body.password;
// etc ...
var oldCognitoIdentityId = null;
var cognitoIdentityId = null;
var answer = { "status": "NO_RETRY" };
const migrate_task = async () => {
// 2 -- Check if migration is required
let result = await isMigrationRequired(username, cognitoUserPoolId);
if (result === false) return "NO_RETRY";
// 3 -- Resolve the CognitoIdentityId of the user within the old pool
result = await getCognitoIdentityId(username, password, oldCognitoUserPoolId, oldCognitoIdentityPoolId, oldCognitoClientId, oldCognitoRegion);
if (result.error != null) {
// Analyse error and return appropriate error code
if (result.error.code === "PasswordResetRequiredException") return "NO_RETRY_PASSWORD_RESET_REQUIRED";
else return "NO_RETRY";
} else oldCognitoIdentityId = result.cognitoIdentityId;
// 4 -- Extract the user's attributes to migrate from the old to the new pool
var attributesToMigrate = await getUserAttributes(username, oldCognitoUserPoolId);
// 5 -- Migrate user from old to new pool
result = await migrateUser(username, password, cognitoUserPoolId, cognitoClientId, attributesToMigrate);
if (result.error !== null) {
// Something went wrong during the migration!
return "NO_RETRY";
}
// 6 -- Resolve the CognitoIdentityId of the user within the new pool
result = await getCognitoIdentityId(username, password, cognitoUserPoolId, cognitoIdentityPoolId, cognitoClientId, cognitoRegion);
if (result.error !== null) {
// Analyse error and return appropriate error code
if (result.error.code === "PasswordResetRequiredException") return "NO_RETRY_PASSWORD_RESET_REQUIRED";
else return "NO_RETRY";
} else cognitoIdentityId = result.cognitoIdentityId;
// 7 -- Migrate the user's API key
result = await migrateApiKey(username, cognitoIdentityId, oldCognitoIdentityId);
// 8 -- Migration complete!
return "RETRY";
}
migrate_task()
.then((value) => {
answer.status = value;
if (value === "RETRY") {
res.status(200).json(answer);
} else res.status(500).json(answer);
})
.catch((error) => {
answer.status = value;
res.status(500).json(answer);
})
});
1 – Extract parameters from the body
All the data required for the migration has been passed by the application to our function via req so we just extract it. Of course do not log the password else it will appear in clear in the execution logs of your Lambda.
Note: you might wish to inject the Cognito pool information directly to the Lambda via environment variables instead of passing via the body of the request.
2 – Check if migration is required
A migration is indicated as required only if the user does not already exist in the new pool. However be aware that this function does not verify the existence of the user in the old pool (the check is made during step 3.):
function isMigrationRequired(username, cognitoUserPoolId) {
return new Promise((resolve, reject) => {
var params = {
Username: username,
UserPoolId: cognitoUserPoolId
};
cognitoidentityserviceprovider.adminGetUser(params, function(lookup_err, data) {
if (lookup_err) {
if (lookup_err.code === "UserNotFoundException") {
// User not found so migration should be attempted!
resolve(true);
} else {
reject(lookup_err) // reject any other error
}
} else {
resolve(false); // User does exist in the pool so no migration required
}
});
})
};
3 – Resolve the CognitoIdentityId of the user within the old pool
Authenticate the user against the old pool using adminInitiateAuth and get his CognitoIdentityId via the getId method. This is required for the migration of the user’s API key. Of course, if the user cannot be authenticated against the old pool, they cannot be migrated so the function returns the error straight away.
function getCognitoIdentityId(username, password, cognitoUserPoolId, cognitoIdentityPoolId, cognitoClientId, cognitoRegion) {
var params = {
AuthFlow: 'ADMIN_NO_SRP_AUTH',
ClientId: cognitoClientId,
UserPoolId: cognitoUserPoolId,
AuthParameters: {
USERNAME: username,
PASSWORD: password
}
};
var result = {
"cognitoIdentityId": null,
"error": null
}
return new Promise((resolve, reject) => {
cognitoidentityserviceprovider.adminInitiateAuth(params, function(initiate_auth_err, data) {
if (initiate_auth_err) {
// Error during authentication of the user against the old pool so this user cannot be migrated!
result.error = initiate_auth_err;
resolve(result);
} else {
// User exists in the old pool so let's get his CognitoIdentityId
var Logins = {};
Logins["cognito-idp." + cognitoRegion + ".amazonaws.com/" + cognitoUserPoolId] = data.AuthenticationResult.IdToken;
params = {
IdentityPoolId: cognitoIdentityPoolId,
Logins: Logins
};
cognitoidentity.getId(params, function(get_id_err, data) {
result.cognitoIdentityId = data.IdentityId;
resolve(result);
});
}
});
});
}
4 – Extract the attribute of user to migrate from the old to the new pool
Resolve the user’s attributes to migrate and force email_verified to true to avoid post-migration issues.
Note: all the attributes must be migrated except sub because this attribute is Cognito pool specific and will be created by the new pool.
function getUserAttributes(username, cognitoUserPoolId) {
var user = null;
var params = {
UserPoolId: cognitoUserPoolId,
Filter: "username = \"" + username + "\""
};
var result = [];
return new Promise((resolve, reject) => {
cognitoidentityserviceprovider.listUsers(params, function(list_err, data) {
if (list_err) console.log("Error while listing users using " + params + ": " + list_err.stack);
else {
data.Users[0].Attributes.map(function(attribute) {
if (attribute.Name === 'email_verified') {
attribute.Value = 'true';
}
if (attribute.Name !== 'sub') result.push(attribute);
});
}
resolve(result);
});
});
}
5 – Migrate user from old to new pool
Our user is now ready to be migrated! So let’s use the admin features of Cognito(adminCreateUser, adminInitiateAuth, and adminRespondToAuthChallenge) to create the user, authenticate the user, and set their password.
function migrateUser(username, password, cognitoUserPoolId, cognitoClientId, attributesToMigrate) {
var params = {
UserPoolId: cognitoUserPoolId,
Username: username,
MessageAction: 'SUPPRESS', //suppress the sending of an invitation to the user
TemporaryPassword: password,
UserAttributes: attributesToMigrate
};
var result = {
"error": null
}
return new Promise((resolve, reject) => {
cognitoidentityserviceprovider.adminCreateUser(params, function(create_err, data) {
if (create_err) {
result.error = create_err;
resolve(result);
} else {
// Now sign in the migrated user to set the permanent password and confirm the user
params = {
AuthFlow: 'ADMIN_NO_SRP_AUTH',
ClientId: cognitoClientId,
UserPoolId: cognitoUserPoolId,
AuthParameters: {
USERNAME: username,
PASSWORD: password
}
};
cognitoidentityserviceprovider.adminInitiateAuth(params, function(initiate_auth_err, data) {
if (initiate_auth_err) {
result.error = initiate_auth_err;
resolve(result);
} else {
// Handle the response to set the password (confirm the challenge name is NEW_PASSWORD_REQUIRED)
if (data.ChallengeName !== "NEW_PASSWORD_REQUIRED") {
result.error = new Error("Unexpected challenge name after adminInitiateAuth [" + data.ChallengeName + "], migrating user created, but password not set")
resolve(result)
}
params = {
ChallengeName: "NEW_PASSWORD_REQUIRED",
ClientId: cognitoClientId,
UserPoolId: cognitoUserPoolId,
ChallengeResponses: {
"NEW_PASSWORD": password,
"USERNAME": data.ChallengeParameters.USER_ID_FOR_SRP
},
Session: data.Session
};
cognitoidentityserviceprovider.adminRespondToAuthChallenge(params, function(respond_err, data) {
if (respond_err) {
result.error = respond_err;
}
resolve(result)
});
}
});
}
});
});
}
6 – Resolve the CognitoIdentityId of the user within the new pool
Our user is now created within the new pool so let’s resolve his CognitoIdentityId required for migrating his API key.
7 – Migrate user’s API key
Migrate the user’s API key by renaming it to point to the user’s CognitoIdentityId resolved during step 6.
function migrateApiKey(username, cognitoIdentityId, oldCognitoIdentityId) {
var params = {
nameQuery: oldCognitoIdentityId
};
return new Promise((resolve, reject) => {
apigateway.getApiKeys(params, function(get_key_err, data) {
params = {
apiKey: apiKeyId,
patchOperations: [{
op: "replace",
path: "/name",
value: cognitoIdentityId
}, {
op: "replace",
path: "/description",
value: "Dev Portal API Key for " + cognitoIdentityId
}]
};
// Update API key name and description to reflect the new CognitoIdentityId
apigateway.updateApiKey(params, function(update_err, data) {
console.log("API key (id: [" + apiKeyId + "]) updated successfully");
resolve(true)
});
});
})
}
8 – Migration complete, so return RETRY to indicate success
The migration is now complete so return RETRY status indicating to the application that the user must be re logged in automatically.
Conclusion
By leveraging AWS serverless technologies we have been able to fully handle the migration of our client’s application users at the backend level. The customer was happy with this solution as it avoided sending requests to the users to reset their password and it realigned the production with staging.
It’s implementing solutions like this that helps set Consegna apart from other cloud consultancies — we are a true technology partner and care deeply about getting outcomes for customers that align with their business goals, not just looking after our bottom line.
Consegna Blog 