DynamoDB Tables

A DynamoDB Table is a logical grouping of items, which represent the data stored in this Table. Given the NoSQL nature of DynamoDB, the Tables do not require that all items in a Table conform to some predefined schema.^[10]

DynamoDB Items

An Item in a DynamoDB is a set of attributes that can be uniquely identified in a Table. An Attribute is an atomic data entity that in itself is a Key-Value pair. The Key is always of String type, while the value can be of one of multiple data types.

An Item is uniquely identified in a Table using a subset of its attributes called Keys.^[10]

DynamoDB Data Types

DynamoDB supports numerical, String, Boolean, Document, and Set Data Types.^[11]

DynamoDB Indices

Primary Key of a Table is the Default or Primary Index of a DynamoDB Table.

In addition, a DynamoDB Table can have Secondary Indices. A Secondary Index is defined on an attribute that is different from Partition Key or Sort Key as the Primary Index.

When a Secondary Index has same Partition Key as Primary Index but a different Sort Key, it is called as the Local Secondary Index.

When Primary Index and Secondary Index have different Partition Key, the Secondary index is known as the Global Secondary Index.^[10]

Syntax

DynamoDB uses JSON for its syntax because of its ubiquity. The create table action demands just three arguments: TableName, KeySchema––a list containing a partition key and an optional sort key––and AttributeDefinitions––a list of attributes to be defined which must at least contain definitions for the attributes used as partition and sort keys. Whereas relational databases offer robust query languages, DynamoDB offers just Put, Get, Update, and Delete operations. Put requests contain the TableName attribute and an Item attribute, which consists of all the attributes and values the item has. An Update request follows the same syntax. Similarly, to get or delete an item, simply specify a TableName and Key.

Data structures

DynamoDB uses hashing and B-trees to manage data. Upon entry, data is first distributed into different partitions by hashing on the partition key. Each partition can store up to 10GB of data and handle by default 1,000 write capacity units (WCU) and 3,000 read capacity units (RCU).^[12] One RCU represents one strongly consistent read per second or two eventually consistent reads per second for items up to 4KB in size.^[13] One WCU represents one write per second for an item up to 1KB in size.

To prevent data loss, DynamoDB features a two-tier backup system of replication and long-term storage.^[14] Each partition features three nodes, each of which contains a copy of that partition's data. Each node also contains two data structures: a B tree used to locate items, and a replication log that notes all changes made to the node. DynamoDB periodically takes snapshots of these two data structures and stores them for a month in S3 so that engineers can perform point-in-time restores of their databases.

Within each partition, one of the three nodes is designated the "leader node". All write operations travel first through the leader node before propagating, which makes writes consistent in DynamoDB. To maintain its status, the leader sends a "heartbeat" to each other node every 1.5 seconds. Should another node stop receiving heartbeats, it can initiate a new leader election. DynamoDB uses the Paxos algorithm to elect leaders.

Amazon engineers originally avoided Dynamo due to engineering overheads like provisioning and managing partitions and nodes.^[9] In response, the DynamoDB team built a service it calls AutoAdmin to manage a database.^[14] AutoAdmin replaces a node when it stops responding by copying data from another node. When a partition exceeds any of its three thresholds (RCU, WCU, or 10GB), AutoAdmin will automatically add additional partitions to further segment the data.^[12]

Just like indexing systems in the relational model, DynamoDB demands that any updates to a table be reflected in each of the table's indices. DynamoDB handles this using a service it calls the "log propagator", which subscribes to the replication logs in each node and sends additional Put, Update, and Delete requests to indices as necessary.^[14] Because indices result in substantial performance hits for write requests, DynamoDB allows a user at most five of them on any given table.^[15]

Query execution

Suppose that a DynamoDB user issues a write operation (a Put, Update, or Delete). While a typical relational system would convert the SQL query to relational algebra and run optimization algorithms, DynamoDB skips both processes and gets right to work.^[14] The request arrives at the DynamoDB request router, which authenticates––"Is the request coming from where/whom it claims to be?"––and checks for authorization––"Does the user submitting the request have the requisite permissions?" Assuming these checks pass, the system hashes the request's partition key to arrive in the appropriate partition. There are three nodes within, each with a copy of the partition's data. The system first writes to the leader node, then writes to a second node, then sends a "success" message, and finally continues propagating to the third node. Writes are consistent because they always travel first through the leader node.

Finally, the log propagator propagates the change to all indices. For each index, it grabs that index's primary key value from the item, then performs the same write on that index without log propagation. If the operation is an Update to a preexisting item, the updated attribute may serve as a primary key for an index, and thus the B tree for that index must update as well. B trees only handle insert, delete, and read operations, so in practice, when the log propagator receives an Update operation, it issues both a Delete operation and a Put operation to all indices.

Now suppose that a DynamoDB user issues a Get operation. The request router proceeds as before with authentication and authorization. Next, as above, we hash our partition key to arrive in the appropriate hash. Now, we encounter a problem: with three nodes in eventual consistency with one another, how can we decide which to investigate? DynamoDB offers the user two options when issuing a read: consistent and eventually consistent. A consistent read visits the leader node. But the consistency-availability trade-off rears its head again here: in read-heavy systems, always reading from the leader can overwhelm a single node and reduce availability.

The second option, an eventually consistent read, selects a random node. In practice, this is where DynamoDB trades consistency for availability. If we take this route, what are the odds of an inconsistency? We'd need a write operation to return "success" and begin propagating to the third node, but not finish. We'd also need our Get to target this third node. This means a 1-in-3 chance of inconsistency within the write operation's propagation window. How long is this window? Any number of catastrophes could cause a node to fall behind, but in the vast majority of cases, the third node is up-to-date within milliseconds of the leader.

HTTP API

Against HTTP API, query items:

POST / HTTP/1.1
Host: dynamodb.<region>.<domain>;
Accept-Encoding: identity
Content-Length: <PayloadSizeBytes>
User-Agent: <UserAgentString>
Content-Type: application/x-amz-json-1.0
Authorization: AWS4-HMAC-SHA256 Credential=<Credential>, SignedHeaders=<Headers>, Signature=<Signature>
X-Amz-Date: <Date>
X-Amz-Target: DynamoDB_20120810.Query

{
    "TableName": "Reply",
    "IndexName": "PostedBy-Index",
    "Limit": 3,
    "ConsistentRead": true,
    "ProjectionExpression": "Id, PostedBy, ReplyDateTime",
    "KeyConditionExpression": "Id = :v1 AND PostedBy BETWEEN :v2a AND :v2b",
    "ExpressionAttributeValues": {
        ":v1": {"S": "Amazon DynamoDB#DynamoDB Thread 1"},
        ":v2a": {"S": "User A"},
        ":v2b": {"S": "User C"}
    },
    "ReturnConsumedCapacity": "TOTAL"
}

Sample response:

HTTP/1.1 200 OK
x-amzn-RequestId: <RequestId>
x-amz-crc32: <Checksum>
Content-Type: application/x-amz-json-1.0
Content-Length: <PayloadSizeBytes>
Date: <Date>
 {
    "ConsumedCapacity": {
        "CapacityUnits": 1,
        "TableName": "Reply"
    },
    "Count": 2,
    "Items": [
        {
            "ReplyDateTime": {"S": "2015-02-18T20:27:36.165Z"},
            "PostedBy": {"S": "User A"},
            "Id": {"S": "Amazon DynamoDB#DynamoDB Thread 1"}
        },
        {
            "ReplyDateTime": {"S": "2015-02-25T20:27:36.165Z"},
            "PostedBy": {"S": "User B"},
            "Id": {"S": "Amazon DynamoDB#DynamoDB Thread 1"}
        }
    ],
    "ScannedCount": 2
}

Go

GetItem in Go:

getItemInput := &dynamodb.GetItemInput{
	TableName: aws.String("happy-marketer"),
	Key: map[string]*dynamodb.AttributeValue{
		"pk": {
			S: aws.String("project"),
		},
		"sk": {
			S: aws.String(email + " " + name),
		},
	},
}
getItemOutput, err := dynamodbClient.GetItem(getItemInput)

DeleteItem in Go:

deleteItemInput := &dynamodb.DeleteItemInput{
	TableName: aws.String("happy-marketer"),
	Key: map[string]*dynamodb.AttributeValue{
		"pk": {
			S: aws.String("project"),
		},
		"sk": {
			S: aws.String(email + " " + name),
		},
	},
}

_, err := dynamodbClient.DeleteItem(deleteItemInput)
if err != nil {
	panic(err)
}

UpdateItem in Go using Expression Builder:

update := expression.Set(
	expression.Name(name),
	expression.Value(value),
)

expr, err := expression.NewBuilder().WithUpdate(update).Build()
if err != nil {
	panic(err)
}

updateItemInput := &dynamodb.UpdateItemInput{
	TableName: aws.String(tableName),
	Key: map[string]*dynamodb.AttributeValue{
		"pk": {
			S: aws.String("project"),
		},
		"sk": {
			S: aws.String("mySortKeyValue"),
		},
	},
	UpdateExpression:          expr.Update(),
	ExpressionAttributeNames:  expr.Names(),
	ExpressionAttributeValues: expr.Values(),
}
fmt.Printf("updateItemInput: %#v\n", updateItemInput)

_, err = dynamodbClient.UpdateItem(updateItemInput)
if err != nil {
	panic(err)
}