How to integrate Amazon S3 bucket storage with MongoDB Realm
“In connected systems, power is defined by both profound concentration and by massive distribution.” — Joshua Cooper Ramo
The purpose of Cosync AssetLink is to streamline the integration of asset data, such as images, videos, and large files, into applications constructed atop MongoDB App Services using the Realm database.
MongoDB Realm stands out as the premier offline-first real-time database for collaborative software development on a global scale. It represents the sole genuinely scalable solution for synchronizing client-side data seamlessly across an extensive array of platforms, encompassing iOS, OS/X, Android, Windows, and Web, all of which operate atop a unified server-less backend infrastructure.
In the realm of collaborative software development, data synchronization is often just one facet of the broader challenge. In most cases, applications involve not only data but also assets that necessitate sharing among users. These assets encompass large, unchanging files like images, music, videos, and documents. Presently, MongoDB Realm imposes a 4MB upper limit on supported object size. Consequently, it doesn’t offer a viable solution for managing assets, even when they are encoded in base 64. This limitation implies that assets might need to be fragmented into multiple 4MB objects, necessitating additional client application code for proper management. Furthermore, given MongoDB Realm’s lack of video and audio streaming capabilities, employing a strategy based on asset chunking would likely prove impractical.
Given these limitations, many application developers opt to incorporate an Asset Storage service, such as Amazon S3, alongside MongoDB Realm to address their storage needs. However, this approach mandates the development of extra cross-platform code on the client side in the form of a storage integration service. Perhaps more significantly, it often necessitates the embedding of sensitive credentials for accessing the storage service directly within the client code, thereby introducing an additional security vulnerability.
At present, the MongoDB Realm development team has made an attempt to tackle this challenge through a deprecated third-party service known as AWS S3 Service Snippets. However, this solution demands that the application first uploads the asset data to the MongoDB Application server, which then forwards it to the Amazon S3 service. Unfortunately, the 4MB size limit on asset storage persists due to the constraints of base64 encoding! Moreover, the notion of having to upload an asset to a MongoDB Realm server before it reaches a bucket storage system (such as S3) seems somewhat inefficient, as there is no inherent reason to prevent direct uploads from the client device to S3.
From the perspective of an application developer, an asset necessitates an HTTPS URL (universal resource locator) for effective integration within a client program. This is essential because image caching services (such as NSUrl or Kingfisher) often rely on URLs to function properly. Moreover, concerning image data, it is frequently important to manage different-sized versions of the image (small, medium, large, and original) to cater to various performance scenarios. Therefore, any asset management approach should encompass support for multiple image sizes.
In the case of video and audio assets, it’s vital that the storage service can stream the data within an application. This functionality can only be achieved through URLs connected to a storage service that provides CDN-like capabilities, as seen with Amazon S3.
Regarding MongoDB Realm, efficient asset management involves the use of data structures to monitor assets within the AssetLink Service. This should encompass the ability to generate write URLs for uploading asset data, as well as support for image asset resizing and video previews. The Cosync AssetLink service effectively bundles all of these functionalities into a convenient package, seamlessly integrating them with any MongoDB Realm application. Additionally, it offers enhanced security compared to many other solutions as Amazon S3 credentials are stored within MongoDB service functions rather than native client code.
Currently, Cosync AssetLink is tailored to function seamlessly with Amazon S3 bucket storage. However, in the future, Cosync has plans to expand its support to include additional storage providers such as Microsoft, Dropbox, and Google.
The integration operates by leveraging a combination of MongoDB App Service server functions and a MongoDB Realm Cosync object, namely CosyncAsset. The server functions, alongside the necessary access keys for Amazon S3 bucket storage, are configured within the MongoDB App Service through the Cosync Portal.
To initiate the upload of an asset to Amazon S3, the client-side program generates a call to the MongoDB function CosyncInitAsset() object that specifies and file path, and content type, and an expiration of the new asset. This will inform the App Service of the client’s intent, which will contact Amazon S3 and return the write URLs for the client to upload the asset to.
The notable advantage of the Cosync AssetLink solution lies in the direct transfer of asset data from the client device to the Amazon S3 bucket. This process bypasses the need to route through a MongoDB server and avoids consuming MongoDB bandwidth. Once the upload is successfully completed, the Cosync AssetLink protocol allows the client to notify the MongoDB App Service that the upload is complete and that an asset can be created around the uploaded data. To do this, the client will call the CosyncCreateAsset() function. In response the server will create asset read URLs and create a CosyncAsset object that wraps around them.
This setup process is straightforward and notably efficient, with the bulk of the work being performed at the client device’s edge rather than on the server side.
Asset upload process
To trigger an asset upload, a client application commences the process by calling the CosyncInitAsset function with the MongoDB App Service. It’s important to note that the Cosync AssetLink module has been tailored to align with the MongoDB Flexible Sync model, as the older partition-based version has been deprecated.
To initiate an asset upload, the client application code is required to complete several key parameters: the filePath, the contentType, and the expirationHours.
async function CosyncInitAsset(
filePath, /* Input file path */
contentType, /* content MIME type for asset */
expirationHours /* Expiration in hours (0 means public) */
)The input filePath adheres to the following format:
filePath := <directory>/<filename>.<extension>Typically, the directory is assigned as either image, video, or audio to categorize the type of asset being uploaded. However, it’s important to note that developers have the flexibility to deviate from this convention. There is no strict constraint requiring it to be a single directory; it can also encompass a directory path, offering developers greater customization.
This function will return a JSON string. If successful, the return value will look as follows:
{
statusCode : 200,
contentId: <String>,
writeUrls : {
writeUrl : <String>,
writeUrlSmall : <String>,
writeUrlMedium : <String>,
writeUrlLarge : <String>,
writeUrlVideoPreview: <String>
}
}These write URLs are where the client will upload the asset data to. The contentId is used as a unique identifier to create the path on the Amazon S3 associated with the asset. The write URLs provided by the Cosync AssetLink service should always be used in conjunction with an HTTPS PUT command. It’s worth mentioning that, at present, the service does not support URLs for multipart HTTPS POST commands.
This path (which is computed by the App Service) specifies the location of the asset within the Amazon S3 bucket and adheres to the ensuing format:
path := [public]<userId><directory>/<filename>-<contentId>.<extension>In cases where the asset is designated as non-expiring, its path will be initialized with the ‘public/’ directory as a prefix. Furthermore, for a given input filePath, the backend will append a timestamp to the file name to safeguard against potential file conflicts. It’s worth noting that all paths are initially categorized based on the userId of the user initiating the asset upload. In simpler terms, all assets uploaded by John Smith, for instance, will be situated under a common root path associated with John Smith’s ‘userId/’ directory.
Assets can be categorized into two types: expiring and non-expiring (aka public). Expiring assets are characterized by an expirationHours property set to a positive value greater than zero, indicating the duration of their validity in hours. If expirationHours is set to zero, the asset is considered public and does not expire. In the case of non-expiring assets, their associated URL(s) remain perpetually valid and do not require periodic refreshment.
Expiring assets, on the other hand, offer enhanced security for customer asset data. For instance, an application may necessitate the creation of a URL containing sensitive information. To safeguard against unauthorized access, developers may opt to set an expiration period, such as 30 minutes, for the link. In the Amazon S3 storage system, non-expiring assets are stored in a public bucket, while expiring assets are housed in a private bucket. Consequently, a typical application will typically require two buckets to accommodate its asset storage needs.
Upon successful completion of the upload, the application client completes the asset creation process by calling the CreateCosyncAsset() function.
async function CosyncCreateAsset(
filePath /* Input file path */
contentId, /* contentId returned by CreateInitAsset() */
contentType, /* content MIME type for asset */
expirationHours, /* Expiration in hours (0 means public) */
size, /* size in bytes of asset */
duration, /* duration in seconds for video and audio assets */
color, /* HEX color for background of image */
xRes, /* x resolution in pixels of image asset */
yRes, /* y resolution in pixels of image asset */
caption, /* optional caption for asset */
extra /* extra information passed to CosyncAsset */
)This function will create a CosyncAsset object that wraps around the asset. This object will be readable by all, but only writable by the user who created the asset.
This function will return a JSON string. If successful, the return value will look as follows:
{
statusCode : 200,
asset: CosyncAsset
}The asset is the CosyncAsset object that was created in the form of a JSON codable. Note: It is the responsibility of the client’s code to write the CosyncAsset to MongoDB Realm, so that it can sync with the rest of the application data.
The contentType property is employed to specify the content type of the asset, with various allowable values detailed in the Content Types table below.
The extra property serves as a container for additional information that the uploading client application code may need to store throughout the lifecycle of the CosyncAsset object. This information can be platform-specific in nature.
The following diagram presents a visual of the asset uploading process:
The CoSync AssetLink services uses the MongoDB Realm third-party AWS Service to implement this functionality. It simply uses this service to compute the pre-signed URLs needed to upload and read asset information. The third-party service simply acts as a broker between MongoDB Realm and Amazon S3, while the CoSync AssetLink packages all of this into an easy to understand bundle that can be seamlessly integrated into a working client application.
Try the Cosync AssetLink Solution
The Cosync AssetLink solution is available to registered Cosync developers, along with the Cosync Auth self-hosted authentication tool for MongoDB Realm. For more information about Cosync, please consult our website at https://cosync.io.
If you’re seeking documentation on the Cosync AssetLink product, visit https://cosync.net.
To enroll as a Cosync developer, navigate to the Cosync Portal website: https://portal.cosync.net.
Swift developers can explore a sample Cosync AssetLink demo program at https://github.com/Cosync/CosyncAssetLinkSwiftDemo.
React Native developers, may God protect their souls, can try a sample Cosync AssetLink demo program written in Javascript at https://github.com:Cosync/CosyncAssetLinkReactNativeDemo.
For the more timid React Native developers, who feel the need to rely on the Expo crutch, they can try a sample Expo version of the Cosync AssetLink demo program at:
https://github.com/Cosync/CosyncAssetLinkReactExpoDemo.
For inquiries, feel free to contact us at info@cosync.io.
We trust you found this information valuable.
